{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "###

✅ **Put your name here**

\n", "####

✅ Put your group member names here

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 19 In-Class Assignment: Perceptrons" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. **Scientific motivation** \n", " - Classifying data (iris types) \n", "2. **Modeling tools** \n", " - Machine Learning (Perceptron)\n", "3. **Programming concepts** \n", " - Creating Classes and re-usable code\n", " - Pulling in data from outside sources \n", " - Using external libraries \n", " \n", "\n", "### Agenda for today's class\n", "\n", "

\n", "\n", "1. Review of pre-class assignment\n", "1. Problem Statement\n", "1. Basics of the perceptron model\n", "1. Loading and inspecting the data\n", "1. Building the perceptron model\n", "1. Plotting the decision boundary\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The Perceptron Model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The perceptron is one of the first used examples of what has come to be called a neural network. Invented in 1958, it was originally hailed as a way to achieve what had come to be called \"Artificial Intelligence\". However, it was quickly proved that the perceptron model was limited. The claims and subsequent refutations halted neural network research for a number of years.\n", "\n", "Perceptrons are used as a kind of **classifier** that we can train using examples. The simplest perceptron is what is known as a binary classifier. By this we mean that, we can provide individual examples of two classes (remember, a binary classifier) where the individuals are represented by some number of features/inputs. All examples use the same input features, but the particular feature values are used for the classification process. The goal is the create the classifier such that, when a never-seen-before individual is provided to the perceptron, it can correctly determine that individual's class\n", "\n", "There are ways to extend the perceptron's ability to deal with multiple classes (to classify the inputs as representing one of `n` classes instead of only two), but for this exercise we will only be concerned with a binary classifer." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The limitations of a binary perceptron are that the classes must be **linearly separable**. It is easier to show than to explain. Look at the two graphs below. The axes represent the range of values for the two input features. The dots represent individual input examples based on their corresponding feature values and the colors represent the class that each individual example belongs to. \n", "\n", "\n", "\n", "For experiment A, it is clear that we can draw a line through the 2D input/feature space such that we can separate the examples of the two classes. For experiment B, no such line separating the two classes exists. Furthermore, it is also clear that we could draw **many** lines for A such that we separate the two classes. \n", "\n", "The limitations therefore are:\n", "- a perceptron can only classify elements that are linearly separable\n", "- a perceptron cannot distinguish which linear boundary is \"better\"." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## It's a line\n", "\n", "The way to think about a perceptron then is that it is learning a line through the feature space. We will discuss that in a minute but accepting that is true, what is a line? A line is just a simple equation with the following form:\n", "\n", "$$y = mx + b $$\n", "\n", "Where `m` is the slope and `b` is the intercept. We are going to train our perceptron such that it finds `m` and `b` where that particular line separates two classes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The setup\n", "\n", "We are going to setup our perceptron as shown in the image below:\n", "\n", "\n", "\n", "Remembering that each individual example is represented as a set of features, a **vector** of features, we provide a **weight** associated with each feature input. When the features of a individual example is provided, we multipy each input by its associated weight and sum those products together. \n", "\n", "You may think of this process as multiplying two vectors (the input vector and the weight vector) together, index to index, and summing the resulting products as what is called the **dot product**. You might note that `numpy` provides such a function.\n", "\n", "Having obtained the dot product of feature inputs and weights, we add what is called a **bias term**. The bias is also associated with a weight (the constant 1 at the top of the inputs) though it's value is constant for all inputs. Altogether the dot product represents the `m` of our line equation and the bias represents the `b` of that equation. \n", "\n", "Having obtained the result, we pass that result through an **activation function**. Our activation function is typically a step function such that it yields either 1 or -1 indicating whether the particular input is part of one class (1) or the other (-1). Remember, binary classifier." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Loading and inspecting the data\n", "\n", "Before we build a machine learning model, we need data to base it off of. The data set we are going to use has been provided for you in the directory for this assignment, `binary-iris.csv`. It is a variation on a classic, simple classification data set for iris flowers from 1936. It is used often as a very simple test of learning systems. The original (see https://en.wikipedia.org/wiki/Iris_flower_data_set ) has 4 classes and 4 inputs, for our experiments we have modified the file to have only 2 inputs and 2 classes. \n", "\n", "**Load the data into python and visualize (with a plot) to get a sense for what it looks like. Use different colors to represent the two different iris classifications.**\n", "\n", "As an aside, try to do it without use Pandas. Let's see if we can get used to opening files and manipulating the strings we get from the file. If that proves too difficult, go ahead and use Pandas." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Do This: Load in the iris.csv file and plot the data based on the iris classifications" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Build a Perceptron class (first cut)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's build a perceptron class but let's skip the learning process for the moment. We created a \"skeleton\" class for you to fill in:\n", "\n", "- a `__init__` method. Basically every class needs this to set instance attributes\n", " - It takes as arguments the labeled data (the data features plus a label at the end)\n", "\n", " - it creates an instance variable `data` for the data from the file. It should contain 2D vector where each row represents an example. The row should consists of two features and a class. The classes should be modified for 1 (iris setosa) and -1 (iris versicolor). \n", "\n", " - it creates a 1D weight vector `weights` of the same shape as the number of features + 1. The **plus 1** is for the bias weight. For now, fill the weights with some value (say 1.0). We'll fix that later. \n", " - it creates a constant bias value `bias`\n", "\n", "- The class needs a `predict` method which takes a single argument, an array of features from a single example in the data set. It will do the following:\n", " - multipy (as a dot product) the argument feature vector and the weights\n", " - multiply the bias by its weight. Add that to the result.\n", " - Apply the activation to the result to say whether the class of that input is -1 vs 1\n", " - return that predicted class value" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## fix the skeleton code below:\n", "\n", "import numpy as np\n", "\n", "class Perceptron():\n", "\n", " def __init__ (self, labeled_data): \n", " # set three data attributes: data, weights, bias\n", " \n", " def predict(self, feature_set):\n", " # predict 1 result. dot product of features * weights + (bias * bias_weght)\n", " # return if the result is > 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Run your code with the main below. See if it works, even though it is not yet a very good classifier." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## get data from file, just using file and string ops\n", "f = open(\"binary-iris.csv\")\n", "header = next(f) # dump the header line\n", "data = []\n", "for line in f:\n", " fields = line.split(\",\")\n", " # need to strip label because, as the last element, it has a \\n\n", " label = (1.0 if fields[2].strip() == \"Iris-setosa\" else -1.0)\n", " # the fields are strings until we conver them\n", " data.append([float(fields[0]), float(fields[1]), label])\n", "f.close()\n", "\n", "p = Perceptron(data)\n", "print(\"weights: \", p.weights)\n", "print(\"bias: \", p.bias)\n", "print(\"prediction: \",p.predict([1,1])) # some arbitrary feature vector, just testing here\n", "print(p.data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Learning\n", "\n", "OK, now the interesting part. We need to learn the value of the weights, including the value of the bias weight, so that the predictions the `predict` method makes are good. How to do that?\n", "\n", "The basic idea is this. We feed in the data that we have where we know the labels (and we do) to `predict`. We then compare the classification we want (from the existing data) and the classification we got (from the `predict` method). We use that difference to update **all the weights**. We do this for **each** of the data. \n", "\n", "However, we need to do one more thing. We need to not **over correct** the weights. If we do that then the weight values might swing wildly over time and never settle down. So we also provide a `learning_rate`. This rate reduces how much the weight changes. Overall then we use the following equation:\n", "\n", "$$ self.weights[i] = self.weights[i] + self.learning\\_rate * (class\\_label - prediction\\_label) * input[i] $$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We do this for each input of the example and each corresponding weight. We also include the bias weight. We do this for some number of iterations because, with a small `learning_rate`, we need to repeat the process to get the weight values correctly set. We also update the bias weight each time we update a weight. We use the same difference, `class_label-prediction_label` for that update (even though there is no label for the bias). If `weights[0]` is the bias weight, then that equation would be\n", "\n", "$$ self.weights[0] = self.weights[0] + learning\\_rate * (class\\_label - prediction\\_ label) $$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Modify our Perceptron class\n", "\n", "We need to update our `Perceptron` class to deal with learning. Copy the class you wrote from above in the cell below and make the following changes:\n", "\n", "- `__init__` besides labeled data, it takes two more parameters: the number of iterations of learning you want and the learning_rate. They should be also stored as instance variables\n", "- a method called `fit`. It performs the number of iterations of learning provided in `__init__` on all the labeled data as described\n", "- method called `errors`. It will print out the number of errors on predicted vs actual class labels and the weights at the end of the run." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Fill in the skeleton\n", "\n", "class Perceptron():\n", "\n", " def __init__ (self, labeled_data, iters, learning_rate): \n", " # attributes now are: data, weights, bias, iterations, learning rate\n", " \n", " def predict(self, feature_set):\n", " # same as before\n", " \n", " def fit(self):\n", " # for all iterations\n", " # for each row in the data\n", " # find the update value, changes the weights (including bias weight)\n", "\n", " def errors(self):\n", " # how many rows (the features) don't match the provided label" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Run your code with the code below. Change the number of iterations, the learning rate, and see what happens. Make sure `errors` prints out the weights as you will need them below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "f = open(\"binary-iris.csv\")\n", "header = next(f) # dump the header line\n", "data = []\n", "for line in f:\n", " fields = line.split(\",\")\n", " # need to strip label because, as the last element, it has a \\n\n", " label = (1.0 if fields[2].strip() == \"Iris-setosa\" else -1.0)\n", " # the fields are strings until we conver them\n", " data.append([float(fields[0]), float(fields[1]), label])\n", "f.close()\n", " \n", "p = Perceptron(data, 10, 0.01)\n", "p.fit()\n", "p.errors()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plot the decision boundary\n", "\n", "If you have made it this far, we can try and visualize this.\n", "\n", "Once we have run the math, we can try and plot the decision boundary as well as the data entries. Let's do some more math. **Remember** the actual bias is the bias constant you used time the bias weight (typically the first weight)\n", "\n", "Our basic calculation is $w \\cdot x + b = 0 $ where `w` and `x` are vectors and the operation $\\cdot$ is the dot product. Since this is in two dimensions (two features for an input), we can rewrite this as: $w_1 * x_1 + w_2 * x_2 + b = 0$. \n", "\n", "That looks a lot like an equation of a line in the form of $Ax + By - C = 0$ if we assume $x_1 == x$ and $x_2 == y$. Let's isolate the x and y intercept:\n", "\n", "$ x = \\frac{-(b - w_2*y)}{w_1} $ but if $y=0$ (which it does at the x intercept) then $x = \\frac{-b}{w_1}$\n", "\n", "$ y = \\frac{-(b - w_1*x)}{w_2}$ but if $x=0$ (which it does at the y intercept) then $y = \\frac{-b}{w_2}$\n", "\n", "We get the following then:\n", "\n", "$slope = -\\frac{w_1}{w_2}$ and $intercept = \\frac{-b}{w_2}$\n", "\n", "Plug in your weights and plot the line. Plot the data as well where they are colored to indicate which class they belong to.\n", "\n", "The data should be stored in the `Perceptron` instance. Get the weights (which `error` should print out) and plot the line. The separate the two classes and plot each as a different color.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## write your code here to plot the decision boundary. Use the weights from above" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n", "### Congratulations, we're done!\n", "\n", "Now, you just need to submit this assignment by uploading it to the course Desire2Learn web page for today's submission folder (Don't forget to add your names in the first cell).\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "© Copyright Michigan State University Board of Trustees" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }