{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "###

✅ **Put your name here**

\n", "####

✅ Put your group member names here

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 09 In-Class Assignment: Machine Learning\n", "\n", "\n", "\n", "\n", "### Agenda for today's class\n", "\n", "

\n", "\n", "1. Review pre-class assignment\n", "1. Train vs Test\n", "1. Example classifier using breast cancer dataset\n", "1. logistic regression classifier" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports for the day" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", "import statsmodels.api as sm\n", "from sklearn import metrics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "# 1. Review of Pre-Class assignment\n", "\n", "We'll discussion any questions that came up as a class." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "# 2. Training vs Testing\n", "\n", "As you learned in the pre-class, classification is an ML process that maps features of an input data set to class labels. Classification is a **supervised** learning approach where example data is used to train the data. We typically divide the data used to train and evaluate the classifier (the result model) into three sets\n", "\n", "- training set\n", "- testing set\n", "- validation set" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✅ **Do This:** As a group, discuss what these three sets represent. It might help to review these terms on the web. The answers down below:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Training set is:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Testing set is:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Validation set is:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Defining the features and building the model\n", "\n", "If you review the image at the top of the notebook, you might notice that one of the first steps in machine learning is to go from \"raw data\" into a set of \"features\" and \"labels\", which we discussed a bit about when we worked with the perceptron model. Extracting features from our data can sometimes be one of the trickier parts of the process and also one of the most important ones. We have to think carefully about exactly what the \"right\" features are for training our machine learning algorithm and, when possible, it is advantageous to find ways to reduce the total number of features we are trying to model. Once we define our features, we can build our model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✅ **Do This:** Now, also as a group, discuss and be prepared to share with the class how you would define a feature vector from an image and how you would build a face recognition model using machine learning. Can you find any libraries that you think would be will suited for this? Which of these libraries might you actually use?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Erase the contents of this cell and put a basic outline of what your group discussed." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.1 Working with data\n", "\n", "There is a common data set used to work with classification called the breast cancer data set. It is actually available in `sklearn` but what fun is working withe cleaned up data. Let's look at the original. In the directory where this notebook is stored is \"breast-cancer-wisconsin.data\" and \"breast-cancer-wisconsin.names\". The data are in \".data\" and the \".names\" describes that data. \n", "\n", "Read in the data, label the columns based on the .names file. Look at the dtypes, anything unusual? Why?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ What's unusual about dtypes? Why?" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Can you write code to identify what the problem is? That is, can you provide a DataFrame of the offending rows that are causing the problem? There are lots of ways to do this and, frankly, it is probably a bit hard so don't get hung up too long on this. Give it a try though:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "OK, we have an imputation problem. Write code to solve it and say what you did.\n", "\n", "By the way, there is an argument `na_values` that you can provide to `read_csv` that will mark a list of characters as if they were `np.nan` using `na_values`, which is pretty darn convenient. Using that will help when importing the data for classification. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## 2.2 : Splitting the dataset for model into training and testing sets\n", "Let's split the data in a training set and final testing set. We want to randomly select 75% of the data for training and 25% of the data for testing.\n", "\n", "You should turn the `class_labels` into 0 (now 2, for benign) and 1 (now 4, for malignant) as the classifier we are using (Logisitic Regression) predicts valuse between 0 and 1." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✅ Do This - You will need to come up with a way to split the data into separate training and testing sets (we will leave the validation set out for now). Make sure you keep the feature vectors and classes together. \n", "\n", "**BIG HINT**: This is a very common step in machine learning, andthere exists a function to do this for you in the `sklearn` library called `train_test_split`. From the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html), you find that takes the features and class labels as input and returns a 4 outputs:\n", "- 2 feature sets (one for training and one for testing)\n", "- 2 class labels sets (the corresponding one for training and for testing)\n", "\n", "Use `train_test_split` to split your data into a training set and a testing set that correspond to 75% and 25% of your data respectively. Check the length of the resulting output to make sure the splits follow what you expected." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question**: Why do we need to separate our samples into a training and testing set. Why can't we just use all the data for both? Wouldn't that make it work better?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Erase the contents of this cell and replace it with your answer to the above question! (double-click on this text to edit this cell, and hit shift+enter to save the text)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## 3 Logistic Regression\n", "\n", "In the pre-class, you watched a video explaining some of the aspects of the logistic regression. The full details on logistic regression require deeper study, but we can gain some insight from looking at the function we are trying to fit to our data. Plot out the curve to the following equation, called the **logistic function***\n", "\n", "$$ f(x) = \\frac{e^{x}} {1+e^{x} } \\equiv \\frac{1}{1+e^{-x} } $$\n", "\n", "Plot that from say -6 to 6." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# your code" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What is interesting about that curve is that all values of x are mapped into the range for y of 0.0-1.0. Assuming you have a binary classifier, that is one that only has two class labels, that is the mapping you want: all combination of features map into the two class labels 0, or 1. Moreover, the graph looks a lot like a [cumulative probability distribution](https://en.wikipedia.org/wiki/Cumulative_distribution_function). The probability that a set of features is of class 1 is 1 to the right and 0 to the left. As you watched in the pre-class video, this is the basis for logistic regression.\n", "\n", "It's a regression because the \"x\" in our logisitic function is actually going to be a regression equation, such that\n", "\n", "$$ fn= b_{0} + b_{1}x_{1} + b_{2}x_{2} + \\ldots $$\n", "\n", "for as many terms as we like and the new logistic function\n", "\n", "$$ f(x) = \\frac{e^{fn(x))}} {1+e^{fn(x)} } \\equiv \\frac{1}{1+e^{-fn(x)} } $$\n", "\n", "Logisitic classification tries to find the parameters $b_{x}$ that gives maximal performance on training, and hopefully testing. Let's let `statsmodels` do that.\n", "\n", "We are going to use all the training data from above and train a logistic regression. It is similar to what we did before with regular regression, to wit:\n", "\n", "Note, very importantly, the `sm.add_constant` on the train vectors. We talked about that when we did OLS in stat models. That column of constant is what the $b_{0}$ or intercept will train against. We need that column to get an intercept." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimization terminated successfully.\n", " Current function value: 0.084633\n", " Iterations 9\n", " Logit Regression Results \n", "==============================================================================\n", "Dep. Variable: label No. Observations: 524\n", "Model: Logit Df Residuals: 514\n", "Method: MLE Df Model: 9\n", "Date: Wed, 30 Sep 2020 Pseudo R-squ.: 0.8687\n", "Time: 08:05:50 Log-Likelihood: -44.348\n", "converged: True LL-Null: -337.75\n", "Covariance Type: nonrobust LLR p-value: 1.419e-120\n", "==============================================================================\n", " coef std err z P>|z| [0.025 0.975]\n", "------------------------------------------------------------------------------\n", "const -9.6023 1.246 -7.709 0.000 -12.044 -7.161\n", "clump 0.4559 0.155 2.950 0.003 0.153 0.759\n", "C-size 0.0893 0.215 0.416 0.677 -0.331 0.510\n", "C-shape 0.2328 0.231 1.006 0.314 -0.221 0.686\n", "adhesion 0.2406 0.127 1.892 0.059 -0.009 0.490\n", "Epi-size 0.0347 0.172 0.202 0.840 -0.302 0.371\n", "Bare-Nuc 0.4782 0.104 4.612 0.000 0.275 0.681\n", "chromatin 0.4797 0.179 2.675 0.007 0.128 0.831\n", "Norm-Nuc 0.0951 0.118 0.808 0.419 -0.136 0.326\n", "mitoses 0.6944 0.334 2.077 0.038 0.039 1.350\n", "==============================================================================\n" ] } ], "source": [ "logit_model = sm.Logit(train_labels, sm.add_constant(train_vectors))\n", "result = logit_model.fit()\n", "print(result.summary() )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The \"Pseudo R-squ\" is the equivalent (mostly) of the R-squared value in Linear regression. Ranges from 0 (poor fit) to 1 (perfect fit). The P values under \"P > |z|\" are measures of significance. The null hypothesis is that the restricted model (say a constant value for `fn`) performs better but a low p-value suggests that we can reject this hypothesis and prefer the full model over the null model. This is similar to the F-test for linear regression.\n", "\n", "Based on that, remove the low-performing columns, reform the train test sets and run it again, print the summary" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question:** How do the fits of the full model and the reduced model compare? What evidence are you using to make compare these two fits?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.1 How'd it go?\n", "\n", "There are a number of ways that we can check the performance of our model; we learn new ways throughout the semester. The major difference in the standard statistics approach and supervised learning approaches is that we test our models using the data that we held out: \"the test data.\" \n", "\n", "That is, we will use our classifier model to make predictions from the test features and we can then compare those predictions to actual test labels. To test accuracy, we can use the output of the `.fit()` method of the model to predict how well the classifier works on the test data (the data it was not trained on). Conveniently that is the `.predict()` method and, again, we use it on the result of the `.fit()`. \n", "\n", "**Note:** The output from `.predict()` is not a 0/1 value as the test labels are, but rather a fraction between 0 and 1 indicating how likely each entry is to be one class or another. We can make the assumption that anything greater than 0.5 would be a 1 class and anything less than 0.5 would be a 0 class. \n", "\n", "✎ So do the following:\n", "- use the `.predict()` method (look it up) to create the predicted labels using the test input\n", "- convert the output of the `.predict()` method to the 0/1 class values of the test labels\n", "- print the resulting predicted class values of the test labels" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One of the first metrics we will use is the `accuracy_score`, which compares the predictions our model made for the test labels and the actual test labels. The `accuracy_score` is one of many metrics we can use and is included in `sklearn.metrics`. Here's the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) on `accuracy_score`.\n", "\n", "✎ Do this:\n", "- Use the `sklearn.metrics` we imported at the top and run the `accuracy_score` on the 0/1 predicted label and the test labels.\n", "- Print your accuracy result" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question:** How well did your model predict the test class labels? Given what you learned in the pre-class assignment about false positives and false negatives, what other questions should we ask about the accuracy of our model?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n", "### Congratulations, we're done!\n", "\n", "Now, you just need to submit this assignment by uploading it to the course Desire2Learn web page for today's submission folder (Don't forget to add your names in the first cell).\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "© Michigan State University Board of Trustees" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }