{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "###

✅ **Put your name here**

" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "# Day 10 In-Class Assignment: K-Nearest Neighbors as a binary classifier\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "### Goals for today's class\n", "\n", "1. Read, impute, and scale the Pima data set\n", "1. Visualize the feature space before and after scaling and compare\n", "1. Build a KNN model and evaluate the quality of it" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Our Imports" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn import metrics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A note**: For most of the rest of the semester, we will be doing we will use the `sklearn` package. The `statsmodels` package provides better statistical support but is not nearly as broad as `sklearn`. They each have their advantages and we work with each for that reason." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## 1. K-Nearest Neighbors (knn)\n", "\n", "Of all the classifiers, knn is one of the simpler methods. As you watched in the pre-class video, a k-nearest neighbor algorithm works as follows:\n", "- each on the n features of the observations is used as a dimension in a feature space\n", "- training is done by placing training set elements in the feature space based on their feature values.\n", " - remember that each of the training elements is already labeled as to its class label.\n", "- to test the accuracy of this model, a test observation from the test set is placed in the feature space based on its n feature values\n", "- the algorithm then establishes the \"distance\" between the test obervations and all the training observations in the model\n", " - we can imagine many ways to establish \"distance\", but whatever we metric we use, we have a set of distances\n", "- the algorithm finds the \"k\", where k is a parameter to be set by the user, nearest neighbors based on our distance metric\n", "- the algorithm labels the test observation based on the majority of labels of the k-nearest training labels." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is an image of the classic iris data and 3 of the 4 features. We use only 3 features because we can more easily draw in 3D, not 4D, but there is no reason not to have many dimensions in a knn model.\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Talk about this basic idea in our group and record that advantages and disadvantages of using knn." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ knn advantages:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ knn disadvantages:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll circle back on this discussion at the end." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## 2. Pima Diabetes Dataset\n", "\n", "The Pima are a Native American group living in what is now central/southern Arizona and Mexico.\n", "There is a high prevelance of Type-2 diabetes in the group so 8 features of 768 individuals were recorded to find possible underlying causes. We are going to use a knn to predict diabetes in Pima inidividuals.\n", "\n", "There are some problems with this data and we need to address each:\n", "- bad data values\n", "- uneven value ranges in the features\n", "- uneven distribution of well vs diabetic individuals.\n", "\n", "Let's look at each. Before we do that, go ahead and read in the file `pima-diabetes.csv`. Please look at the file before you try and read it in. \n", "\n", "✎ - Do this. Read the data file (pima-diabetes.csv) in **without** modifying the file. Give columns approriate names (did you look at the file to see what each column represents?). Run `.describe` on the DataFrame. Do you notice anything unusual?" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question:** What did you notice? Remember these data are meant to represent real health information." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1 Problems with the data\n", "\n", "There are no missing values (no '?' values, no numpy.nan, etc.) but there are some problems. It takes a little (but not a lot) of medical knowledge to notice a couple of things:\n", "\n", "- The min of blood pressure is 0. That's bad, because 0 blood pressure means you are dead. These were live people that were sampled, not corpses, so that is some bad data.\n", "- same problem with insulin (0 insulin means dead), BMI (Body Mass Index, 0 means 0 weight, not good) and glucose (0 sugar in the bloodstream means dead)\n", "- a \"Triceps skin fold thickness\" of 0 sounds wrong. If you try to measure, with calibres, how much of a skin fold you can make, 0 sounds like it wasn't done.\n", "- pregnancy count has a max of 17. OK, maybe but that seems a bit high\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's just worry about the 0's for now and try what we have done before. If we took the approach of throwing out any row with a 0 for: glucose, BP, insulin, BMI and fold, how many rows would we be throwing out? \n", "\n", "✎ Do this. Count the number of rows for each of the features above that has a zero for the value of the feature. Print the rows that would be dropped if we tossed out any row where the value of one of those features is zero." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question:** What is the total in each category? How many total rows would be thrown out?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.2 Imputing data\n", "\n", "It is probably clear that we cannot throw out rows based on any 0 in those 5 categories. There wouldn't be much data left for modeling! Furthermore it looks like two of the categories (which ones???) have a very high number of 0's. What should we do?\n", "\n", "We discussed other ways of imputing data when we performed linear regression. For this case, replacing each missing value with the median for each category might be OK, but we can probably do better than that. Let's separate out the positive (label == 1) and negative (label == 0) cases and replace the 0 values for each feature with their column mean. This is a standard approach that keeps us data with the same class label together when imputing. That is, we will calculate the means of features from data in the same class.\n", "\n", "Two tips:\n", "- If we replace the 0's with `np.nan`, then the mean of that column will not use the `np.nan` in the calculation\n", "- Unfortunately, the class labels are 1 and 0 so we have to separate out the labels columns so we don't lose all the label==0 values.\n", "\n", "✎ Do this.\n", "- make a DataFrame without the labels (just the features). \n", "- separate that DataFrame based on the class label (0 or 1, diabetes or not) into two DataFrames, positive and negative.\n", "- replace all the `np.nan` in the positive and negative DataFrames with their column means\n", "- merge the positive and negative back together. The `concat` method is helpful and its documentation is [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html).\n", "- add the labels back to the merged DataFrame" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.3 Visualizing the Feature Space\n", "\n", "If we build a full model using all the features available, we will have a 8-dimensional model. We cannot visualize 8 dimensions at once to see how the data is distributed the feature space. So, instead, we will need to visualize 2D subspaces where the we make a scatter plot of two features at a time.\n", "\n", "✎ Do this. Make a few scatter plots of pairs of features to see how the data is distributed. You might find it useful to crib this code from your pre-class assignment (reuse and refactor code when you can!). Answer the question below after you do so." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question:** Do you notice anything about the plots you made? Is it possible to tell from any of these how the data might separate into the two classes? What about the values of the features (i.e., their scale)? Are all the features comparably scaled?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.4 Standardize the range of values\n", "\n", "If you look at the values, some columns range from 1-17, some from 14-846. Those different ranges greatly affect the distance calculation that knn uses. Moving 800 units is much different from moving 8 units. It would be helpful to get all the now imputed values into the same range. This is called, \"standardizing\" your data and it's quite common to do when building models of data.\n", "\n", "We can use the `StandardScaler` found in `sklearn.preprocessing`. We will first need to create a `StandardScaler()` object, we can call is `scaler`. Then we will use the `.fit_transform()` method to scale the feature vectors. The documentation for `StandardScaler` is [here](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html).\n", "\n", "✎ Scale the features vectors. Be careful not to scale the labels! Describe the data when you have finished." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do this. Plot the some pairs of features again and take note of the scale of the axes now." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question:** How do the scale of the variables compare to the previous plots? Do the same pairs of plots appear to have the same structure (i.e., the same distribution of data)?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.5 Starting our model: Train and Test\n", "\n", "We need to set up training and testing sets to make our knn model and evalaute it. You will need to create the train/test sets. Decide as a group what would be a good division between training and testing sets would be.\n", "\n", "**Question:** What is that division is and why did your group select it?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Your train/test split and reasoning." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do this. Now, write the code to make the split and visualize (however you like) the count of the two class labels in the training set." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.6 Outcome Imbalances: Equal values of both positive and negative cases\n", "\n", "One can imagine that having a largely unequal number of positive or negative cases would greatly affect a knn. This is often called outcome imbalance and can be a very challenging problem to solve. \n", "\n", "**Question:** Why might you expect there to be a problem for knn if the outcome classes are very imbalanced?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Uneven set size problems:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## 3 KNN\n", "\n", "To run the knn we can use the `KNeighborsClassifier` found in `sklearn.neighbors` To do so you:\n", "- make a `KNeighborsClassifier` with an argument of `n_neighbors=?` whatever the k of the knn should be. This returns a knn classifier (let's just call it `knn`)\n", "- call `knn.fit` on the training data\n", "- use `knn.predict` on the testing data to generate the predicted values.\n", "\n", "Do that now:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.1 How'd did the model do?\n", "\n", "The `sklean.metrics` module provides a number of measures that we discussed in the pre-class assignment. Using the [documentation](https://scikit-learn.org/stable/modules/classes.html#sklearn-metrics-metrics) for `sklearn.metrics`, provide the following evaluative measures of your model:\n", "- confusion matrix\n", "- accuracy\n", "- precision\n", "- recall (sensitivity)\n", "- $F_1$ score" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question:** Based on these metrics, how did your model perform? Provide as plain langauge an interpretation as you can. If it helps, review your answers to the pre-class assignment." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.2 Visualizing the ROC curve\n", "\n", "As we discussed in the pre-class assignment, the ROC curve can provide more evidence as to the quality of our model. We can also calculate the AUC to determine how well our classifier performed.\n", "\n", "\n", "✎ Do this. Plot the ROC curve with the diagonal (the \"chance line\") also labeled. Using `sklearn.metrics`, print the `auc` for this model." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "# your code" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question:** Based on these additional metrics, how did your model perform? You might look up how AUC is used to determine the quality of a model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n", "### Congratulations, we're done!\n", "\n", "Now, you just need to submit this assignment by uploading it to the course Desire2Learn web page for today's submission folder (Don't forget to add your name in the first cell)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "© Copyright Michigan State University Board of Trustees" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3 CMSE", "language": "python", "name": "python3cmse" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 4 }