{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "###

✅ Put your name here.

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# CMSE 202 Homework 03\n", "\n", "\n", "\n", "\n", "\n", "\n", "### Assignment instructions\n", "\n", "Work through the following assignment, making sure to follow all of the directions and answer all of the questions.\n", "\n", "There are **25 points** possible on this assignment. Point values for each part are included in the section headers.\n", "\n", "This assignment is due at 11:59 pm on **Friday October 23rd**. It should be uploaded into the \"Homework Assignments\" submission folder for Homework 3 in your D2L webpage. Submission instructions can be found at the end of the notebook.\n", "\n", "**Hint**: It is possible you are asked to do something you are not familiar with. That's why you have internet access. Do some smart searches and see what you can find! \n", "\n", "\n", "### Our imports" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn import metrics\n", "import statsmodels.api as sm\n", "from sklearn.model_selection import cross_val_score\n", "from sklearn.model_selection import GridSearchCV" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1: Setting up a repository for tracking changes (3 points)\n", "\n", "For this assignment, you're going to add it to the CMSE202 repository you created in class so that you can track your progress on the assignment and preserve the final version that you turn in. In order to do this you need to:\n", "\n", "* Navigate to your `/CMSE202/repos` repository and create a new directory called `hw-03`.\n", "* Move this notebook into that new directory in your repository, then add it and commit it to your repository.\n", " * Finally, to test that everything is working, \"git push\" the file so that it ends up in your GitHub repository.\n", "\n", "Important: Make sure you've added your TA as a collaborators\\ to your respository with \"Read\" access so that we can see your assignment.\n", "\n", "* Section 001: tuethan\n", "* Section 002: Luis-Polanco\n", "* Section 003: DavidRimel\n", "\n", "Also important: Make sure that the version of this notebook that you are working on is the same one that you just added to your repository! If you are working on a different copy of the noteobok, none of your changes will be tracked.\n", "\n", "\n", "If everything went as intended, the file should now show up on your GitHub account CMSE202 repository under the `hw-03` directory that you just created. Periodically, you'll be asked to commit your changes to the repository and push them to the remote GitHub location. Of course, you can always commit your changes more often than that, if you wish. It can be good to get into a habit of committing your changes any time you make a significant modification, or when you stop working on the project for a bit." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: Load, prepare and plot the data (5 points)\n", "\n", "In this homework we will be working with the yeast dataset and building logistic regression and k-nearest neighbors classifier class. The data file is *yeast.data* and its description is in *yeast.names*. Read the description and get a sense of the meaning of the dataset. In this part, we will load and clean up the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 2.1 (1 point)** Load the *yeast.data* as a pandas dataframe and give appropriate names to the columns. Then drop the columns **sequence name**, **pox** and **vac**. What's the size of this dataset now?" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 2.2 (1 point)** Find the number of unqiue entries in the class label column" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 2.3 (1 point)** We are only interested in data with label **CYT (cytosolic or cytoskeletal)** and **MIT (mitochondrial)**. Make a new dataframe containing\n", "data with only these two types of labels, and redefine label **CYT** into **0**, and **MIT** into **1**. What's the size of the dataset now?" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 2.4 (2 points)** Make a scatter plot including every sample in the dataset with: the mcg feature on the x-axis, the gvh feature on the y-axis, and different colors for each class label. Make your observation. Are the two classes distinguishable using only those two features?" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Logistic Regression\n", "\n", "In the next part we will build a logistic regression model for the data classification.\n", "\n", "## Part 3: Prepare data and build the logistic regression model (7 points)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 3.1 (2 points)** Apply the \"train_test_split\" function in the *sklearn* package to split the data in 70% for training and 30% for testing. Using common variable names like x_train, y_train, x_test and y_test might help later." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 3.2 (2 points)** Perform the logistic regression. \n", "* Discuss your results. How well does your model fit your data? What evidence are you using to make the determination? \n", "* Based on the P values under \"P > |z|\", which two features **in this dataset** are the least significant and can be dropped?" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 3.3 (3 points)** Drop the two least important features found in the previous question and perform the logistic regression again. Then use the use the `sklearn.metrics` we imported at the top and run the `accuracy_score` on the 0/1 predicted label and the test labels, and print the accuracy of this model.\n", "\n", "* Discuss your results. How well does your reduced model fit your data? What evidence are you using to make the determination?" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# K-Nearest Neighbors\n", "\n", "In the next part we will be building a class that will use the k-nearest neighbors algorithm (kNN) to make predictions on the same dataset. From the previous part (logistic regression), you have selected **4 features** that are important for classification. We will **only** use those 4 features in this part.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 4: KNN classifier, cross-validation and hyperparameter tuning (10 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 4.1 (3 points)** Test drive the KNN classifier. Use the same train and test data you created in question 3.4 to build a KNN classifier with K=3. \n", "- make a `KNeighborsClassifier` with an argument of `n_neighbors=3`. This returns a knn classifier (let's just call it `knn`)\n", "- call `knn.fit` on the training data\n", "- use `knn.predict` on the testing data to generate the predicted values.\n", "- print the confusion matrix.\n", "- print the train and test score using `knn.score`.\n", "- plot the ROC curve with the diagonal (the \"chance line\") also labeled. Using `sklearn.metrics`, print the `auc` for this model." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## k-Fold Cross-Validation\n", "Cross-validation is when the dataset is randomly split up into ‘k’ groups. One of the groups is used as the test set and the rest are used as the training set. The model is trained on the training set and scored on the test set. Then the process is repeated until each unique group as been used as the test set.\n", "For example, for 5-fold cross validation, the dataset would be split into 5 groups, and the model would be trained and tested 5 separate times so each group would get a chance to be the test set. This can be seen in the graph below.\n", "\n", "\n", "\n", "The train-test-split method we used in earlier is called ‘holdout’. Cross-validation is better than using the holdout method because the holdout method score is dependent on how the data is split into train and test sets. Cross-validation gives the model an opportunity to test on multiple splits so we can get a better idea on how the model will perform on unseen data.\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 4.2 (2 points)** Look up `cross_val_score` in `sklearn.model_selection`. We will still use n_neighbors=3, and a cross-validation value of 5. `cross_val_score` takes in our k-NN model and our data as parameters. Then it splits our data into 5 groups and fits and scores our data 5 seperate times, recording the accuracy score in an array each time. We will save the accuracy scores in the cv_scores variable. Then find the average of the cv_scores, that will provide you a more accurate understanding of the accuracy of the model.\n", "\n", "* Discuss your results. How well do your models fit your data? \n", "* What are you using to judge that fit (i.e., how should we think about the accuracy score as a measure of quality of the model)?\n", "* How does the quality of the KNN model compare to logistic regression?" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Hyperparameter tuning\n", "\n", "\n", "Almost all machine learning models have hyperparamters. Hyperparameters are setting(s) in the model that the user needs to choose before learning takes place. For example, in k-nearest neighbors, the number of neighbors to consider n_neighbors, is the hyperparameter. An important task in machine learning is hyperparameter tuning, which is finding the optimal hyperparmeter. We will now explore the optimal choice of this parameter for this dataset.\n", "\n", "**Question 4.3 (3 points)** Consider the range of `n_neighbors` from 1 to 100, and fix the cross-validation value to be 5. \n", "- For each value of n_neighbors, compute the means of the cv_scores. \n", "- Make a plot with the x-axis being n_neighbors, y-axis being the mean of cv_scores.\n", "- Find the optimal choice of n_neighbors with the largest value of the mean of cv_scores.\n", "\n", "Discuss your results\n", "* How does the quality of this model compare to the earlier models that you made with KNN and logisitic regression?\n" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we will use a more efficient method: `GridSearchCV` in `sklearn.model_selection` to find the optimal n_neighbors.\n", "\n", "**Question 4.4 (2 points)** Look up `GridSearchCV` in `sklearn.model_selection`. We will still use a cross-validation value of 5. Use `best_params_` in `GridSearchCV` to find the optimal n_neighbors. Does it agree with the results from question 4.3?" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "---\n", "### Assignment wrap-up\n", "\n", "Please fill out the form that appears when you run the code below. **You must completely fill this out in order to receive credit for the assignment!**" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from IPython.display import HTML\n", "HTML(\n", "\"\"\"\n", "\n", "\"\"\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Congratulations, you're done!\n", "\n", "Submit this assignment by uploading it to the course Desire2Learn web page. Go to the \"Homework Assignments\" folder, find the dropbox link for Homework 3, and upload your notebook **and the script you wrote**." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }