{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# CMSE 202 Homework 4 (Individual)\n",
"\n",
"## Using SVM and PCA to predict the outcome of chess games\n",
"\n",
"### Goals for this homework assignment\n",
"\n",
"By the end of this assignment, you should be able to:\n",
"\n",
"* Use `git` to track your work and turn in your assignment\n",
"* Read and impute data to prepare it for modeling\n",
"* Build, fit, and evaluate an SVC model of data\n",
"* Use PCA to reduce the number of important features\n",
"* Build, fit, and evaluate an SVC model of pca transformed data\n",
"* Systematically investigate the effects of the number of components on an SVC model of data\n",
"\n",
"\n",
"### Assignment instructions:\n",
"\n",
"Work through the following assignment, making sure to follow all of the directions and answer all of the questions.\n",
"\n",
"There are 25 points possible on this assignment. Point values for each part are included in the section headers.\n",
"\n",
"This assignment is due at 11:59 pm on Friday, November 13th. It should be pushed to your repo (See Part 1). "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"## Our imports\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pylab as plt\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.model_selection import GridSearchCV\n",
"from sklearn.metrics import classification_report\n",
"from sklearn.metrics import confusion_matrix, accuracy_score\n",
"from sklearn.svm import SVC\n",
"from sklearn.decomposition import PCA"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 1. Adding notebook to your turn-in repository\n",
"\n",
"Like you did for Homework 3, you're going to add it to the CMSE202 repository you created in class so that you can track your progress on the assignment and preserve the final version that you turn in. In order to do this you need to:\n",
"\n",
"* Navigate to your /CMSE202/repos repository and create a new directory called hw-04.\n",
"* Move this notebook into that new directory in your repository, then add it and commit it to your repository.\n",
" * Finally, to test that everything is working, \"git push\" the file so that it ends up in your GitHub repository.\n",
"\n",
"Important: Make sure you've added your TA as a collaborators to your respository with \"Read\" access so that we can see your assignment. (*If you did this for Homework 3, you do not need to do it again*)\n",
"\n",
"* Section 001: tuethan\n",
"* Section 002: Luis-Polanco\n",
"* Section 003: DavidRimel\n",
"\n",
"Also important: Make sure that the version of this notebook that you are working on is the same one that you just added to your repository! If you are working on a different copy of the notebook, none of your changes will be tracked.\n",
"\n",
"If everything went as intended, the file should now show up on your GitHub account CMSE202 repository under the hw-04 directory that you just created. Periodically, you'll be asked to commit your changes to the repository and push them to the remote GitHub location. Of course, you can always commit your changes more often than that, if you wish. It can be good to get into a habit of committing your changes any time you make a significant modification, or when you stop working on the project for a bit."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 2. Chess Game Data\n",
"\n",
"The data you will work are configurations of a chess end game. It assumes that a pawn is one move away from [\"queening\"](https://en.wikipedia.org/wiki/Promotion_(chess)) and \n",
"the other pieces can be moved to perform different offensive or defensive actions. For each of the 36 potential features, there are several potential values for each (entries in a given column). **The details of the data matter a bit less for our purposes, but we are attempting to predict the won/loss by a given side.** If you really want to know about the data, you can look into a [classic text on Artificial Intelligence by Shapiro](https://www.amazon.com/Encyclopedia-Artificial-Intelligence-Stuart-Shapiro/dp/0471807486).\n",
"\n",
"You will first do this with a full model, then investigate how well the model works after a PCA has been done on the data.\n",
"\n",
"### 2.1 Read in the data\n",
"\n",
"First you need to read in the data from `kr-vs-kp.data`. You can look at `kr-vs-kp.names` to see how the data is structured. But we give you the code for the column naming as there are so many features and they are unlabeled in the `.data` file.\n",
"\n",
"```cols = [\"bkblk\",\"bknwy\",\"bkon8\",\"bkona\",\"bkspr\",\"bkxbq\",\"bkxcr\",\"bkxwp\",\"blxwp\",\"bxqsq\",\"cntxt\",\"dsopp\",\"dwipd\",\n",
" \"hdchk\",\"katri\",\"mulch\",\"qxmsq\",\"r2ar8\",\"reskd\",\"reskr\",\"rimmx\",\"rkxwp\",\"rxmsq\",\"simpl\",\"skach\",\"skewr\",\n",
" \"skrxp\",\"spcop\",\"stlmt\",\"thrsk\",\"wkcti\",\"wkna8\",\"wknck\",\"wkovl\",\"wkpos\",\"wtoeg\",\"won\"]```\n",
" \n",
"✎ Do this - Read in the data from `kr-vs-kp.data` using the columns listed above. Print the `.head()` of the dataframe."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2 Imputing the data\n",
"\n",
"There are no missing data in this data file, but there are some other issues. \n",
"\n",
"When you print the head of this data set, you probably noticed that all the features and labels are strings. We need to replace them with numerical values for modeling. For the `won` column replace winning with a 1 and losing with a 0. For the other columns, there are seven strings. Replace them using the following table:\n",
"\n",
"| raw data | replaced |\n",
"| -------- | -------- |\n",
"| f | 1 |\n",
"| l | 2 |\n",
"| n | 3 |\n",
"| t | 4 |\n",
"| w | 5 |\n",
"| b | 6 |\n",
"| g | 7 |\n",
"\n",
"**Note:** this choice really matters and for the models we have learned can really influence the results of our model. We do this because we need to for the model, but we haven't critically thought about the mapping that makes the most sense. There are other models (e.g., [tree-based alogrithms](https://en.wikipedia.org/wiki/Random_forest)) that can handle these categorical data without this mapping.\n",
"\n",
"✎ Do this - Replace the entries in the columns as indicated above. Print the `.head()` of the dataframe to show you have succesfull done so."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.3 Separate features and class labels\n",
"\n",
"As we have seen in our analyses using `sklearn` it is advantageous to separate our dataframes into `features` and `labels` for the analysis we are intending to do.\n",
"\n",
"✎ Do this - Separate the data frame into two: a features dataframe and a labels dataframe."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question:** How balanced is your outcome variable? Why does it matter for the outcome to be balanced?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Answer here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 3. Building an SVC model\n",
"\n",
"For this classification problem, we will use an support vector machine. As you learned in the midterm review, we could easily replace this with any `sklearn` classifier we choose. We will use a linear kernel.\n",
"\n",
"### 3.1 Splitting the data\n",
"\n",
"✎ Do this - Split your data into a training and testing set with a train size representing 75% of your data. Print the lengths to show you have the right number of entries."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3.2 Modeling the data and evaluting the fit\n",
"\n",
"As you have done this a number of times, we ask you to do most of the analysis in one cell.\n",
"\n",
"✎ Do this - Build a linear SVC model (`C=100`), fit it to the training set, use the test features to predict the outcomes. Evaluate the fit using the confusion matrix and classification report.\n",
"\n",
" **Note:** You should look at the documentation on the confusion matrix because the way `sklearn` outputs false positives and false negatives is different from what most images on the web indicate."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question:** How accurate is your model? What eveidence are you using to determine that? How many false positives and false negatives does it predict?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Answer here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 4. Finding and using the best hyperparameters\n",
"\n",
"We have fit one model and determined it's performance, but is it the best model? We can use `GridSearchCV` to find the best model (given our choices of parameters). Once we do that, we will use that best model going forward. **Note:** you would typically rerun this grid search in a production environment to continue to verify the best model, but we are not for the sake of speed.\n",
"\n",
"### 4.1 Grid search\n",
"\n",
"✎ Do this - Using the following parameters (`C` = 1, 10, 100, 1000 and `gamma` = 1e-4, 1e-3, 0.01, 0.1) for both a `linear` and `rbf` kernel use `GridSearchCV` with the `SVC()` model to find the best fit parameters. Print the \"best estimators\"."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.2 Evaluating the best fit model\n",
"\n",
"Now that we have found the \"best estimators\", let's determine how good the fit is.\n",
"\n",
"✎ Do this - Use the test features to predict the outcomes for the best model. Evaluate the fit using the confusion matrix and classification report. \n",
"\n",
"**Note:** You should look at the documentation on the confusion matrix because the way `sklearn` outputs false positives and false negatives is different from what most images on the web indicate."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question:** How accurate is this best model? What evidence are you using to determine that? How many false positives and false negatives does it predict?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Answer here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 5. Using Principal Components\n",
"\n",
"The full model uses 36 features to predict the results. And you likely found that the model is incredibly accurate. But in some cases, we might have even more features (which means much more computational time), and we might not need nearly the level of accuracy we can achieve with the full data set. So, we will see how close we can get with fewer features. But instead of simply removing features, we will use a PCA to determine the featurse that contribute the most the model (through their accounted variance) and use those to build our SVC model.\n",
"\n",
"### 5.1 Building a PCA\n",
"\n",
"We will start with a small number of components (say, 4) to see how well we can predict the outcomes of the games.\n",
"\n",
"✎ Do this - Using `PCA()`, fit a pca to your training features with 4 components. Transform both the test and training features using this pca. Plot the `explained_variance_` versus component number."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question:** What is the total explained variance captured by this PCA (we will use this later, just quote the number)? How well do you think a model with this many featuers will perform? Why?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Answer here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5.2 Fit and Evaluate an SVC model\n",
"\n",
"Using the pca transformed features, we will train and test an SVC model using the \"best estimators\".\n",
"\n",
"✎ Do this - Using the pca transformed training data, build and train an SVC model. Predict the classes using the pca transformed test data. Evaluate the model using the classfication report, and the confusion matrix."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question:** How accurate is this model? What evidence are you using to determine that? How many false positives and false negatives does it predict? How does it compare to the full model?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Answer here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5.3 Repeat your analysis with more components\n",
"\n",
"You probably found that the model with 4 features didn't work so well. What if we increase the number of components (say to 30, which is still 6 fewer than the full data set). What happens now?\n",
"\n",
"✎ Do this - Repeat your analysis from 5.1 and 5.2 using 30 components instead."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question:** What is the total explained variance captured by this PCA? How accurate is this model? What evidence are you using to determine that? How many false positives and false negatives does it predict? How does it compare to the 4 component model? To the full model?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Answer here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 6. How well does a PCA work?\n",
"\n",
"Clearly, the number of components we use in our PCA matters. Let's investigate how they matter by systematically building a model for any number of selected components.\n",
"\n",
"### 6.1 Accuracy vs. Components\n",
"\n",
"We will do this by writing a function that creates the PCA, the SVC model, fits the training data, predict the labels using test data, and returns the accuracy scores and the explained variance. So your function will take as input:\n",
"* the number of components\n",
"* the training features\n",
"* the test features\n",
"* the training labels\n",
"* the test labels\n",
"and it will return the accuracy scores for an SVC model fit to pca transformed features and the total explained variance.\n",
"\n",
"✎ Do this - Create this function, which you will use in the next section."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 6.2 Compute accuracies\n",
"\n",
"Now that you have created a function that returns the accuracy for a given number of components, we will use that to plot the how the accuracy of your SVC model changes when we increase the number of components used in the PCA.\n",
"\n",
"✎ Do this - For 1 to 36 components, use your function above to compute and store (as a list) the accuracy of your models."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 6.3 Plot accuracy vs number of components\n",
"\n",
"Now that we have those numbers, it makes sense to look at the accuracy vs components.\n",
"\n",
"✎ Do this - Plot the accuracy vs components."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question:** Where does it seem like we have diminishing returns, that is, no major increase in accuracy as we add additional components to the PCA?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Answer here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 6.4 Plot total explained variance vs number of components\n",
"\n",
"✎ Do this - Plot the total explained variance vs components. "
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"## your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question:** Where does it seem like we have diminishing returns, that is, no major increase in explained variance as we add additional components to the PCA? How does that number of components compare to the diminishing returns for accuracy?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Answer here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 7. Assignment wrap-up¶\n",
"Please fill out the form that appears when you run the code below. **You must completely fill this out in order to receive credit for the assignment!**"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
""
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from IPython.display import HTML\n",
"HTML(\n",
"\"\"\"\n",
"\n",
"\"\"\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Congratulations, you're done!\n",
"Submit this assignment by uploading it to the course Desire2Learn web page. Go to the \"Homework Assignments\" folder, find the dropbox link for Homework 4, and upload your notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}