{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "###

✅ Put your name here

\n", "####

✅ Put your group member names here

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 06 In-Class Assignment: Multidimensional Regression\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Goals\n", "* read in yet another data format\n", "* work with bad data\n", "* multiple regression with " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Our imports" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import statsmodels.api as sm\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1. Working with example data \n", "\n", "We are going to work with some data generated by U.N.E.S.C.O. (United Nations Education, Scientific, and Cultural Organization) and data they collected relating to poverty and inequality in the world. There are two files you need to do the work:\n", "\n", "- `poverty.dat` which is the data file itself\n", "- `poverty.txt` which describes the data columns as **fixed width column** data. That is, this file describes the columns of the data for each category. For example, the data in columns 1-6 of `poverty.dat` contain the \"live birth rates per 1,000 population\".\n", "\n", "Conveniently there is a fixed width column pandas data reader. Look it up and read in the data.\n", "\n", "Again, no column headers on this so look at the `poverty.txt` file and give them short but useful names.\n", "\n", "✎ Do This - Read in the data into a DataFrame and print the `head()`." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.1 Type of the data\n", "\n", "Now look at the `.dtypes` of your DataFrame and describe to me anything unusual. Can you explain why? Please write below. Don't skimp and look ahead, think about it and answer! We'll all wait ⏲" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.2 Handling missing data - Imputation\n", "\n", "Let's face it, sometimes data is bad. Values are not recorded, or are mis-recorded, or are so far out of expectation that you expect there is something wrong. On the other hand, just **changing** the data seems like cheating. We have to work with what we have, and if we have to make changes it would be good to do that programmatically so that it is recorded for others to see. \n", "\n", "The process of imputation is the statistical replacement of missing/bad data with substitute values. We have that problem here. In the **GNP** column some of the values are set to \" \\* \" indicating missing data. When pandas read in the column the only type that makes sense for both characters and numbers is a string. Therefore it set the type to `object` instead of the expected `int64` or `float64`.\n", "\n", "### Using numpy.nan\n", "\n", "For better or worse, pandas assumes that \"bad values\" are marked in the data as numpy **NaN**. NaN is short for \"Not a Number\". If they are so marked we have access to some of the imputation methods, replacing NaN with various values (mean, median, specific value, etc.). \n", "\n", "There are two ways to do this:\n", "1. you can do a `.replace` on the column using a dictionary of {value to replace : new value, ...} pairs. Remember to save the result. This leaves you with changing the column type using `.astype` but you will have convert to a float, perhaps `\"float64\"` would be good. You cannot convert a `np.nan` to an integer but you can to a float.\n", "2. you can convert the everything that can be converted to a number using `.to_numeric`. Conveniently if it can't do the conversion on a particular value it is set to a `np.nan`\n", "\n", "✎ Do This - Convert the missing entries in the GNP column to `np.nan` and show the head of your modified DataFrame. Also print the `dtypes` to show that the column has change type." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Changing numpy.nan\n", "\n", "Now that \"bad values\" are marked as `numpy.nan`, we can use the DataFrame method `fillna` to change those values. For example:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 600.0\n", "1 2250.0\n", "2 2980.0\n", "3 0.0\n", "4 2780.0\n", " ... \n", "92 220.0\n", "93 110.0\n", "94 220.0\n", "95 420.0\n", "96 640.0\n", "Name: GNP, Length: 97, dtype: float64" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "poverty_df[\"GNP\"].fillna(0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "returns a new DataFrame where all the `np.nan` in the GNP column are replaced with 0. You can do other things are well, for example:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Birth RateDeath RateInfant MortalityLife Expectancy MLife Expectancy FGNPGroupCountry
024.75.730.869.675.5600.0000001Albania
112.511.914.468.374.72250.0000001Bulgaria
213.411.711.371.877.72980.0000001Czechoslovakia
312.012.47.669.875.95741.2527471Former_E._Germany
411.613.414.865.473.82780.0000001Hungary
...........................
9252.215.6103.049.952.7220.0000006Uganda
9350.514.0106.051.354.7110.0000006Tanzania
9445.614.283.050.353.7220.0000006Zaire
9551.113.780.050.452.5420.0000006Zambia
9641.710.366.056.560.1640.0000006Zimbabwe
\n", "

97 rows × 8 columns

\n", "
" ], "text/plain": [ " Birth Rate Death Rate Infant Mortality Life Expectancy M \\\n", "0 24.7 5.7 30.8 69.6 \n", "1 12.5 11.9 14.4 68.3 \n", "2 13.4 11.7 11.3 71.8 \n", "3 12.0 12.4 7.6 69.8 \n", "4 11.6 13.4 14.8 65.4 \n", ".. ... ... ... ... \n", "92 52.2 15.6 103.0 49.9 \n", "93 50.5 14.0 106.0 51.3 \n", "94 45.6 14.2 83.0 50.3 \n", "95 51.1 13.7 80.0 50.4 \n", "96 41.7 10.3 66.0 56.5 \n", "\n", " Life Expectancy F GNP Group Country \n", "0 75.5 600.000000 1 Albania \n", "1 74.7 2250.000000 1 Bulgaria \n", "2 77.7 2980.000000 1 Czechoslovakia \n", "3 75.9 5741.252747 1 Former_E._Germany \n", "4 73.8 2780.000000 1 Hungary \n", ".. ... ... ... ... \n", "92 52.7 220.000000 6 Uganda \n", "93 54.7 110.000000 6 Tanzania \n", "94 53.7 220.000000 6 Zaire \n", "95 52.5 420.000000 6 Zambia \n", "96 60.1 640.000000 6 Zimbabwe \n", "\n", "[97 rows x 8 columns]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "poverty_df[\"GNP\"].fillna(poverty_df[\"GNP\"].mean() )\n", "poverty_df.fillna({\"GNP\": poverty_df[\"GNP\"].mean() })" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first version changes any `np.nan` in the `GNP` column to be the mean of the column. The second takes a dictionary where the the key of the dictionary is the column to change and the value is what to replace the `np.nan` with. Note you could replace with other values like: median, min, max, or some other fixed value.\n", "\n", "Remember that all of these examples return either a new Series (when working with just a column) or a DataFrame (if working with the entire element). Nothing is changed in the original unless you assign the result or use `inplace=True` in the call.\n", "\n", "Finally, if you decide that the right thing to do is **remove** any row with a `np.nan` value, we can use the `.dropna` method of DataFrames as shown below:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "97 91\n" ] } ], "source": [ "len(poverty_df)\n", "poverty_df_dropped = poverty_df.dropna()\n", "print(len(poverty_df), len(poverty_df_dropped))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What do you think\n", "\n", "In the cell below, discuss with your group what you think is the best thing to do with the \"bad values\" in the DataFrame given the discussion above. Write your result below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Multiple Regression\n", "\n", "In the past, we have limited ourselves to either a single feature or, in the pre-class, doing polynomial regression with other features we created. However, we can just as easily use all, or some combination of all, the features available to make a OLS model. The question is, is it a good idea to just use all the possible features available to make a model?\n", "\n", "Please discuss that idea with your group and record your answer below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1 Infant Mortality model\n", "\n", "Using the U.N.E.S.C.O. data, we can make a model of \"Infant Mortality\" as the dependent variable against all the other available features. As a hint, an easy way to do this is the make the `sm.OLS` model with \"Infant Mortality\" as the first argument (the dependent variable) and then the entire DataFrame where \"Infant Mortality is dropped as the second argument. **You should also drop the \"Country\" column as unique strings don't play well in basic linear models.**\n", "\n", "✎ Do This - Make an OLS model that predicts \"Infant Mortality\" using the other variables (dropping the \"Country\" column) and print the `.summary` of that process. " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are several interesting things about this `.summary`. Let's start with things you have seen before. Look for the adjusted $R^2$ statistic. What does this adjusted $R^2$ tell you about how well your model fits your data?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's look at something new, the \"P\" values associated with the features used in the model. [P values](https://en.wikipedia.org/wiki/P-value) are used widely in statisical testing to judge if a result is statistically significant. Those P values that are 0 (or typically less 0.05) indicate a feature that is \"significant\" in its ability to predict the dependent variable. Those closer, but not equal to, 0 are less significant. It depends on what you did with your \"bad values\", but list below the top three features and the overall Adjusted R-squared using all the features." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.2 A \"reduced\" model using only the \"significant\" features\n", "\n", "Modeling data is as much a craft as it is a science. We often seek the simplest models that explain or data well because they are typically more interpretable, easier to explain, and provide the information on the main influences of the system we are studying. There are reasons we might want a more complex model to capture the details and the nuance of the system. But for the U.N.E.S.C.O. data that we have, we are likely able to capture most of the system using a smaller number of features. _These ideas are related to the pre-class modeling you did with increasingly higher powers of `x`._\n", "\n", "✎ Do This - Redo the model with only the top three features you found above vs \"Infant Mortality\". Print the summary." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Review this model and the one you constructed earlier in the notebook. Reporthow the Adjusted R-squared value changed from using only the top three vs using all the available features. How well does this reduced model appear to fit your data?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Visualization - How well does our model fit our data?\n", "\n", "We have been checking how our models fit our data using both plots of the fitted values and the residuals. These plots are generated from the information stored in [various attributes of the OLS results object](https://www.statsmodels.org/stable/generated/statsmodels.regression.linear_model.OLSResults.html). We will continue to use the top two plots from `.graphics.plot_regress_exog` to investigate our fits. But you could also construct the plots directly using the attributes of the OLD results object. Note that you will need one plot for each feature in the model as each figure is only produced for a given choice of feature.\n", "\n", "✎ Do This - Create three `.graphics.plot_regress_exog` figures, one for each of the features in your model. Pay special attention to the top two plots: the fitted values figure and the residual plot." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Based on these figures, how well does it appear your reduced model fit your data? Do you have any concerns about the distribution of the residuals?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n", "### Congratulations, you're done with your in-class assignment!\n", "\n", "Now, you just need to submit this assignment by uploading it to the course Desire2Learn web page for today's submission folder (Don't forget to add your names in the first cell)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }