{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "###

✅ Put your name here.

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Homework Assignment #2 (Individual)\n", "## Modeling the World Happiness Index" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Goals for this homework assignment\n", "By the end of this assignment, you should be able to:\n", "* Debug Python code.\n", "* Get more experience with Pandas using excel spread sheets.\n", "* Make plots of your model using matplotlib.\n", "* Practice writing and executing a Python script.\n", "\n", "### Assignment instructions\n", "\n", "Work through the following assignment, making sure to follow all of the directions and answer all of the questions.\n", "\n", "There are **25 points** possible on this assignment. Point values for each part are included in the section headers.\n", "\n", "**This assignment is due at 11:59 pm on Friday, October 9th.** It should be uploaded into the \"Homework Assignments\" submission folder for Homework #2 in your D2L webpage. Submission instructions can be found at the end of the notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 1: Reading, Describing, and Cleaning the Data (8 total points)\n", "Before we read in the data and begin working with it, let's import the libraries that we would typically use for this task. You can always come back to this cell and import additional libraries that you need." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import statsmodels.api as sm\n", "import seaborn as sea" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The United Nations compiles an annual publication called the [World Happiness Report](https://en.wikipedia.org/wiki/World_Happiness_Report) that ranks countries by the \"happiness score\" or \"life ladder\". These scores are based on a surveys of citizens of each country on a variety of factors such as:\n", "* Gross Domestic Product (GDP) per capita\n", "* Perceptions of social support\n", "* Life expectancy\n", "* Freedom of choice\n", "\n", "[Data from the UN](https://worldhappiness.report/) is available from 2005-2019, which we have downloaded in a `.csv` format. The survey has changed over time, so that some factors and questions have been omitted. So some of the data is incomplete or missing.\n", "\n", "### 1.1 Read the data (2 points)\n", "\n", "✎ Do This - Read in the survey data from `happiness_index.csv` into a DataFrame and print the `.head()` of the data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2 Clean the data (3 total points)\n", "\n", "#### 1.2.1 Remove and track excess data (1 point)\n", "\n", "In order to build a linear regression model to predict the happiness score (`Life Ladder`), we need to remove the excess features that are not present in every year's data. \n", "\n", "✎ Do This - Drop the columns that are not the following:\n", "* Country name\n", "* year\n", "* Life Ladder\n", "* Log GDP per capita\n", "* Social support\n", "* Healthy life expectancy at birth\n", "* Freedom to make life choices\n", "* Generosity\n", "* Perceptions of corruption\n", "* Positive affect\n", "* Negative affect\n", "* Confidence in national government\n", "* Democratic Quality\n", "* Delivery Quality\n", "\n", "Make sure to make a list of the columns that you dropped. This is important if you need to report what you didn't consider in the model. Print the `.head()` of the DataFrame after dropping the unneeded columns." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 1.2.2 Focus on the 2018 data (1 point)\n", "\n", "Your data should include scores for a variety of years (2005-2019) for each country. Let's focus on a single year: 2018. \n", "\n", "✎ Do This - Filter your data to focus only on the latest available data from 2018. Print the `.head()` of that DataFrame." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 1.2.3 Missing data (1 point)\n", "\n", "Not all the countries have reported values for every feature. You should check if any countries have missing data. We could impute these missing data in a variety of ways, but instead to make our work a little simpler, we will simply drop a country if it has any missing data. However, we should keep track of any coutnries we drop to know which countries are excluded from the model.\n", "\n", "✎ Do This - Drop any country from the data that is missing data. Keep track of any countries you dropped from the data set (e.g., by printing or making a list)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.3 Describe the data (3 points)\n", "\n", "Now that you have a cleaned data set, let's look into the data. We can start that by describing the data and making histograms of it. \n", "\n", "✎ Do This - Focusing on the data collected in the survey (i.e., not the country name or year), describe the data and make histograms describing the distributions of the various features." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Making a model (9 total points)\n", "\n", "### 2.1 The correlation matrix (2 points total)\n", "\n", "To get our model started, we need to first investigate the correlations between the variables. \n", "\n", "#### 2.1.1 Visualizing the correlation matrix (1 point)\n", "Just the matrix isn't as useful as a visualization like a heat map of the correlation matrix.\n", "\n", "✎ Do This - For the variables in the survey (i.e., not the country and year), create a heat map that represents the correlation matrix. Answer the questions below as well." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.1.2 Determining which features will matter (1 point)\n", "\n", "Look at your correlation matrix visualization, specifically at how the variables correlate with `Life Ladder`. Which variables do you expect to be most important to your linear model? How can you tell from the correlation matrix (be specific)?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.2 Constructing and evaluating the full model (7 points total)\n", "\n", "We will start modeling our data by predicting the `Life Ladder` score based on all the remaining features. We will also evaluate the model using a variety of techniques.\n", "\n", "#### 2.2.1 Build and summarize the model (2 points)\n", "\n", "✎ Do This - Construct a linear OLS regression model that predicts the `Life Ladder` score from the other features. Print the `.summary()` of the model. Make sure to answer the questions below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.2.2 Evaluating model fit and determining important features (1 point)\n", "\n", "Based on this full model, how well do the data fit a linear model? How can you tell? It is typical to use the threshold of $p<0.05$ to determine features that are important for a model. Which features appear to be most important to the linear model? How does this determination fit with your original determination from the correlation matrix (it's ok if the do not match exactly)?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.2.3 Evaluating fit using predicted values (2 points)\n", "\n", "Another way to determine how well your model fits your data is to compare the predicted values and the actual values. If the model were perfect ($R^2 =1$), every predicted value and actual value would match exactly. This is generally not the case.\n", "\n", "✎ Do This - Plot the predicted values against the actual values of `Life Ladder`, include a line that would correspond to a perfect fit. Make sure to answer the questions below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "How well do the predicted values of your model compare to the actual values? Does your model overpredict or underpredict `Life Ladder` scores for any range of scores? Are there any outliers in the predicted values or actual data?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.2.4 Determining feature relationships to the model (3 points)\n", "\n", "Given the model, we should look into how the `Life Ladder` score changes based on each feature. This will help us better understand the model.\n", "\n", "✎ Do This - Create a scatter plot of the actual and predicted `Life Ladder` scores vs each feature in your model. Be sure to answer the questions below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The coefficients in the linear model for each feature (i.e., in the `.summary()`) describe the model's slope contribution from each feature. Using the graphs you created, describe qualitatively how each coefficient in the model that you created fits with the plots. That is, do you see the linear relationships for each feature that you expect given the coefficients?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Creating a reduced model (8 total points)\n", "\n", "From the work you have done above, it is probably clear that some features aren't particularly important for makng a linear model.\n", "\n", "### 3.1 Selecting a reduced set of features (2 points)\n", "\n", "Based on your analysis above, indicate which features you would drop from a reduced model. Explain why you would drop each of these features using the evidence for your analysis in the above section." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.2 Creating a reduced model (6 points)\n", "\n", "Now that you have determined which features you will exclude from your reduced model, let's construct that reduced model.\n", "\n", "✎ Do This - Create the reduced linear OLS model using only the subset of features that you indicated above. Print the summary of the results of the model fit. Be sure to answer the questions below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2 points\n", "Based on these results, compare the fit of this reduced model to the full model. What evidence are you using to make this comparison?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 3.3 Continuing to evaluate the reduced model (4 points)\n", "\n", "While we have used the summary to evaluate the model, we should do a deeper investigation into the model fit.\n", "\n", "✎ Do This - Using the techniques described in Section 2, or other methods of your choosing, evaluate the model fit using **at least** two additional methods. Be sure to answer the questions below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Based on your analysis above, how does the reduced model compare to the full model? Is the reduced model believable? \n", "\n", "Summarize the steps that you used to build and evaluate the quality of the reduced model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "### Assignment wrap-up\n", "\n", "Please fill out the form that appears when you run the code below. **You must completely fill this out in order to receive credit for the assignment!**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.display import HTML\n", "HTML(\n", "\"\"\"\n", "\n", "\"\"\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Congratulations, you're done!\n", "\n", "Submit this assignment by uploading it to the course Desire2Learn web page. Go to the \"Homework Assignments\" folder, find the dropbox link for Homework #2, and upload it there." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "© Copyright Michigan State University Board of Trustees" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }