✅ Put your name here
\n", "####✅ Put your group member names here
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 3 In-Class Assignment: Datasest\n", "\n", "![](\"https://miro.medium.com/max/819/1*Dss7A8Z-M4x8LD9ccgw7pQ.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Goals for today's In-class assignment \n",
"\n",
"1. Make sure that everyone remembers how to do the basics with Pandas.\n",
"2. Do a little data analysis with some existing data sets.\n",
"3. Make some simple visualizations of the data.\n",
"\n",
"### Assignment instructions\n",
"\n",
"Work through the notebook making sure to write all necessary code and answer any questions.\n",
"\n",
"**This assignment is due by the end of class,** and should be uploaded into the \"In-Class assignments\" submission folder for Day 3. Submission instructions can be found at the end of the notebook.\n",
"\n",
"## Agenda for today:\n",
"\n",
"1. Review of questions about pre-class assignment\n",
"2. Load a dataset and describe it\n",
"3. Plot the data set\n",
"4. Assignment wrap-up"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"# our imports\n",
"%matplotlib inline\n",
"import pandas as pd\n",
"from pandas.plotting import scatter_matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"----\n",
"# 1. Review of pre-class assignment\n",
"\n",
"Did anyone have any specific issues with the pre-class assignment?\n",
"\n",
"Let's take a moment to highlight some key concepts. Discuss with your group the following prompts and write down a brief definition of each of these concepts.\n",
"\n",
"* What are the features of a Pandas Series?\n",
"\n",
"✎ Do This - Write your disussion notes here.\n",
"\n",
"* What are the differences between a DataFrame and a Series?\n",
"\n",
"✎ Do This - Write your disussion notes here.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"# 2. Loading a Data set\n",
"\n",
"Though we made a DataFrame by hand, the goal is typically to read some data **into** a data frame so we can work with it. \n",
"\n",
"Pandas is pretty flexible about reading in data. However, it sometimes needs a little help. Let's start with a toy data set, the Iris Data set . \n",
"\n",
"We will work with what are called **toy** datasets to try and work through some of the foundational issues. But **data wrangling**, working with data that is messy is far more the norm than you might think. So we will ask you to do some wrangling along the way, the kind you would probably have to do in actual applications.\n",
"\n",
"## 2.1 Learn to look\n",
"\n",
"It is always going to be the case that you are going to have to look things up. The Python data analysis/modeling world if very complicated with a large number or reasonably complicated toolsets, all with their own working parts. We become better Data Science folks by learning how to search for better, more efficient, clearer ways to do things. That comes with experience but also with a healthy ability to look around and figure things out. \n",
"\n",
"### Iris\n",
"\n",
"The iris data set is pretty straight forward (follow the link above). To make it a little more interesting we provide it in a raw form that you might encounter with other data. In the class directory you will find two files: `iris.data` and `iris.names`. Open them both and discuss what you see. If you don't have your own favorite editor, you can use the `Spyder` editor that comes with Anaconda (though you might have to selected **All files** in the pull down menu of file types to open).\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question 1** Describe the data and the information in the names file. What characteristics are provided there? Perhaps the iris link above will help."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Do This - Erase the contents of this cell and replace it with your answer to the above question! (double-click on this text to edit this cell, and hit shift+enter to save the text). Please use appropriate Markup formating to make your answer's easy to read."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.2 Reading in a file\n",
"\n",
"Pandas supports a number of file formats, but one of the most common is the one provided by the `read_csv` function. Typically, the phrase **csv** stands for \"comma separated values\", a text file format that excel and other spread sheets use. In a csv file, each line represents a line in a spreadsheet and each cell value is separated from the next by a comma.\n",
"\n",
"However, you can use other separators as well. Look at the documentation for read_csv "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Do This - Read the ```iris.data``` file in with the appropriate headers."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# replace this cell with code to read in the iris.data file with column headers. Use appropriate headers"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"### ANSWER ###\n",
"\n",
"df = pd.read_csv(\"iris.data\", delim_whitespace=True, names=[\"sepal_l\", \"sepal_w\", \"petal_l\", \"petal_w\", \"species\"])\n",
"# df.set_index(\"species\", inplace=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.3 Describing your data\n",
"\n",
"The `.describe()` methods tells you about the data. If the data is too big to see (height and width) you can use the two options below to show more of the data. You can change the values to suit your needs. "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# expand what is shown (rows and cols) by pandas\n",
"pd.set_option('display.max_rows', 500)\n",
"pd.set_option('display.max_columns', 500)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Do This - Run the `.describe` method on the read-in data. With an eye towards classifying the species of input based on the four features, do you discern anything helpful? Answer in the cell below"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# use describe here"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Index(['sepal_l', 'sepal_w', 'petal_l', 'petal_w', 'species'], dtype='object')"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"### ANSWER ###\n",
"df.describe()\n",
"df.columns"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Do this - Answer here: Did you find any useful features? What makes those features useful in classfying the species of the irises?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.4 Grouping your data\n",
"You can perform operations to group elements using the `.groupby` method. The result of the invocation of `.groupby` is a `GroupBy` object. If you were to print it you see only the class instance name as nothing is computed until an operation is performed on the groups, much like other iterators. However, you can `.describe` the result of a groupby. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"✎ Do This - make a group using the iris data frame and groupby the class/species of the data. Then run `.describe` on the group result. Now take a look at the data from the group. Do you see any data features you could use to classify the data?"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# print out your groupby here"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | sepal_l | \n", "sepal_w | \n", "petal_l | \n", "petal_w | \n", "||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| \n", " | count | \n", "mean | \n", "std | \n", "min | \n", "25% | \n", "50% | \n", "75% | \n", "max | \n", "count | \n", "mean | \n", "std | \n", "min | \n", "25% | \n", "50% | \n", "75% | \n", "max | \n", "count | \n", "mean | \n", "std | \n", "min | \n", "25% | \n", "50% | \n", "75% | \n", "max | \n", "count | \n", "mean | \n", "std | \n", "min | \n", "25% | \n", "50% | \n", "75% | \n", "max | \n", "
| species | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| Iris-setosa | \n", "50.0 | \n", "5.006 | \n", "0.352490 | \n", "4.3 | \n", "4.800 | \n", "5.0 | \n", "5.2 | \n", "5.8 | \n", "50.0 | \n", "3.418 | \n", "0.381024 | \n", "2.3 | \n", "3.125 | \n", "3.4 | \n", "3.675 | \n", "4.4 | \n", "50.0 | \n", "1.464 | \n", "0.173511 | \n", "1.0 | \n", "1.4 | \n", "1.50 | \n", "1.575 | \n", "1.9 | \n", "50.0 | \n", "0.244 | \n", "0.107210 | \n", "0.1 | \n", "0.2 | \n", "0.2 | \n", "0.3 | \n", "0.6 | \n", "
| Iris-versicolor | \n", "50.0 | \n", "5.936 | \n", "0.516171 | \n", "4.9 | \n", "5.600 | \n", "5.9 | \n", "6.3 | \n", "7.0 | \n", "50.0 | \n", "2.770 | \n", "0.313798 | \n", "2.0 | \n", "2.525 | \n", "2.8 | \n", "3.000 | \n", "3.4 | \n", "50.0 | \n", "4.260 | \n", "0.469911 | \n", "3.0 | \n", "4.0 | \n", "4.35 | \n", "4.600 | \n", "5.1 | \n", "50.0 | \n", "1.326 | \n", "0.197753 | \n", "1.0 | \n", "1.2 | \n", "1.3 | \n", "1.5 | \n", "1.8 | \n", "
| Iris-virginica | \n", "50.0 | \n", "6.588 | \n", "0.635880 | \n", "4.9 | \n", "6.225 | \n", "6.5 | \n", "6.9 | \n", "7.9 | \n", "50.0 | \n", "2.974 | \n", "0.322497 | \n", "2.2 | \n", "2.800 | \n", "3.0 | \n", "3.175 | \n", "3.8 | \n", "50.0 | \n", "5.552 | \n", "0.551895 | \n", "4.5 | \n", "5.1 | \n", "5.55 | \n", "5.875 | \n", "6.9 | \n", "50.0 | \n", "2.026 | \n", "0.274650 | \n", "1.4 | \n", "1.8 | \n", "2.0 | \n", "2.3 | \n", "2.5 | \n", "