{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "###

✅ Put your name here

\n", "####

✅ Put your group member names here

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 3 In-Class Assignment: Datasest\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Goals for today's In-class assignment \n", "\n", "1. Make sure that everyone remembers how to do the basics with Pandas.\n", "2. Do a little data analysis with some existing data sets.\n", "3. Make some simple visualizations of the data.\n", "\n", "### Assignment instructions\n", "\n", "Work through the notebook making sure to write all necessary code and answer any questions.\n", "\n", "**This assignment is due by the end of class,** and should be uploaded into the \"In-Class assignments\" submission folder for Day 3. Submission instructions can be found at the end of the notebook.\n", "\n", "## Agenda for today:\n", "\n", "1. Review of questions about pre-class assignment\n", "2. Load a dataset and describe it\n", "3. Plot the data set\n", "4. Assignment wrap-up" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# our imports\n", "%matplotlib inline\n", "import pandas as pd\n", "from pandas.plotting import scatter_matrix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "# 1. Review of pre-class assignment\n", "\n", "Did anyone have any specific issues with the pre-class assignment?\n", "\n", "Let's take a moment to highlight some key concepts. Discuss with your group the following prompts and write down a brief definition of each of these concepts.\n", "\n", "* What are the features of a Pandas Series?\n", "\n", "✎ Do This - Write your disussion notes here.\n", "\n", "* What are the differences between a DataFrame and a Series?\n", "\n", "✎ Do This - Write your disussion notes here.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "# 2. Loading a Data set\n", "\n", "Though we made a DataFrame by hand, the goal is typically to read some data **into** a data frame so we can work with it. \n", "\n", "Pandas is pretty flexible about reading in data. However, it sometimes needs a little help. Let's start with a toy data set, the Iris Data set . \n", "\n", "We will work with what are called **toy** datasets to try and work through some of the foundational issues. But **data wrangling**, working with data that is messy is far more the norm than you might think. So we will ask you to do some wrangling along the way, the kind you would probably have to do in actual applications.\n", "\n", "## 2.1 Learn to look\n", "\n", "It is always going to be the case that you are going to have to look things up. The Python data analysis/modeling world if very complicated with a large number or reasonably complicated toolsets, all with their own working parts. We become better Data Science folks by learning how to search for better, more efficient, clearer ways to do things. That comes with experience but also with a healthy ability to look around and figure things out. \n", "\n", "### Iris\n", "\n", "The iris data set is pretty straight forward (follow the link above). To make it a little more interesting we provide it in a raw form that you might encounter with other data. In the class directory you will find two files: `iris.data` and `iris.names`. Open them both and discuss what you see. If you don't have your own favorite editor, you can use the `Spyder` editor that comes with Anaconda (though you might have to selected **All files** in the pull down menu of file types to open).\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 1** Describe the data and the information in the names file. What characteristics are provided there? Perhaps the iris link above will help." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Erase the contents of this cell and replace it with your answer to the above question! (double-click on this text to edit this cell, and hit shift+enter to save the text). Please use appropriate Markup formating to make your answer's easy to read." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.2 Reading in a file\n", "\n", "Pandas supports a number of file formats, but one of the most common is the one provided by the `read_csv` function. Typically, the phrase **csv** stands for \"comma separated values\", a text file format that excel and other spread sheets use. In a csv file, each line represents a line in a spreadsheet and each cell value is separated from the next by a comma.\n", "\n", "However, you can use other separators as well. Look at the documentation for read_csv " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Read the ```iris.data``` file in with the appropriate headers." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# replace this cell with code to read in the iris.data file with column headers. Use appropriate headers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.3 Describing your data\n", "\n", "The `.describe()` methods tells you about the data. If the data is too big to see (height and width) you can use the two options below to show more of the data. You can change the values to suit your needs. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# expand what is shown (rows and cols) by pandas\n", "pd.set_option('display.max_rows', 500)\n", "pd.set_option('display.max_columns', 500)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Run the `.describe` method on the read-in data. With an eye towards classifying the species of input based on the four features, do you discern anything helpful? Answer in the cell below" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# use describe here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do this - Answer here: Did you find any useful features? What makes those features useful in classfying the species of the irises?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.4 Grouping your data\n", "You can perform operations to group elements using the `.groupby` method. The result of the invocation of `.groupby` is a `GroupBy` object. If you were to print it you see only the class instance name as nothing is computed until an operation is performed on the groups, much like other iterators. However, you can `.describe` the result of a groupby. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - make a group using the iris data frame and groupby the class/species of the data. Then run `.describe` on the group result. Now take a look at the data from the group. Do you see any data features you could use to classify the data?" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# print out your groupby here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Answer here: Did you find any useful features? What makes those features useful in classfying the species of the irises? How did using ```groupby()``` help (or not) in finding useful features?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n", "## 3 Visualizing your data\n", "\n", "Many time we are limited in understanding our data because it is complex, has many features, or is quite large. So many visualizations (plots, charts, and graphs) can help represent our data in ways that help us gain greater insight into the trends, features, and classes we want to understand." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.1 Making a scatter matrix\n", "\n", "It would be better if we were to visualize the differences. Since the number of features is small, we can use a plotting feature of pandas called a `scatter_matrix`. We imported `scatter_matrix` at the start of this notebook so call it on our dataframe. Use a `marker` that you like. Look up some details on `scatter_matrix` for more information." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Answer here: Does this visualization help you find any useful features? We will do color-coding of classes next, but are there other possible ideas you have that might help you discern useful features?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.2 Color coding your data\n", "\n", "That probably isn't as helpful as we'd like. We need a separate color indication for each dot so we can tell things apart. Essentially, we need to create an array such that, for each observation in the DataFrame, we a value of color associated with that species. It could be just an integer, or it could be one of the standard colors such as \"red\". Then redo the scatter matrix using `c=your_color_array` to give us different colors. Might want to make it a little bigger as well so you can see it. " ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "OK, how do we read this now?\n", "* The diagonal shows the distribution of the four numeric variables of our example data.\n", "* In the other cells of the plot matrix, we have the scatterplots (i.e. correlation plot) of each variable combination of our dataframe. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Are you better abel to discern features that might be useful in classfying irises? What can you say about the features of each iris species? Can you separate one species easily? If so, using which feature(s)?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3 Separating species of irises" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will use the feature(s) you found above to try to separate iris species. Later we will learn how to do this using models, but for now, you will try to do this by slicing the data frame." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - One of these species is obviously easier to separate than the others. Can you write a Boolean expression on the dataframe and isolate that one species? " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Can you write a series of Boolean expressions that separate the other two? Are these as effective as the first? Answer in the cell below what you might do to improve separation?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here: How effective was just Boolean separation of the 3 species? How could you do better?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---------\n", "### Congratulations, you're done with your in-class assignment!\n", "\n", "Now, you just need to submit this assignment by uploading it to the course Desire2Learn web page for today's in-class assignment submission folder. (Don't forget to add all of the appropriate names in the first cell)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "© Copyright Michigan State University Board of Trustees" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }