✅ Put your name here
\n", "####✅ Put your group member names here
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 04 In-Class Assignment: Analyzing Covid Data\n", "\n", "This notebook is an modified version of one authored by [Brian O'Shea](https://web.pa.msu.edu/people/osheabr/) at CMSE in the early part of the COVID-19 epidemic.\n", "\n", "## Goals for today's In-class assignment \n", "\n", "1. Read JSON data into a Pandas DataFrame\n", "2. Perform calculations with the `datetime` class\n", "3. Read COVID data from the web into a Pandas DataFrame\n", "4. Calculate the Doubling Time for the virus\n", "5. Represent that data in plots and discuss meaningful representations of data\n", "\n", "## Information on the data we will use\n", "\n", "This notebook uses Pandas to retrieve JSON data files created by the [Johns Hopkins Coronavirus Resource Center](https://coronavirus.jhu.edu) (and stored at [The COVID Tracking Project](https://covidtracking.com/)) to make real-time tracking plots of the growth of the COVID-19 pandemic in the US and Michigan, as well as estimate the doubling time of the number of positive cases. These data files are updated daily, so re-running the notebook will get the most current data. It's easy to change the state that is plotted by changing the `my_state` variable. There's also quite a bit more information in the data files!\n", "\n", "**Important note:** The data file tracks the number of people who have _tested positive_ of COVID in the US and Michigan, not an estimate of the total number of positive cases in the US and Michigan." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The imports " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import math\n", "import datetime" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## 1. Working with JSON data\n", "\n", "It is a bit of a story to explain what JSON (pronounced \"J\" \"son\") data is but it is worth our time. To understand JSON you have to know a little bit about JavaScript. If you read the wikipedia page you will read:\n", "\n", "> Alongside HTML and CSS, JavaScript is one of the core technologies of the World Wide Web.[8] JavaScript enables interactive web pages and is an essential part of web applications. The vast majority of websites use it for client-side page behavior,[9] and all major web browsers have a dedicated JavaScript engine to execute it. \n", "\n", "So the fact that modern browsers have a dynamic character (pages change based on all kinds of inputs) is largely due to an underlying programming language called JavaScript. \n", "\n", "One problem is that it is likely the case that you want your web server (not on your machine) to dynamically generate content, to be rendered using JavaScript. How to get the dynamic content from the server to your local machine? The exchange of information has to use text to be universal. How do you translate existing JavaScript objects into text and back? You guessed it, that is what JSON is.\n", "\n", "From wikipedia JSON page:\n", "> JavaScript Object Notation (JSON) is an open standard file format, and data interchange format, that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value).\n", "\n", "> JSON is a language-independent data format. It was derived from JavaScript, but many modern programming languages include code to generate and parse JSON-format data. The official Internet media type for JSON is application/json. JSON filenames use the extension .json. \n", "\n", "The term `serializable` is worth noting. By this we mean that actual object instances, with all their variable settings and methods, can be stored into a format and then re-created \"as they were\" in another location. That is, the `state` of a system can be stored and then re-created at a later date. Note that JSON was originally set up for JavaScript, but as a \"state saver\" can be used in all kinds of place. Like Python!\n", "\n", "Bottom line: JSON is a way to exchange information across the internet. We are going to use that!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.1 DataFrame to JSON and back\n", "\n", "The `.read_json` and `.to_json` allow us to read and write a DataFrame to JSON. Let's do that:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "my_dict = { \"tens\": [10, 11, 12, 13, 14, 15],\n", " \"twentys\" : [20, 21, 22, 23, 24, 25],\n", " \"thirtys\" : [30, 31, 32, 33, 34, 35]\n", " }" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Do the following:\n", "* turn that dictionary into a DataFrame\n", "* print the DataFrame\n", "* write out the DataFrame as a JSON file (look up `to_json`)\n", "* take a look at that file with your editor\n", "* read that JSON file back into a DataFrame with a different name (look up `read_json`)\n", "* print out the new DataFrame" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2 Read JSON over the internet\n", "\n", "But here is the really cool stuff. You don't have to have a JSON file on your local machine to read it in. You can actually read it from a source on the internet and use it locally. Remember, JSON is an exchange file format so that makes perfect sense!" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " sepalLength sepalWidth petalLength petalWidth species\n", "0 5.1 3.5 1.4 0.2 setosa\n", "1 4.9 3.0 1.4 0.2 setosa\n", "2 4.7 3.2 1.3 0.2 setosa\n", "3 4.6 3.1 1.5 0.2 setosa\n", "4 5.0 3.6 1.4 0.2 setosa\n", ".. ... ... ... ... ...\n", "145 6.7 3.0 5.2 2.3 virginica\n", "146 6.3 2.5 5.0 1.9 virginica\n", "147 6.5 3.0 5.2 2.0 virginica\n", "148 6.2 3.4 5.4 2.3 virginica\n", "149 5.9 3.0 5.1 1.8 virginica\n", "\n", "[150 rows x 5 columns]\n" ] } ], "source": [ "iris_df = pd.read_json(\"https://raw.githubusercontent.com/domoritz/maps/master/data/iris.json\")\n", "print(iris_df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ah, it's our favorite iris dataset but read in from a file on the internet. Open that url once in your browser. Should look pretty understandable at this point." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n", "## 2. Analyzing COVID data\n", "\n", "You must be living under a rock to not know about tracking the COVID-19 virus outbreak. Let's see if we can do some data analysis of a time-series, JSON file. \n", "\n", "Gets the data from [covidtracking.com](https://covidtracking.com) - specifically, the page [https://covidtracking.com/api](https://covidtracking.com/api). This page has historical data for the US as a whole, for states, and for counties. The JSON files have a ton of data in addition to what we're showing. The two URLs we are most interested in are:\n", "* https://covidtracking.com/api/us/daily which is the US daily report as a JSON file\n", "* https://covidtracking.com/api/v1/states/daily.json which is the data for all 50 states.\n", "\n", "Before you do anything else, you should open those two links in your browser and look at the data. JSON is pretty readable and you need to read a little about what's there so we can do some more work." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Do the following:\n", "* read the US daily into a DataFrame\n", "* read the state data into a DataFrame\n", " * make a DataFrame that contains just the state of Michigan from the state data\n", "* print the `.head` of the US and MI DataFrames." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1 Convert data to DateTime\n", "\n", "There's a date field in the data but it is not in pandas datetime format. That would be helpful since this is a time series. \n", "\n", "✎ Do This - Let's do what we did before:\n", "* Let's make a Series from the `date` field using `to_datetime`. \n", "Look at the format of the date of the JSON file in your browser and write the appropriate format specifier to get it into datetime format.\n", "* assign the Series to the DataFrame under the name \"datetime\"\n", "* make a new DataFrame with all the positive cases in the US along with the datatime (just the two columns)\n", "* print the head of the 2-column, positive US cases\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Do the same for the Michigan State Data. \n", "\n", "✎ Do This - Make a two column DataFrame with datetime and positives." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "#your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.2 Plot cumulative positive tests for US and MI\n", "\n", "Again, this data is large and thus we need to represent in a way that makes it understandable to us.\n", "\n", "✎ Do This - Using the `.plot` method, make two plots: one for the MI positive cases and one for US positive cases. " ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "