✅ Put your name here
\n", "####✅ Put your group member names here
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 04 In-Class Assignment: Analyzing Covid Data\n", "\n", "This notebook is an modified version of one authored by [Brian O'Shea](https://web.pa.msu.edu/people/osheabr/) at CMSE in the early part of the COVID-19 epidemic.\n", "\n", "## Goals for today's In-class assignment \n", "\n", "1. Read JSON data into a Pandas DataFrame\n", "2. Perform calculations with the `datetime` class\n", "3. Read COVID data from the web into a Pandas DataFrame\n", "4. Calculate the Doubling Time for the virus\n", "5. Represent that data in plots and discuss meaningful representations of data\n", "\n", "## Information on the data we will use\n", "\n", "This notebook uses Pandas to retrieve JSON data files created by the [Johns Hopkins Coronavirus Resource Center](https://coronavirus.jhu.edu) (and stored at [The COVID Tracking Project](https://covidtracking.com/)) to make real-time tracking plots of the growth of the COVID-19 pandemic in the US and Michigan, as well as estimate the doubling time of the number of positive cases. These data files are updated daily, so re-running the notebook will get the most current data. It's easy to change the state that is plotted by changing the `my_state` variable. There's also quite a bit more information in the data files!\n", "\n", "**Important note:** The data file tracks the number of people who have _tested positive_ of COVID in the US and Michigan, not an estimate of the total number of positive cases in the US and Michigan." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The imports " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import math\n", "import datetime" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## 1. Working with JSON data\n", "\n", "It is a bit of a story to explain what JSON (pronounced \"J\" \"son\") data is but it is worth our time. To understand JSON you have to know a little bit about JavaScript. If you read the wikipedia page you will read:\n", "\n", "> Alongside HTML and CSS, JavaScript is one of the core technologies of the World Wide Web.[8] JavaScript enables interactive web pages and is an essential part of web applications. The vast majority of websites use it for client-side page behavior,[9] and all major web browsers have a dedicated JavaScript engine to execute it. \n", "\n", "So the fact that modern browsers have a dynamic character (pages change based on all kinds of inputs) is largely due to an underlying programming language called JavaScript. \n", "\n", "One problem is that it is likely the case that you want your web server (not on your machine) to dynamically generate content, to be rendered using JavaScript. How to get the dynamic content from the server to your local machine? The exchange of information has to use text to be universal. How do you translate existing JavaScript objects into text and back? You guessed it, that is what JSON is.\n", "\n", "From wikipedia JSON page:\n", "> JavaScript Object Notation (JSON) is an open standard file format, and data interchange format, that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value).\n", "\n", "> JSON is a language-independent data format. It was derived from JavaScript, but many modern programming languages include code to generate and parse JSON-format data. The official Internet media type for JSON is application/json. JSON filenames use the extension .json. \n", "\n", "The term `serializable` is worth noting. By this we mean that actual object instances, with all their variable settings and methods, can be stored into a format and then re-created \"as they were\" in another location. That is, the `state` of a system can be stored and then re-created at a later date. Note that JSON was originally set up for JavaScript, but as a \"state saver\" can be used in all kinds of place. Like Python!\n", "\n", "Bottom line: JSON is a way to exchange information across the internet. We are going to use that!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.1 DataFrame to JSON and back\n", "\n", "The `.read_json` and `.to_json` allow us to read and write a DataFrame to JSON. Let's do that:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "my_dict = { \"tens\": [10, 11, 12, 13, 14, 15],\n", " \"twentys\" : [20, 21, 22, 23, 24, 25],\n", " \"thirtys\" : [30, 31, 32, 33, 34, 35]\n", " }" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Do the following:\n", "* turn that dictionary into a DataFrame\n", "* print the DataFrame\n", "* write out the DataFrame as a JSON file (look up `to_json`)\n", "* take a look at that file with your editor\n", "* read that JSON file back into a DataFrame with a different name (look up `read_json`)\n", "* print out the new DataFrame" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " tens twentys thirtys\n", "0 10 20 30\n", "1 11 21 31\n", "2 12 22 32\n", "3 13 23 33\n", "4 14 24 34\n", "5 15 25 35\n", " tens twentys thirtys\n", "0 10 20 30\n", "1 11 21 31\n", "2 12 22 32\n", "3 13 23 33\n", "4 14 24 34\n", "5 15 25 35\n" ] } ], "source": [ "### ANSWER ###\n", "\n", "df1 = pd.DataFrame(my_dict)\n", "print(df1)\n", "df1.to_json(\"temp.json\")\n", "df2 = pd.read_json(\"temp.json\")\n", "print(df2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2 Read JSON over the internet\n", "\n", "But here is the really cool stuff. You don't have to have a JSON file on your local machine to read it in. You can actually read it from a source on the internet and use it locally. Remember, JSON is an exchange file format so that makes perfect sense!" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " sepalLength sepalWidth petalLength petalWidth species\n", "0 5.1 3.5 1.4 0.2 setosa\n", "1 4.9 3.0 1.4 0.2 setosa\n", "2 4.7 3.2 1.3 0.2 setosa\n", "3 4.6 3.1 1.5 0.2 setosa\n", "4 5.0 3.6 1.4 0.2 setosa\n", ".. ... ... ... ... ...\n", "145 6.7 3.0 5.2 2.3 virginica\n", "146 6.3 2.5 5.0 1.9 virginica\n", "147 6.5 3.0 5.2 2.0 virginica\n", "148 6.2 3.4 5.4 2.3 virginica\n", "149 5.9 3.0 5.1 1.8 virginica\n", "\n", "[150 rows x 5 columns]\n" ] } ], "source": [ "iris_df = pd.read_json(\"https://raw.githubusercontent.com/domoritz/maps/master/data/iris.json\")\n", "print(iris_df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ah, it's our favorite iris dataset but read in from a file on the internet. Open that url once in your browser. Should look pretty understandable at this point." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n", "## 2. Analyzing COVID data\n", "\n", "You must be living under a rock to not know about tracking the COVID-19 virus outbreak. Let's see if we can do some data analysis of a time-series, JSON file. \n", "\n", "Gets the data from [covidtracking.com](https://covidtracking.com) - specifically, the page [https://covidtracking.com/api](https://covidtracking.com/api). This page has historical data for the US as a whole, for states, and for counties. The JSON files have a ton of data in addition to what we're showing. The two URLs we are most interested in are:\n", "* https://covidtracking.com/api/us/daily which is the US daily report as a JSON file\n", "* https://covidtracking.com/api/v1/states/daily.json which is the data for all 50 states.\n", "\n", "Before you do anything else, you should open those two links in your browser and look at the data. JSON is pretty readable and you need to read a little about what's there so we can do some more work." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do This - Do the following:\n", "* read the US daily into a DataFrame\n", "* read the state data into a DataFrame\n", " * make a DataFrame that contains just the state of Michigan from the state data\n", "* print the `.head` of the US and MI DataFrames." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", " ALL US \n", " date states positive negative pending hospitalizedCurrently \\\n", "0 20200909 56 6329405 77613114 6551.0 32562.0 \n", "1 20200908 56 6298422 77059685 8983.0 32263.0 \n", "2 20200907 56 6276203 76573198 8986.0 31999.0 \n", "3 20200906 56 6247522 76013582 8946.0 32361.0 \n", "4 20200905 56 6214407 75310873 9021.0 33502.0 \n", "\n", " hospitalizedCumulative inIcuCurrently inIcuCumulative \\\n", "0 381926.0 6646.0 18322.0 \n", "1 379866.0 6542.0 18189.0 \n", "2 378907.0 6630.0 18156.0 \n", "3 378415.0 6699.0 18125.0 \n", "4 377926.0 6766.0 18089.0 \n", "\n", " onVentilatorCurrently ... lastModified total \\\n", "0 1903.0 ... 2020-09-09T00:00:00Z 83949070 \n", "1 1879.0 ... 2020-09-08T00:00:00Z 83367090 \n", "2 1879.0 ... 2020-09-07T00:00:00Z 82858387 \n", "3 1912.0 ... 2020-09-06T00:00:00Z 82270050 \n", "4 1956.0 ... 2020-09-05T00:00:00Z 81534301 \n", "\n", " totalTestResults posNeg deathIncrease hospitalizedIncrease \\\n", "0 83942519 83942519 1089 2060 \n", "1 83358107 83358107 358 959 \n", "2 82849401 82849401 225 492 \n", "3 82261104 82261104 449 489 \n", "4 81525280 81525280 918 1287 \n", "\n", " negativeIncrease positiveIncrease totalTestResultsIncrease \\\n", "0 553429 30983 584412 \n", "1 486487 22219 508706 \n", "2 559616 28681 588297 \n", "3 702709 33115 735824 \n", "4 761056 44910 805966 \n", "\n", " hash \n", "0 e5c534217742be6f09560d25616f96c0d1915b4c \n", "1 c43bc49c6f42a78c85c704d97acdf1044d89bc3b \n", "2 23ba2e4ce89a07512fa56cc86442f9da440d644c \n", "3 b4363094c4b840e8a8660bf8213749232fae73db \n", "4 5a3e0e97caa4d2d9536e2ec55bc6124938a6b430 \n", "\n", "[5 rows x 25 columns]\n", "\n", " Michigan \n", " date state positive negative pending hospitalizedCurrently \\\n", "24 20200909 MI 119863.0 2883414.0 NaN 616.0 \n", "80 20200908 MI 118902.0 2861182.0 NaN 616.0 \n", "136 20200907 MI 118403.0 2844275.0 NaN 616.0 \n", "192 20200906 MI 117191.0 2801811.0 NaN 616.0 \n", "248 20200905 MI 117191.0 2801811.0 NaN 616.0 \n", "\n", " hospitalizedCumulative inIcuCurrently inIcuCumulative \\\n", "24 NaN 158.0 NaN \n", "80 NaN 158.0 NaN \n", "136 NaN 158.0 NaN \n", "192 NaN 158.0 NaN \n", "248 NaN 158.0 NaN \n", "\n", " onVentilatorCurrently ... posNeg deathIncrease hospitalizedIncrease \\\n", "24 70.0 ... 3003277 76 0 \n", "80 76.0 ... 2980084 1 0 \n", "136 71.0 ... 2962678 4 0 \n", "192 71.0 ... 2919002 0 0 \n", "248 71.0 ... 2919002 8 0 \n", "\n", " hash commercialScore \\\n", "24 da7d758f7b17c69f974d0a8ad53647858950d47a 0 \n", "80 801f7d8795e74bece92bae3adff400cd245cb1bf 0 \n", "136 565ca9242bc12540e899aa405fe0e5dbfa627fb3 0 \n", "192 f1437d5a6117d5b7364470917e6d6b5310f319fe 0 \n", "248 ea5cefa02687fc1df1ba2a8b835240b119e42a2f 0 \n", "\n", " negativeRegularScore negativeScore positiveScore score grade \n", "24 0 0 0 0 \n", "80 0 0 0 0 \n", "136 0 0 0 0 \n", "192 0 0 0 0 \n", "248 0 0 0 0 \n", "\n", "[5 rows x 54 columns]\n" ] } ], "source": [ "### ANSWER ###\n", "\n", "us_df = pd.read_json(\"https://covidtracking.com/api/us/daily\")\n", "all_states_df = pd.read_json(\"https://covidtracking.com/api/v1/states/daily.json\")\n", "\n", "# select out MI\n", "MI_df = all_states_df[all_states_df['state']==\"MI\"]\n", "print('\\n ALL US \\n', us_df.head())\n", "print('\\n Michigan \\n', MI_df.head())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1 Convert data to DateTime\n", "\n", "There's a date field in the data but it is not in pandas datetime format. That would be helpful since this is a time series. \n", "\n", "✎ Do This - Let's do what we did before:\n", "* Let's make a Series from the `date` field using `to_datetime`. \n", "Look at the format of the date of the JSON file in your browser and write the appropriate format specifier to get it into datetime format.\n", "* assign the Series to the DataFrame under the name \"datetime\"\n", "* make a new DataFrame with all the positive cases in the US along with the datatime (just the two columns)\n", "* print the head of the 2-column, positive US cases\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 2020-09-09\n", "1 2020-09-08\n", "2 2020-09-07\n", "3 2020-09-06\n", "4 2020-09-05\n", " ... \n", "227 2020-01-26\n", "228 2020-01-25\n", "229 2020-01-24\n", "230 2020-01-23\n", "231 2020-01-22\n", "Name: date, Length: 232, dtype: datetime64[ns]\n" ] }, { "data": { "text/html": [ "| \n", " | positive | \n", "datetime | \n", "
|---|---|---|
| 0 | \n", "6329405 | \n", "2020-09-09 | \n", "
| 1 | \n", "6298422 | \n", "2020-09-08 | \n", "
| 2 | \n", "6276203 | \n", "2020-09-07 | \n", "
| 3 | \n", "6247522 | \n", "2020-09-06 | \n", "
| 4 | \n", "6214407 | \n", "2020-09-05 | \n", "
| \n", " | positive | \n", "datetime | \n", "
|---|---|---|
| 24 | \n", "119863.0 | \n", "2020-09-09 | \n", "
| 80 | \n", "118902.0 | \n", "2020-09-08 | \n", "
| 136 | \n", "118403.0 | \n", "2020-09-07 | \n", "
| 192 | \n", "117191.0 | \n", "2020-09-06 | \n", "
| 248 | \n", "117191.0 | \n", "2020-09-05 | \n", "