{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "###

✅ Put your name here

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 4 Pre-Class Assignment: Working Time-series Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Goals for today's pre-class assignment \n", "\n", "1. Learn the methods and types of the ```datetime``` class in the ```datetime``` Python module\n", "2. Creating and manipulating time series\n", "3. Using ```f-strings``` to print dates/times\n", "\n", "**This assignment is due by 11:59 p.m. the day before class,** and should be uploaded into D2L for Day-04-preclass. Submission instructions can be found at the end of the notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Imports for the notebook" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "\n", "import datetime\n", "import pandas as pd\n", "from IPython.display import HTML" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n", "## 1. Date and Time in Python\n", "\n", "We imported `datetime` above as that is the major module for time in Python. It can be a little confusing that the `datetime` module also contains a `datetime` class so that, expressions like `datetime.datetime` are used. We need to keep that straight.\n", "\n", "It's always good to see what's in the `datetime` class. \n", "\n", "✎ Do this - Use the `dir()` function and list the contents (the variables, the methods and the hidden elements). Do that below please." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# show contents of datetime" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.1 Components of ```datetime```\n", "\n", "The major components of `datetime` are:\n", "\n", "* `date` – Allows us to manipulate dates independent of time (month, day, year).\n", "* `time` – Allows us to manipulate time independent of date (hour, minute, second, microsecond).\n", "* `timedelta` — A duration of time used for manipulating dates and measuring.\n", "* `tzinfo` — An abstract class for dealing with time zones.\n", "* `datetime` - A combination of the `date` and `time` class" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For a lot of the work you need to do, the `datetime` object is the way to go. Like a `date` object, datetime assumes the current Gregorian calendar extended in both directions; like a `time` object, datetime assumes there are exactly 3600*24 seconds in every day.\n", "\n", "With that in mind, let's take a quick look at a `datetime` object.\n", "\n", "✎ Do this - Use the dir function to list its contents." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# show contents of datetime" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ```datetime``` class can tell us what time it is right now (or at least when the line of code is executed). We can use the method `now` to return the time it is \"now\"." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-09-10 12:42:35.219308\n", "\n" ] } ], "source": [ "now_inst = datetime.datetime.now()\n", "print(now_inst)\n", "print(type(now_inst))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To be clear, the call `datetime.datetime.now()` is interpreted as:\n", "* using the contents of the `datetime` module\n", "* using the `datetime` class in that module\n", "* using the `now` method of that class\n", "\n", "Which yields a `datetime.datetime` instance from the `type` call (an instance of the `datetime` class in the `datetime` module). Isn't coding fun \n", "😄" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A `datetime.datetime` instance has a number of instance variables. \n", "\n", "✎ Do this - List a few of these instance variables as you did above." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# what are some of the components" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.2 Printing dates and times\n", "\n", "One of the nice things about ```datetime``` is that its types can be used to form more human readable date and time information. In fact, we are able to format this information in a variety of ways through its types.\n", "\n", "✎ Do this - Print the year, day and month from the `now_inst`. Do a nicely formatted job. Do you know about f-strings ? It is the modern Python way to go." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Nice time strings\n", "\n", "But that is just ugly. It would be nice to print the **name** of the month, the **day of the week** and other niceties. Of course there is a way to do that, called `strftime` (string-from-time I assume, not a great name). In order to make this work there are a host of settings you can use to go from a datetime object to a nice string. Knowing some of these is important, here's a list of conversion elements . Each format element, a % followed by a single character, represents a particular way to format a time element. You can mix and match these format elements in a string (with other string parts) and extract, in a nice string way, some part of the time from your `datetime.dateime` instance. Some examples" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Nice string: Thu Sep 10 12:42:35 2020\n", "Year and Month: 2020 -- September\n", "Month(as a number) and Day: 09 10\n" ] } ], "source": [ "print(\"Nice string: \",now_inst.strftime(\"%c\")) # nice string based on your locale\n", "print(\"Year and Month: \", now_inst.strftime(\"%Y -- %B\"))\n", "print(\"Month(as a number) and Day: \", now_inst.strftime(\"%m %d\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The string conversion **won't** do the ordinal suffixes we are used to like \"July 23**rd**\" or \"June 1**st**\" but we can fix that (if you don't remember, here are the examples of ordinal suffix . \n", "\n", "✎ Do this - Write a function `ordinal_suffix` below that takes in a integer day and returns, as a **string**, the number with the appropriate ordinal suffix. For example `ordinal_suffix(21) ⇒ \"21st\"` Think about it (write some examples down from English) " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# your function ordinal_suffix here" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "# write a couple of tests here to show your function works." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## 2. Working with time series data\n", "\n", "Data that includes some time aspect is quite common in science and engineering. There are a number of interesting research questions for such data and a wide variety of methods to analyze them. A very relevant data set with a temporal aspect is the [Coronavirus data set](https://ourworldindata.org/coronavirus-source-data), which is used to generate many figures and charts appearing on news sites and reports. We will eventually investigate this data set, but to start we will work with a simpler data set on sunspots.\n", "\n", "### 2.1 Reading in the data\n", "✎ Do this - In the directory is a text file called \"sunspots.txt\". It is a text file, but you can read it in with `read_csv` and using the white space delimiter parameter. But try that once and see what happens" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "# read in the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do this - What is the error? Look it up and see if you can figure it out before you move on. I'll wait ⏱" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.2 Correcting the reading problem\n", "\n", "The problem is that a new set of column values show up **way** down in the data list (Line 14336 to be precise). The parser for column separation made the column determination based on the first set of column values. That changed further down. \n", "\n", "There are multiple ways to solve the problem. That is kind of bad news because it would be better if there were only one. The one you might explore is `read_fwf `. The \"fwf\" stands for \"fixed width file\", where the values being read show up in columns aligned by spaces. \n", "\n", "The other is to stay with `read_csv` which is the way we will go. The way to fix this is to provide the `names` of all the column headers, indicating how many columns there are. Go ahead and do that now. The other file \"sun-descrip.txt\" tells you about the file and it's contents.\n", "\n", "✎ Do this - Read the file in by specifying the column names." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "# read in the data again" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.3 Cleaning up our data set\n", "\n", "If you look at the `dtypes` you will see that we don't have a datetime, but do have different columns with date information. Our goal now is to:\n", "* read the required columns into a new DataFrame\n", "* turn that info into a datetime.datetime Series\n", "* add the the datetime to the DataFrame\n", "* get rid of the extraneous data columns now that we have a datetime.\n", "\n", "The relevant function that goes from information to a datetime is called `pandas.to_datetime`. It takes the same format elements as `strftime`. Let's look at an example:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "String rep: 10 09 2020\n", "\n", "Date: 2020-09-10 00:00:00\n" ] } ], "source": [ "now_inst = datetime.datetime.now() # a dateime of right now\n", "now_str = now_inst.strftime(\"%d %m %Y\") # day, month, year\n", "print(\"String rep: \", now_str)\n", "day_month_year = pd.to_datetime(now_str, format=\"%d %m %Y\") # format must match the string format\n", "print(type(day_month_year)) # a \n", "print(\"Date: \", day_month_year)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We use `strftime` to extra a string the represents (using the format specifiers we provide) the time. We can use `to_datetime` to convert that string back to a datetime. We can provide the format of the string (again using font specifiers) or we can let the system figure it out for us. Better we do it but try it and see.\n", "\n", "✎ Do this - Create a DataFrame of elements of time in the original dataframe and turn that into a datetime Series using `to_datetime`" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "# make the series" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's add that datetime Series to our dataframe. You might be tempted to use `.append`. **Don't**. \n", "\n", "I tried it and it is very memory and CPU intensive. Better to just `.assign`. The syntax of `.assign` is a little tricky but look it up. You have to use the '=' sign in the assign call.\n", "\n", "Then `.drop` the now useless columns (the old date stuff). Look at your `.dtypes` and see if you have a datetime in there now.\n", "\n", "✎ Do this - Using `assign` and `drop`, create add the datetime Series to our dataframe." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.4 Visualizing our data set\n", "\n", "Our sunspot data set is too large to infer trends by simply looking at the numbers, so let's great a plot to look for patterns and trends.\n", "\n", "\n", "✎ Do this - Plot the datetime (x axis) vs the sunspot count. Do you see any patterns there? Tell me about the pattern." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "# plot datetime vs sunspot count" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Do this - What pattern do you see? Can you find information online why this pattern might exist?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n", "# 3. Assignment wrap-up\n", "\n", "Hopefully you were able to get through all of that. We'll be trouble-shooting any issues you had\n", "\n", "**You must completely fill this out in order to receive credit for the assignment!**" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "HTML(\n", "\"\"\"\n", "\n", "\"\"\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---------\n", "### Congratulations, you're done with your pre-class assignment!\n", "\n", "Now, you just need to submit this assignment by uploading it to the course Desire2Learn web page for Day 3's pre-class submission folder (Don't forget to add your name in the first cell)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }