{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <p style=\"text-align: right;\"> &#9989; Put your name here</p>\n",
    "#### <p style=\"text-align: right;\"> &#9989; Put your group member names here</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Day 07 In-Class Assignment: Command Line\n",
    "\n",
    "What if you wanted to make a map like the one below? How would you go about doing it using Python? Perhaps someone else already created a Python package that you could use!\n",
    "\n",
    "<img src=\"https://www.270towin.com/maps/DW1lp.png\" width=\"60%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Agenda for today's class\n",
    "\n",
    "</p>\n",
    "\n",
    "1. Getting data from the web\n",
    "1. Learning how to use `conda` and `pip`\n",
    "1. Practice with `conda` and `pip`\n",
    "1. Visualizing data with maps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 1: Review\n",
    "\n",
    "Did anyone have any specific issues with the pre-class assignment?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Below are all of the important commands/topics we learned in the pre-class assignment. Please review these and if any were unclear to you please ask.*\n",
    "\n",
    "* `ls`\n",
    "    * `ls -a`\n",
    "* `pwd`\n",
    "* `cd`\n",
    "    * `cd [FOLDER]`\n",
    "    * `cd [PATH_TO_FOLDER]`\n",
    "    * `cd ~`\n",
    "    * `cd /`\n",
    "    * `cd ..`\n",
    "    * `cd .`\n",
    "    * `cd -`\n",
    "* `mkdir [NEW_FOLDER]`\n",
    "* `touch [NEW_FILE]`\n",
    "* `cp`\n",
    "    * `cp [FILENAME] [NEW_FILENAME]`\n",
    "    * `cp -r [FOLDER] [NEW_FOLDER]`\n",
    "    * Using paths instead of names\n",
    "* `rm`\n",
    "    * `rm [FILENAME]`\n",
    "    * `rm -r [FOLDERNAME]`\n",
    "    * Using paths instead of names\n",
    "* `mv`\n",
    "    * `mv [PATH_TO_FILE/FOLDER] [NEW_PATH_TO_FILE/FOLDER]`\n",
    "    * `mv [PATH_TO_FOLDER] [NEW_PATH_TO_FOLDER]`\n",
    "    * Using command to rename files/folders\n",
    "* `nano`\n",
    "    * `nano [FILENAME]`\n",
    "    * `nano [NEW_FILENAME]`\n",
    "    * Commands are on screen (`^` means control)\n",
    "* `vim`\n",
    "    * `vim [FILENAME]`\n",
    "    * `vim [NEW_FILENAME]`\n",
    "    * Vim modes\n",
    "        * Command Mode\n",
    "            * `:wq`\n",
    "            * `:q!`\n",
    "            * `i` to switch to Insert\n",
    "        * Insert Mode\n",
    "            * `Esc` to switch to Command     \n",
    "    * Vim Swap Files and Swap File options\n",
    "* `cat [FILENAME]`\n",
    "* `spyder`\n",
    "    * `spyder [FILENAME]`\n",
    "    * `touch [NEW_FILE] && spyder [new_file]`\n",
    "* `jupyter notebook`\n",
    "    * `jupyter notebook [FILENAME]`\n",
    "    * Closing using `ctrl` + `c`\n",
    "* Opening more than one terminal at a time\n",
    "* `curl [URL]`\n",
    "    * `curl -O [URL]`\n",
    "    * `curl -o [URL]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "# 2. Installing Third Party Packages\n",
    "\n",
    "How do we go about installing new third party packages that aren't already a part of our Python installation so we can use them with our Jupyter Notebooks and Python?\n",
    "\n",
    "There are two major ways we will use in this class.\n",
    "\n",
    "1. Through our `conda` package manager that ships with Anaconda\n",
    "2. Through `pip` package manager that comes apart of python. The `pip` package manager pulls from the [Python Package Index](https://pypi.org/) (PyPI)\n",
    "\n",
    "Installation is generally pretty straight forward using either of these ways.\n",
    "\n",
    "Let's test out installing a package using `conda` and a package using `pip`.\n",
    "\n",
    "**Question:** What happens when you run `conda --version` and `pip --version`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <font size=8 color=\"#009600\">&#9998;</font> Do This - Erase this first line of this cell and answer the question above. (double-click on this text to edit this cell, and hit shift+enter to save the text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conda installation\n",
    "\n",
    "The anaconda distribution we are working with maintains its own package installation system called `conda`. When you install anaconda it also installs the `conda` package manager. A package manager allows you to:\n",
    "* install new python packages along with **all the required dependency packages**. This last point is very important as it can be difficult to know what other python packages you need for the package you are loading.\n",
    "    * Package dependencies are pretty much like the ingredients. In order to make bread you need flour, sugar, and yeast. Well in order to use a packages such as `pandas` you need `setuptools`, `NumPy`, `python-dateutil`, and `pytz` packages. \n",
    "* update packages you already have with new versions\n",
    "\n",
    "When you installed anaconda, you got a lot of packages already installed. You can look at [anaconda windows 64 packages](https://docs.anaconda.com/anaconda/packages/py3.7_win-64/) or [anaconda mac packages](https://docs.anaconda.com/anaconda/packages/py3.7_osx-64/) to see a list\n",
    "\n",
    "However, even anaconda doesn't provide every package. Take a look at [https://pypi.org/](https://pypi.org/), the Python Package Index (often called PyPi). There are many there that are not in the anaconda set that you downloaded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prettytable\n",
    "\n",
    "Occasionally there are times when you're looking at simple datasets in Python and it would be handy to be able to manipulate and visualize the data in simple [ASCII formats](https://en.wikipedia.org/wiki/ASCII) without dealing with all of the Pandas dataframe overhead. Or perhaps you need to output the a data table in an HTML format so that you can render the table on the web. To achieve these sorts of tasks, you can install a package called `prettytable` using `conda` like so:\n",
    "\n",
    "`conda install -c conda-forge prettytable`\n",
    "\n",
    "This uses conda to install the package by locating it on Conda-Forge, which is a community maintained set of conda packages. When you run the command, you'll see that conda finds the packages and also checks if it should update any other packages at the same time and will let you know everything it plans to do before you confirm that you want it to happen. Once you've installed the package, you should be able to import it. Some documentation for `prettytable` can be found here: [https://code.google.com/archive/p/prettytable/](https://code.google.com/archive/p/prettytable/)\n",
    "\n",
    "If you were able to get `prettytable` installed, test it out and make sure it's working by trying to run some of the examples from this tutorial page:\n",
    "\n",
    "[http://zetcode.com/python/prettytable/](http://zetcode.com/python/prettytable/)\n",
    "\n",
    "Note that some of the tutorials code will have `#!/usr/bin/python3` on the top. You can ignore this, this \n",
    "\n",
    "**Important note:** If you're trying to do this on JupyterHub, rather than on your personal computer, you will get a error that you don't have permissions to install new packages. There are ways to work around this, but we don't address this for now as this is a more advanced procedure. If this applies to you, please include a note in this cell box. If there is someone in the group that is not using JupyterHub, ask them if they could share their screen so you can view what they are doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put your prettytable testing code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pip installation\n",
    "\n",
    "The more general Python package manager is called `pip`. It works for any Python install, not just an anaconda install. Generally speaking, if you can use `conda` then you should, but it is worth looking at the more general `pip` which is available anywhere\n",
    "\n",
    "### cmocean\n",
    "\n",
    "There is a great package called [`cmocean`](http://matplotlib.org/cmocean/) that contains several colormaps that are commonly using for plotting oceanography data. One of the awesome things about the colormaps is that they are designed to be color-blind friendly, meaning that even if you're colorblind you can still extract information from the colormap.\n",
    "\n",
    "Try installing the package by doing the following on the command line:\n",
    "\n",
    "`pip install \"cmocean[plots]\"`\n",
    "    \n",
    "This not only installs the main cmocean module, but also installs the \"plots\" submodule, which is necessary to get some of the examples from the cmocean documentation page to work. When you run the command, you'll see that pip finds the packages and also checks if it should install or update any other packages at the same time. Once you've installed the package, you should be able to import it. Test it out in this notebook! Try making a plot using one of the new colormaps you have available. **You also have to include `import cmocean.plots` if you try to run any of the examples that use `cmocean.plots`.**\n",
    "\n",
    "**Important note:** If you're doing this on JupyterHub, you'll need to modify the command a bit in order to make sure you don't run into permission issues with the installation process. If you're not install pip packaged on your local machine, you need to use the `--user` flag, like so:\n",
    "\n",
    "`pip install --user \"cmocean[plots]\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test out importing cmocean here.\n",
    "# Try using the plot below\n",
    "\n",
    "import cmocean\n",
    "import cmocean.cm as cmo\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "fig = plt.figure(figsize=(8, 3))\n",
    "ax = fig.add_subplot(1, 2, 1)\n",
    "Z = np.random.randn(10,10)\n",
    "ax.pcolormesh(Z, cmap=cmo.matter)\n",
    "\n",
    "ax = fig.add_subplot(1, 2, 2)\n",
    "lightcmap = cmocean.tools.lighten(cmo.matter, 0.5)\n",
    "ax.pcolormesh(Z, cmap=lightcmap)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 3. Getting data from the web\n",
    "For this assignment we need to download a couple of data files.  Here are direct links to the files:\n",
    "\n",
    "https://raw.githubusercontent.com/wrobstory/vincent/master/examples/data/US_Unemployment_Oct2012.csv\n",
    "\n",
    "https://raw.githubusercontent.com/python-visualization/folium/master/examples/data/us-states.json\n",
    "\n",
    "We need to figure out a way to get a copy of these files into your `assignments` repository.  You could just download them to your desktop or downloads folder and then move them to the right place, but there's a better way! Let's use the `curl` terminal command we learned in the pre-class assignment instead. \n",
    "\n",
    "***Hint***: This time we are going to use a trick.  You can actually run terminal commands from inside python. All you need to do is add the \"!\" to the beginning of your command. \n",
    "\n",
    "*Note to Git-Bash Users: When running `!` this will be using command prompt (cmd.exe) which runs on dos-commands. So commands like `ls` will not work. However commands such as `pwd`, `curl`, `conda`, and `pip` are commands that are used by both command prompt and unix-like shells, so those should work with `!`.*\n",
    "\n",
    "**&#9989; Do This:** Try it out -- run the `pwd` command by using the following python code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**&#9989; Do This:** Now download the two data files we need for class using the `curl` command prefaced by the python terminal command (!). **Important**: you want to make sure that curl downloads and saves the file on to your computer, but by default it will just print the contents of the file to the screen. You should have figured out how to save it in your pre-class assignment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**&#9989; Do This:** As a group inspect the two new files. Try opening the files in python and displaying the first few rows of data.\n",
    "\n",
    "**Question :** What types of files are they? What type of data do they seem to contain?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=8 color=\"#009600\">&#9998;</font> Do This - Erase the contents of this cell and replace it with your answer to the above question!  (double-click on this text to edit this cell, and hit shift+enter to save the text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Put your code here. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "# 4. Visualizing data with maps, `folium`\n",
    "\n",
    "Finding good ways to create map visualizations in Python can be tricky. \n",
    "\n",
    "**&#9989; Do This:** The code below doesn't work. Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline \n",
    "\n",
    "import folium\n",
    "map_msu = folium.Map(location=[42.7284006,-84.4804778])\n",
    "folium.Marker([42.7284006,-84.4804778], popup='MSU').add_to(map_msu)\n",
    "map_msu\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** What does the above example do?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=8 color=\"#009600\">&#9998;</font> Do This - Erase the contents of this cell and replace it with your answer to the above question!  (double-click on this text to edit this cell, and hit shift+enter to save the text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** Assuming we did everything up to this point correctly, what does the following code do? *Warning*: there may be a bug in the code!\n",
    "\n",
    "*Hint: You will need to look at the documentation [here](https://python-visualization.github.io/folium/quickstart.html#Choropleth-maps)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import folium\n",
    "import pandas as pd\n",
    "\n",
    "state_geo = r'./us-states.json'\n",
    "state_unemployment = r'./US_Unemployment_Oct2012.csv'\n",
    "\n",
    "state_data = pd.read_csv(state_unemployment)\n",
    "\n",
    "#Let Folium determine the scale\n",
    "map = folium.Map(location=[48, -102], zoom_start=3)\n",
    "map.choropleth(geo_path=state_geo, data=state_data,\n",
    "             columns=['State', 'Unemployment'],\n",
    "             key_on='feature.id',\n",
    "             fill_color='YlOrRd', fill_opacity=0.7, line_opacity=0.2,\n",
    "             legend_name='Unemployment Rate (%)', smooth_factor=0.1,highlight=True)\n",
    "\n",
    "map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: If you're running into issues getting Python to run from the command line, contact your instructors as soon as possible. There might be a simple solution!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 5. Running some sample Python code\n",
    "\n",
    "Now that you have a text editor and a command line program ready to use in the pre-class, we can breifly introduce what are known as \"python scripts\". They are the same python we know and love, but just in a slightly different format. Instead of python being cell-based like these jupyter notebooks, scripts are essentially just one giant block of python code that is run all at once.\n",
    "\n",
    "In the in-class folder of the course assignments repository, along with this jupyter notebook, you may have noticed a file call \"fib.py\". This is very simple piece of code that prints out the Fibonnaci sequence, which should be: 0, 1, 1, 2, 3, 5, 8, 13, 21, ...\n",
    "\n",
    "Try running the code on the command line by doing the following on the command line:\n",
    "\n",
    "<code>$ python fib.py</code> \n",
    "\n",
    "It looks like there is a bug in this code. Open the file in your preferred text editor and try to figure out what going wrong. Try to fix the code, save the file, and then run it again.\n",
    "\n",
    "Did it work? Write down what the bug was in the cell below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=8 color=\"#009600\">&#9998;</font> **What was the bug?**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------\n",
    "# 6. IPython\n",
    "\n",
    "You have been working with jupyter notebooks for some time. But did you know you can use the interpreter that notebooks use on the command line? You can and it is very helpful.\n",
    "\n",
    "That interpreter is called `ipython`, where `i` is for \"interactive\". We can open an ipython interepreter on our command line and get many of the advantages of both python and command line at the same time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting ipython\n",
    "\n",
    "When you get anaconda, you get an ipython command console as well. A few ways to open it:\n",
    "\n",
    "- The easiest way is to open Spyder. It opens and ipython CLI automatically!\n",
    "- On Windows, open up \"Anaconda Prompt\" or your command prompt of choice (Ubuntu, Windows Terminal, GitBash). However, note that when using Ubuntu or Windows Terminal, the plots you create will not pop up in any window. In order to view plots you will have to download them.\n",
    "- On Mac use `Terminal`\n",
    "\n",
    "Having opened a command line/prompt, type `ipython`. You should get a window that looks something like the one below\n",
    "\n",
    "<img src=\"https://drive.google.com/uc?id=1XKduBRwtRBjhusIRzFFY5j_j3Qdb9G9g\">\n",
    "\n",
    "If it doesn't say \"Python 3.7.\" at the top of that window, kill the session with `Ctrl-D` (while holding the try `Ctrl` key press `d`). Start again with `ipython3`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What can I do?\n",
    "\n",
    "There are three \"kinds\" of things you can do in ipython\n",
    "- regular python stuff\n",
    "- ipython specific things\n",
    "- command line stuff (ls, cd, etc.)\n",
    "\n",
    "Let's take a quick look at each"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regular python\n",
    "\n",
    "Anything you have done before in python you can do here: set variables, import packages, draw graphs. Take a look at the window below.\n",
    "\n",
    "- note that you should be able to pop up a window of a graph. If not you can always save it as a file.\n",
    "\n",
    "<img src=\"https://drive.google.com/uc?id=1zN8k-mv-mJHM-uUr9PRP9QquEuNS9lxe\" width=\"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Try typing the following into the ipython command line interpreter. You should see a plot pop up**  \n",
    "*Windows Ubuntu/Windows Terminal Users: Note that the plots will not display for you. You will need to save the plot in order to view it**\n",
    ">```\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "ary = np.arange(1,10,0.1)\n",
    "plt.plot(ary)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=8 color=\"#009600\">&#9998;</font> Do This - Where you able to successfully plot?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ipython history things\n",
    "\n",
    "ipython has a large number of \"helper\" abilities to aid in your console use. Some simple things:\n",
    "\n",
    "ipython has an `In[n]` and `Out[n]` at the front of each line. The `n` is a history number that remembers the input provide or the output produced.\n",
    "\n",
    "- `up-arrow` key or `down-arrow` key on your keyboard moves through the history of what you have typed and presents a line that you can edit/change and then rerun\n",
    "- the `history -n` command gives you a list of inputs, with a \"history number\" of all that you have typed\n",
    "- the `rerun x` where `x` is a history number will rerun that command\n",
    "- the `run` command will load a script file and run it\n",
    "   - the `reload` will reload an updated file\n",
    "- the special characters `_`, `__` and `___` are the **outputs** from the previous, before that and before that operation.\n",
    "\n",
    "<img src=\"https://drive.google.com/uc?id=1ygF1ZxymbkFEec_UOU-etULaRPKs7hrG\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Look at the window above and try repeating the steps. The `history:5` refers to the line execution number 5 on the left hand side (`ary = np.arange(1,10,1`)**\n",
    "- (history:5), Using the up-arrow key to give you the `arange` line again, change the increment value to `1`. Then hit enter and rerun it.\n",
    "- (history:6) Replot by using up-arrow and re-running the `plt` line\n",
    "- (history:7) Look at your input history by typing `history -n`\n",
    "- (history:8) Type `ary` and press enter to list the array\n",
    "- (history:9) Find the history number of when the `ary` had an increment of `.1` (in the case above this is `history:3`). Then rerun it with the command `rerun [HISTORY:NUMBER]` to rerun that line (`rerun 3` in the case above).\n",
    "- (history:10) Type `ary` and press enter to list the changed array\n",
    "- (history:11) To view what `ary` was before we reset it we can use the 2nd previous output(`__`) and print it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=8 color=\"#009600\">&#9998;</font> Try the above! Where you able to use all of the shortcuts?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Querying elements\n",
    "\n",
    "You can ask questions about an element using ipython and get a lot of good information\n",
    "\n",
    "- follow any variable or other object with a `?` and ipython will print out information about the object. I created a `rng` range element on history:15. Typing `rng?` followed by enter tells me something about the object\n",
    "- if the object can take a method (and most can), then I can type a . and a tab and get a list of possible operations. For example, typing `rng.` followd by a tab gives me a list of operations and another tab moves through that list providing a brief description. In the image below I have selected (via a tab> `rng.count` and it is described below the method.\n",
    "\n",
    "<img src=\"https://drive.google.com/uc?id=10Iaje9ro3pWHteiXlfBeSx8YHx16xAJu\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=8 color=\"#009600\">&#9998;</font> Do This - Assign a variable to a python object (a list, a dict, something you like). Query what that variable is with a `?`. Look and see how many methods there are associated with that type (as we did with `rng` above). Note there be many and tabbing through all might be a bit tedious."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Command line operations\n",
    "\n",
    "Most of the shell commands we have used will work in ipython. For example\n",
    "- `ls` will list the files in the currently connected directory\n",
    "- `pwd` will tell you your current directory\n",
    "- `cd` will allow you to change what the current directory is.\n",
    "\n",
    "If you want to type a more complicated shell command, you preced it with a `!`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=8 color=\"#009600\">&#9998;</font> Do This - list the contents of your current directory. Now change directories to a sub directory and list the contents again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the quick tutorial\n",
    "\n",
    "The folks at ipython have a quick tutorial. You should take a look as it might give you some more useful details.\n",
    "\n",
    "<a href=\"https://ipython.readthedocs.io/en/stable/interactive/tutorial.html\"> Link to the official Ipython Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 7. Playing with ipython and creating your own file\n",
    "Let's just try to write a basic python program that does the kind of file loading that pandas does for us automatically.\n",
    "\n",
    "The way to do this is to try some things in the ipython CLI, get it to work, then add it to the editor. You can `reload` the file anytime you want to run things again.\n",
    "\n",
    "## Playing with ipython and writing file\n",
    "\n",
    "The following are the steps that are needed to open a file and create a list of lists of the contents.\n",
    "\n",
    "**First try following just in ipython to see how to accomplish the task.** \n",
    "\n",
    "**Then** starting writing a function called `cpandas` that:\n",
    "- takes a single argument, the name of a file to open\n",
    "- returns a list of lists, kind of like a Pandas dataframe\n",
    "- Note: The function name and arguments are already provided. Just delete `pass` and add your own code.\n",
    "\n",
    "Note that you can load a file with the ipython `run` command. This allows you to set up an edit-run cycle:\n",
    "- edit in your favorite editor\n",
    "- `run` the file again and see the results.\n",
    "- repeat as necessary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you need to know the directory to this notebook\n",
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. To get started, open up a terminal and make sure to navigate to this current folder in the terminal (run the cell above to see your current notebooks directory path.) Remember from the pre-class you can use `cd` to change directories\n",
    "1. Using your terminal create a new python script file named `cpandas.py` (The command to do this is `touch cpandas.py`). \n",
    "1. Then you need to open either one of the following  \n",
    "    A. Spyder using the command `spyder cpandas.py` in your command line  \n",
    "    B. An ipython interpreter and your favorite text editor (meaning two terminal windows) that has `cpandas.py` open in the text editor. \n",
    "1. If you are using `Spyder` make sure your ipython interpreter is running in the same directory as your `cpandas.py` file. To make sure of this, right-click on the tab called `cpandas.py`. Then click `Set console working directory`.\n",
    "1. In the same directory as this in-class notebook is the file `iris.data`, the same one we have worked with before. We're probably going to get sick of this dataset but we know it so it's convenient to work with it.\n",
    "1. Let's do a standard, every day, python opening of that file. The python command is `open`. Go ahead and type `open` in your ipython terminal. It should tell you that `open` is a function with various default values for its parameters. If you now type `open?`, the `?` will give you the docstring associated with `open`\n",
    "1. Let's open the file. Assign to some variable (I used `f_obj`) the result of opening `iris.data`\n",
    "1. Type in `f_obj` and see what type it is.\n",
    "1. Type in `f_obj.` and then the `TAB` key on your keyboard and see what the possible methods are you can use on a file object you've opened\n",
    "1. Let's go ahead and read in the contents of the file to a list named what ever you like (such as `iris`). There are lots of ways to do this, but you could:\n",
    "    1. use the `.readlines` method\n",
    "    1. do a `for line in f_obj:` loop. Grabs a single line of the file and puts it in the variable `line` (or whatever you chose to call the iterator variable.\n",
    "1. Now look at your list and what type each lines is\n",
    "1. Notice that you can't use your `f_obj` anymore, it is \"exhausted\": all the lines have been read. However, if you want to reopen it you can:\n",
    "    1. `f_obj.close()` the file\n",
    "    1. The using the up arrow of your keyboard, find the previous `open` and rerun it.\n",
    "    1. If you don't want to hit a bunch of up-arrows, you can do the following:\n",
    "        1. `%history -n` will list the line numbers of previous commands\n",
    "        1. `In[n]` will redo the `n`th command.\n",
    "1. We need to fix the type of the 4 columns of numbers, converting them with `float`. You can do the following in the `for line in f_obj:` loop or you create a seperate loop with your `iris` array.\n",
    "    1. The string `.split(\" \")` method will return a list of elements, each separated by a space (or an alternate character if you provide it as a parameter).\n",
    "    1. Notice if there is any `/n` characters and you might get rid of them\n",
    "    1. convert the numers in list with `float()`\n",
    "    1. Save the array as a new list\n",
    "1. You should end up with a list of lists, where each line is a list.\n",
    "\n",
    "The idea is that we can experiment in ipython and check out what we want, asking questions (what is the type, what methods are there, etc.) and then, having figured things out, write a file.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write a function to index `cpandas` list of lists\n",
    "\n",
    "A python list of list cannot be indexed like a numpy array\n",
    "Write another function in `cpandas.py` called `cpandas_idx`:\n",
    "- takes two arguments: a list of lists and a column number\n",
    "- returns a list of just the column values.\n",
    "- Note: The function name and arguments are already provided. Just delete `pass` and add your own code.\n",
    "\n",
    "Test your code our with the below cell!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loads in cpandas.py functions cpandas and cpandas_idx\n",
    "from cpandas import cpandas, cpandas_idx\n",
    "\n",
    "# Uses cpandas function\n",
    "iris = cpandas(\"iris.data\")\n",
    "\n",
    "print(\"First three rows of Iris.data list of lists:\")\n",
    "print(iris[0:3])\n",
    "\n",
    "# Uses cpandas_idx function\n",
    "iris_c2 = cpandas_idx(iris,2)\n",
    "\n",
    "print(\"\\nSecond Column of iris.data\")\n",
    "print(iris_c2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "### Congratulations, we're done!\n",
    "\n",
    "Now, you just need to submit this assignment by uploading it to the course <a href=\"https://d2l.msu.edu/\">Desire2Learn</a> web page for today's submission folder (Don't forget to add your names in the first cell)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&#169; Copyright 2020,  Michigan State University Board of Trustees."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
