{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "###

✅ Put your name here.

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# CMSE Homework 05\n", "\n", "## Using Tensorflow/Keras for classification\n", "\n", "\n", "### Assignment instructions\n", "\n", "Work through the following assignment, making sure to follow all of the directions and answer all of the questions.\n", "\n", "There are 25 points possible on this assignment. Point values for each part are included in the section headers.\n", "\n", "This assignment is due roughly two weeks from now at 11:59 pm on **Friday Dec. 4th**. It should be uploaded into the \"Homework Assignments\" submission folder for Homework #5 in your D2L webpage. Submission instructions can be found at the end of the notebook.\n", "\n", "**Hint**: It is possible you are asked to do something you are not familiar with. That's why you have internet access. Do some smart searches and see what you can find! \n", "\n", "### Goals for this homework assignment\n", "\n", "The goal for this assignment is for you to learn about neural network implemented by __Tensorflow__ and __Keras__ to do some basic classification tasks. Unless you already have experience with Tensorflow or Keras or similar libraries, you will not understand everything about this right away. You probably haven't even installed Tensorflow yet. Don't worry, that is perfectly fine. We will walk through one example in detail. So don't worry if some steps seem arbitrary or look like magic to you! We've got to start somewhere.\n", "\n", "[Tensorflow](https://www.tensorflow.org/) is a free and open-source software library for machine learning. It is one of the most popular software package for training and inference of deep neural networks. [Keras](https://keras.io/) is the high-level API of TensorFlow 2.0. \n", "\n", "\n", "To get started, you need to set up the required software. You can follow the detailed instruction from the [installation guide of Tensorflow](https://www.tensorflow.org/install) which also includes Keras. In most cases, you can simply type\n", "\n", "`pip install --upgrade tensorflow`\n", "\n", "Now you are ready for action!\n", "\n", "__Note__ Following the installation guide, you should have installed Tensorflow V2.3.1, and Numpy should be downgraded to V1.18.5. If you have trouble with installation or you run into issues with dead kernels, please check the versions of the packages installed and asked for help during help hours!\n", "\n", "Below is the import for this assignement." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "## Our import\n", "import tensorflow as tf\n", "from tensorflow.keras.datasets import fashion_mnist\n", "from tensorflow.keras.datasets import mnist\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Dense\n", "from tensorflow.keras.utils import to_categorical\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 1. Adding notebook to your turn-in repository\n", "\n", "Like you did for Homework 3, you're going to add it to the CMSE202 repository you created in class so that you can track your progress on the assignment and preserve the final version that you turn in. In order to do this you need to:\n", "\n", "* Navigate to your /CMSE202/repos repository and create a new directory called hw-05.\n", "* Move this notebook into that new directory in your repository, then add it and commit it to your repository.\n", " * Finally, to test that everything is working, \"git push\" the file so that it ends up in your GitHub repository.\n", "\n", "Important: Make sure you've added your TA as a collaborators to your respository with \"Read\" access so that we can see your assignment. (*If you did this for Homework 3, you do not need to do it again*)\n", "\n", "* Section 001: tuethan\n", "* Section 002: Luis-Polanco\n", "* Section 003: DavidRimel\n", "\n", "Also important: Make sure that the version of this notebook that you are working on is the same one that you just added to your repository! If you are working on a different copy of the notebook, none of your changes will be tracked.\n", "\n", "If everything went as intended, the file should now show up on your GitHub account CMSE202 repository under the hw-04 directory that you just created. Periodically, you'll be asked to commit your changes to the repository and push them to the remote GitHub location. Of course, you can always commit your changes more often than that, if you wish. It can be good to get into a habit of committing your changes any time you make a significant modification, or when you stop working on the project for a bit." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: Classifying MNIST data (6 points)\n", "\n", "\n", "We will now walk through a concrete example of a neural network, which makes use of Keras to learn to classify hand-written digits.\n", "\n", "The problem we are trying to solve here is to classify grayscale images of handwritten digits (28 pixels by 28 pixels), into their 10 \n", "categories (0 to 9). The dataset we will use is the MNIST dataset, a classic dataset in the machine learning community, which has been \n", "around for almost as long as the field itself and has been very intensively studied. It's a set of 60,000 training images, plus 10,000 test \n", "images, assembled by the National Institute of Standards and Technology (the NIST in MNIST) in the 1980s. You can think of \"solving\" MNIST \n", "as the \"Hello World\" of deep learning -- it's what you do to verify that your algorithms are working as expected. As you become a machine \n", "learning practitioner, you will see MNIST come up over and over again, in scientific papers, blog posts, and so on.\n", "\n", "The MNIST dataset comes pre-loaded in Keras, in the form of a set of four Numpy arrays:\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "(train_images, train_labels), (test_images, test_labels)=mnist.load_data()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`train_images` and `train_labels` form the \"training set\", the data that the model will learn from. The model will then be tested on the \n", "\"test set\", `test_images` and `test_labels`. Our images are encoded as Numpy arrays, and the labels are simply an array of digits, ranging \n", "from 0 to 9. There is a one-to-one correspondence between the images and the labels.\n", "\n", "Let's have a look at the training data:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_images.shape\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "len(train_labels)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_labels" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's have a look at the test data:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_images.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "len(test_labels)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_labels" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 2.1 (1 points)** What is the size of the training and testing dataset for MNIST, respectively?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 2.2 (2 points)** Make a histogram for the training label. Is this dataset balanced?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also take a look at the individual images." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "digit = train_images[6]\n", "plt.imshow(digit, cmap=plt.cm.binary)\n", "train_labels[6]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Before training, we will preprocess our data by reshaping it into the shape that the network expects, and scaling it so that all values are in \n", "the `[0, 1]` interval. Previously, our training images for instance were stored in an array of shape `(60000, 28, 28)` of type `uint8` with \n", "values in the `[0, 255]` interval. We transform it into a `float32` array of shape `(60000, 28 * 28)` with values between 0 and 1." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_images = train_images.reshape(train_images.shape[0],-1)/255.0\n", "test_images = test_images.reshape(test_images.shape[0],-1)/255.0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We also need to categorically encode the labels." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_labels = to_categorical(train_labels)\n", "test_labels = to_categorical(test_labels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let’s start our neural network models.\n", "\n", "\n", "\n", "

\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our workflow will be as follow: first we will present our neural network with the training data, `train_images` and `train_labels`. The \n", "network will then learn to associate images and labels. Finally, we will ask the network to produce predictions for `test_images`, and we \n", "will verify if these predictions match the labels from `test_labels`.\n", "\n", "Let's build our network.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "network = Sequential()\n", "network.add(Dense(10, input_dim=784, activation='relu'))\n", "network.add(Dense(10, activation='softmax'))\n", "network.compile(optimizer='rmsprop',\n", " loss='categorical_crossentropy',\n", " metrics=['accuracy'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Sequential model is the easiest model Keras has. The first line of `model.add` method is adding your hidden layer with 10 cells, coming from 784 input cells.\n", "\n", "The second add method is adding your output layer to the network. This has 10 cells.\n", "\n", "The Relu and Softmax activation options are non-linear. Being able to use non-linear data makes Neural Network particularly useful. Generally, neural networks can map any data distribution at any level of complexity.\n", "You don’t have to know what Relu and Softmax are. You just need to follow these tips:\n", "1. Use Relu whenever possible, on every hidden layer.\n", "2. Use Softmax on output layers with more than two categories to be predicted.\n", "3. Use Sigmoid on an output layer with two categories.\n", "\n", "After creating your model, call `compile` method to finish your model. It usually takes three parameters. Always use `categorical_crossentropy` for multi-categories, and `binary_crossentropy` for two categories. Use `adam` or `rmsprop` as the optimizer since both of them are pretty good. And you need accuracy as the `metric` to check your network performance." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are now ready to train our network, which in Keras is done via a call to the `fit` method of the network: \n", "we \"fit\" the model to its training data. We will use 10% of the training data as the validation data, hence validation_split was set to 0.1." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "history=network.fit(train_images, train_labels, epochs=20, validation_split=0.1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Four quantities are being displayed during training: the \"loss\" of the network over the training and validation data, and the accuracy of the network over \n", "the training and validation data.\n", "\n", "`epoch` is the number of training loops we will do. One epoch will expose all of our training data once to the network. More epochs means the network will know better about our data.\n", "\n", "We can plot a history of training and validation loss and accuracy." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.clf() # clear figure\n", "\n", "loss = history.history['loss']\n", "val_loss = history.history['val_loss']\n", "\n", "epochs = range(1, len(loss) + 1)\n", "\n", "plt.plot(epochs, loss, 'bo', label='Training loss')\n", "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n", "plt.title('Training and validation loss')\n", "plt.xlabel('Epochs')\n", "plt.ylabel('Loss')\n", "plt.legend()\n", "\n", "plt.show()\n", "\n", "acc = history.history['accuracy']\n", "val_acc = history.history['val_accuracy']\n", "epochs=range(1, len(acc)+1)\n", "\n", "plt.plot(epochs, acc, 'bo', label='Training acc')\n", "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n", "plt.title('Training and validation accuracy')\n", "plt.xlabel('Epochs')\n", "plt.ylabel('Accuracy')\n", "plt.legend()\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "The dots are the training loss and accuracy, while the solid lines are the validation loss and accuracy. *Note that your own results may vary \n", "slightly due to a different random initialization of your network.*\n", "\n", "As you can see, the training loss decreases with every epoch and the training accuracy increases with every epoch. That's what you would \n", "expect when running gradient descent optimization -- the quantity you are trying to minimize should get lower with every iteration. But that \n", "isn't the case for the validation loss and accuracy: they seem to peak at the eleventh epoch. What you are seeing is **overfitting**: after the 11th epoch, we are over-optimizing on the training data, and we \n", "ended up learning representations that are specific to the training data and do not generalize to data outside of the training set.\n", "\n", "In this case, to prevent overfitting, we could simply stop training after 10 epochs.\n", "\n", "**Question 2.3 (2 points)** Let's train a new network from scratch for 10 epochs. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Answer ###\n", "\n", "network = Sequential()\n", "network.add(Dense(10, input_dim=784, activation='relu'))\n", "network.add(Dense(10, activation='softmax'))\n", "network.compile(optimizer='rmsprop',\n", " loss='categorical_crossentropy',\n", " metrics=['accuracy'])\n", "history=network.fit(train_images, train_labels, epochs=10, validation_split=0.1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now evaluate the result on our test data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_loss, test_acc = network.evaluate(test_images, test_labels)\n", "print('test_acc:', test_acc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 2.4 (1 points)** Run the code above. What is the testing accuracy?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✎ Answer here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "This concludes our very first example -- you just saw how we could build and a train a neural network to classify handwritten digits, in \n", "less than 20 lines of Python code!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 3: Classifying Fashion-MNIST data (19 points)\n", "\n", "\n", "We will now work on another dataset called fashion MNIST. Fashion MNIST is a dataset of ten categories of clothing and accessories, in grayscales. \n", "\n", "\n", "

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 3.1 (2 points)** Load the Fashion-MNIST dataset. *Hint: Keras has this dataset preloaded. Look at the import at the beginning of this notebook*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 3.2 (3 points)** Explore the data. Find the size of the training and testing dataset. How many categories are in the labels?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 3.3 (2 points)** Plot the first couple of images in the training set and their labels. What do you observe?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 3.4 (2 points)** Now following the previous example of MNIST, reshape the dataset and categorically encode the labels." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 3.5 (5 points)** Build the network. Use the **same** network architecture and parameters as the MNIST example, and train with 10 epochs. What is the accuracy you see for this dataset? Do you observe overfitting during this training?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should have got a decent accuracy out of this dataset too, but not as high as the accuracy from the MNIST dataset, because the images here are more complicated. Now to improve the accuracy, you can try to widen/deepend the feedforward neural network. You are encouraged to look at the previous example and search on the internet on how to do that. As long as you see some improvement, it will be fine.\n", "\n", "**Question 3.6 (5 points)** Try changing the network architecture (deepen/widen the network) to improve the accuracy. What changes did you make? What accuracy did you get this time?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Put your code here ###\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Congratulations! You have learned about the basics of the Tensorflow package, which is a powerful tool for machine learning. There are tons of on-line resources about Tensorflow. You are encouraged to check them out when you have time." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "### Assignment wrap-up\n", "\n", "Please fill out the form that appears when you run the code below. **You must completely fill this out in order to receive credit for the assignment!**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.display import HTML\n", "HTML(\n", "\"\"\"\n", "\n", "\"\"\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Congratulations, you're done!\n", "\n", "Submit this assignment by uploading it to the course Desire2Learn web page. Go to the \"Homework Assignments\" folder, find the dropbox link for Homework #5, and upload your notebook **and the script you wrote**." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 }