✅ Put your name here.
\n", "\n", "####✅ Put your group member names here.
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 22 In-Class Assignment: Artificial Neural Network\n", "\n", "\n", "\n", "![](\"https://ml4a.github.io/images/temp_fig_mnist.png\")
\n",
"From: Machine Learning for Artists - https://ml4a.github.io/
\n", "\n", "\n", "\n", "1. **Scientific motivation** \n", " - Data Analysis / Pattern Recognition\n", "2. **Modeling tools** \n", " - Artificial Neural networks\n", " - Error Calculations\n", "3. **Programming concepts** \n", " - More Debugging\n", " - Selecting and using libraries\n", "4. **Python Programming Concepts** \n", " - More Understanding classes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Agenda for today's class \n", "\n", "1. Review pre-class assignment\n", "1. Modify code to be more flexible\n", "1. Use our ANN on the \"Digits\" dataset\n", "1. Finding/Using Neural Networks Libraries" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "# 1. Review pre-class assignment\n", "\n", "Below we summarize the steps involved in designing and training a feed-forward artificial neural network. We will use the [partSix.py](./partSix.py) file provided in the \"Neural Networks Demystified\" module which can be downloaded from github:\n", "\n", " git clone https://github.com/stephencwelch/Neural-Networks-Demystified\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# %load partSix.py\n", "# Neural Networks Demystified\n", "# Part 6: Training\n", "#\n", "# Supporting code for short YouTube series on artificial neural networks.\n", "#\n", "# Stephen Welch\n", "# @stephencwelch\n", "\n", "\n", "## ----------------------- Part 1 ---------------------------- ##\n", "import numpy as np\n", "\n", "# X = (hours sleeping, hours studying), y = Score on test\n", "X = np.array(([3,5], [5,1], [10,2]), dtype=float)\n", "y = np.array(([75], [82], [93]), dtype=float)\n", "\n", "# Normalize\n", "X = X/np.amax(X, axis=0)\n", "y = y/100 #Max test score is 100\n", "\n", "## ----------------------- Part 5 ---------------------------- ##\n", "\n", "class Neural_Network(object):\n", " def __init__(self): \n", " #Define Hyperparameters\n", " self.inputLayerSize = 2\n", " self.outputLayerSize = 1\n", " self.hiddenLayerSize = 3\n", " \n", " #Weights (parameters)\n", " self.W1 = np.random.randn(self.inputLayerSize,self.hiddenLayerSize)\n", " self.W2 = np.random.randn(self.hiddenLayerSize,self.outputLayerSize)\n", " \n", " def forward(self, X):\n", " #Propogate inputs though network\n", " self.z2 = np.dot(X, self.W1)\n", " self.a2 = self.sigmoid(self.z2)\n", " self.z3 = np.dot(self.a2, self.W2)\n", " yHat = self.sigmoid(self.z3) \n", " return yHat\n", " \n", " def sigmoid(self, z):\n", " #Apply sigmoid activation function to scalar, vector, or matrix\n", " return 1/(1+np.exp(-z))\n", " \n", " def sigmoidPrime(self,z):\n", " #Gradient of sigmoid\n", " return np.exp(-z)/((1+np.exp(-z))**2)\n", " \n", " def costFunction(self, X, y):\n", " #Compute cost for given X,y, use weights already stored in class.\n", " self.yHat = self.forward(X)\n", " J = 0.5*sum((y-self.yHat)**2)\n", " return J\n", " \n", " def costFunctionPrime(self, X, y):\n", " #Compute derivative with respect to W and W2 for a given X and y:\n", " self.yHat = self.forward(X)\n", " \n", " delta3 = np.multiply(-(y-self.yHat), self.sigmoidPrime(self.z3))\n", " dJdW2 = np.dot(self.a2.T, delta3)\n", " \n", " delta2 = np.dot(delta3, self.W2.T)*self.sigmoidPrime(self.z2)\n", " dJdW1 = np.dot(X.T, delta2) \n", " \n", " return dJdW1, dJdW2\n", " \n", " #Helper Functions for interacting with other classes:\n", " def getParams(self):\n", " #Get W1 and W2 unrolled into vector:\n", " params = np.concatenate((self.W1.ravel(), self.W2.ravel()))\n", " return params\n", " \n", " def setParams(self, params):\n", " #Set W1 and W2 using single paramater vector.\n", " W1_start = 0\n", " W1_end = self.hiddenLayerSize * self.inputLayerSize\n", " self.W1 = np.reshape(params[W1_start:W1_end], (self.inputLayerSize , self.hiddenLayerSize))\n", " W2_end = W1_end + self.hiddenLayerSize*self.outputLayerSize\n", " self.W2 = np.reshape(params[W1_end:W2_end], (self.hiddenLayerSize, self.outputLayerSize))\n", " \n", " def computeGradients(self, X, y):\n", " dJdW1, dJdW2 = self.costFunctionPrime(X, y)\n", " return np.concatenate((dJdW1.ravel(), dJdW2.ravel()))\n", "\n", "def computeNumericalGradient(N, X, y):\n", " paramsInitial = N.getParams()\n", " numgrad = np.zeros(paramsInitial.shape)\n", " perturb = np.zeros(paramsInitial.shape)\n", " e = 1e-4\n", "\n", " for p in range(len(paramsInitial)):\n", " #Set perturbation vector\n", " perturb[p] = e\n", " N.setParams(paramsInitial + perturb)\n", " loss2 = N.costFunction(X, y)\n", " \n", " N.setParams(paramsInitial - perturb)\n", " loss1 = N.costFunction(X, y)\n", "\n", " #Compute Numerical Gradient\n", " numgrad[p] = (loss2 - loss1) / (2*e)\n", "\n", " #Return the value we changed to zero:\n", " perturb[p] = 0\n", " \n", " #Return Params to original value:\n", " N.setParams(paramsInitial)\n", "\n", " return numgrad \n", " \n", "## ----------------------- Part 6 ---------------------------- ##\n", "from scipy import optimize\n", "\n", "\n", "class trainer(object):\n", " def __init__(self, N):\n", " #Make Local reference to network:\n", " self.N = N\n", " \n", " def callbackF(self, params):\n", " self.N.setParams(params)\n", " self.J.append(self.N.costFunction(self.X, self.y)) \n", " \n", " def costFunctionWrapper(self, params, X, y):\n", " self.N.setParams(params)\n", " cost = self.N.costFunction(X, y)\n", " grad = self.N.computeGradients(X,y)\n", " return cost, grad\n", " \n", " def train(self, X, y):\n", " #Make an internal variable for the callback function:\n", " self.X = X\n", " self.y = y\n", "\n", " #Make empty list to store costs:\n", " self.J = []\n", " \n", " params0 = self.N.getParams()\n", "\n", " options = {'maxiter': 200, 'disp' : True}\n", " _res = optimize.minimize(self.costFunctionWrapper, params0, jac=True, method='BFGS', \\\n", " args=(X, y), options=options, callback=self.callbackF)\n", "\n", " self.N.setParams(_res.x)\n", " self.optimizationResults = _res\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"Input Data\", X)\n", "print(\"Output Data\", y)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Untrained Random Network\n", "NN = Neural_Network()\n", "y1 = NN.forward(X)\n", "print(\"Untrained Output\", y1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Training step\n", "T = trainer(NN)\n", "T.train(X,y)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Trained Network\n", "y2 = NN.forward(X)\n", "print(\"Trained Output\",y2)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✅ **DO THIS:** Calculate and compare the [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error) for untrained network (```y1```) and the trained network (```y2```). " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Put your code here\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n", "# 2. Modify code to be more flexible\n", "\n", "The code for our Neural Network example above assumes an input layer size of 2, hidden layer size of 3 and an output layer size of 1. \n", "\n", "\n", "✅ **DO THIS:** Modify the code in Section 1 above so that the user can specify these as inputs when creating the Neural_Network object. The default values should stay the same. Rerun the above example to make sure it still works. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "# 3. Use our ANN on the \"Digits\" dataset.\n", "\n", "Here is the code copied from our previous Machine Learning module which downloads the \"digits\" dataset and separates it into training and testing sets. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "scrolled": true }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9f01025ef4ef4cff96fcdee80f354ea5", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=898, description='i', max=1796), Output()), _dom_classes=('widget-intera…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%matplotlib inline\n", "import matplotlib.pylab as plt\n", "import numpy as np\n", "from sklearn.datasets import fetch_lfw_people, load_digits\n", "from sklearn.model_selection import train_test_split\n", "# ATTENTION: you might need to change the above line to \"from sklearn.cross_validation import train_test_split\" if you're using an older version of sklearn\n", "\n", "sk_data = load_digits();\n", "\n", "#Cool slider to browse all of the images.\n", "from ipywidgets import interact\n", "def browse_images(images, labels, categories):\n", " n = len(images)\n", " def view_image(i):\n", " plt.imshow(images[i], cmap=plt.cm.gray_r, interpolation='nearest')\n", " plt.title('%s' % categories[labels[i]])\n", " plt.axis('off')\n", " plt.show()\n", " interact(view_image, i=(0,n-1))\n", "browse_images(sk_data.images, sk_data.target, sk_data.target_names)\n", "\n", "\n", "feature_vectors = sk_data.data\n", "class_labels = sk_data.target\n", "categories = sk_data.target_names\n", "\n", "N, h, w = sk_data.images.shape\n", "train_vectors, test_vectors, train_labels, test_labels = train_test_split(feature_vectors, class_labels, test_size=0.25, random_state=1)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0.2 0.6 0.6 ... 0.9 0.1 0.5]\n" ] } ], "source": [ "print(train_labels/10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✅ **DO THIS:** Make changes to and finish the following code to work with the \"digits\" data. Some of the work has already been done for you. Please consider the following when making changes:\n", "\n", "* For this new input to work, you need to transform the training and testing data into a format that can work with the class that was developed. Use the example from above and the functions such as ```type``` and ```shape``` to figure out how to transform the data into inputs suitable for training the Neural Network. This will be the first step before you can run the example code below.\n", "* Modify the number of Input, Output and Hidden layers to match the new problem. (I've supplied \"?\" for now, you should think about what these could/should be.)\n", "* Make sure your inputs and outputs are normalized between zero (0) and one (1). " ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "scrolled": true }, "outputs": [ { "ename": "SyntaxError", "evalue": "invalid syntax (