Assignment 1
Neural Networks and Deep Learning

CSCI 7222
Spring 2015

Assigned Jan 14
Due Jan 21

Assignment submission

Submit all assignments via CU's desire2learn system.  Submission instructions are as follows:
  1. Go to Desire2learn web site and enter identikey and password, or access through mycuinfo -> Student -> Course information -> Website
  2. Once on the desire2learn web site, select our course, which is labeled "Tpcs Nonsymbolic Ai"
  3. Select the "assessments" tab in the upper right corner of the page
  4. Select "dropbox" from the drop down menu
  5. Select "assignment 1" and then upload the file

Goal

The goal of this assignment is to introduce neural networks in terms of ideas you are already familiar with:  linear regression and linear-threshold classification.

Part 1

Consider the following table that describes a relationship between two input variables (x1,x2) and an output variable (y).

x1
 
x2
 
y
 
.1227
 
.2990
 
+0.1825
 
.3914
 
.6392
 
+0.8882
 
.7725
 
.0826
 
-1.9521
 
.8342
 
.0823
 
-1.9328
 
.5084
 
.8025
 
+1.2246
 
.9983
 
.7404
 
-0.0631

This is part of a larger data set that I created which you can download either in matlab or text format. Using matlab or your favorite software, find the least squares solution to y = w1 * x1 + w2 * x2 + b

Part 2

Using the LMS algorithm, write a program that determines the coefficients {w1,w2,b} via incremental updating and multiple passes through the training data. You will need to experiment with updating rules (online, batch, minibatch), step sizes (i.e., learning rates), stopping criteria, etc.

Be prepared to share some of the lessons you learned from these experiments with the class next week.

Part 3

Turn this data set from a regreesion problem into a classification problem simply by using the sign of y (+ or -) as representing one of two classes. In the data set you download, you'll see a variable z that represents this binary (0 or 1) class.  Use the perceptron learning rule to solve for the coefficients {w1, w2, b} of this classification problem.

Note: if you do this right, your solution to Part 3 will require only a few lines of code added to the code you wrote for Part 2.

Part 4

In machine learning, we really want to train a model based on some data and then expect the model to do well on "out of sample" data. Try this with the code you wrote for Part 3:  Train the model on the first {25, 50, 75} examples in the data set and test the model on the final 25 examples. How does performance on the test set vary with the amount of training data?