CSCE 478/878 (Fall 2001) Homework 2

Assigned Friday, October 12
Due Monday, November 5 at 11:59:59 p.m. Now due Tuesday, November 6 at 11:59:59 p.m.

When you hand in your results from this homework, you should submit the following, in separate files:

1. Source code in the language of your choice (in plain text files).
2. A makefile and a README file facilitating compilation and running of your code (include a description of command line options). If we cannot easily re-create your experiments, you might not get full credit.
3. All your data (in plain text files).
4. Your writeup (in pdf format) of the results for all the homework problems, including the last problem (only pdf will be accepted, and you should only submit one pdf file). Include all your plots in this file, as well as a detailed summary of your experimental setup, results, and conclusions. If you have several plots, you might put a few example ones in the main text and defer the rest to an appendix. Remember that the quality of your writeup strongly affects your grade. See the web page on ``Tips on Presenting Technical Material''.

1. (10 pts) Do Problem 4.2 on p. 124

2. (15 pts) Do Problem 4.5 on p. 124

3. (10 pts) Do Problem 5.3 on p. 152

4. (50 pts for 478, 65 pts for 878) You are to implement a GD-based and/or an EG-based backpropagation algorithm (478 students only need to do one, and the second is extra credit; 878 students must do both). You will then train and test your algorithm on three different data sets from the UCI Machine Learning Repository. You may choose any data sets that you wish, but you should note the following when making your selections.
   • If you choose data sets with more than two classes, you will have to adapt your binary classifier to work with multiclass data. This is possible, but requires more work, which will also result in extra credit.
   • If some features are symbolic, you will need to represent them numerically as inputs to Perceptron and Winnow.
   • It is better to use larger data sets so you can more easily split the data into training and testing sets. Otherwise you must use cross-validation.
   • Beware of data sets with unspecified attribute values. If you opt for such sets and cleverly handle these cases, you will receive extra credit.

   Split each data set into independent training and test (validation) sets. Then choose three different network architectures, and train each on the training set and test on the test set. The testing should be done after every round of training, so you should generate plots like those in Figure 4.9 on p. 110. Also, for each error value plotted, compute its 95% confidence interval (you may incorporate the interval into the plot if you wish, which will improve readability). Repeat this for three values of the learning rate. Make sure that the three architectures and three values of the learning rate all differ significantly, allowing for a very broad range of experiments. Thus, 478 students will run 3 x 3 x 3 = 27 experiments (3 data sets, 3 architectures, 3 learning rate values) and 878 students will run 54.

   You are to submit a detailed, well-written report, with real conclusions and everything. In particular, you should answer the following questions. Did training error go to 0? When? Did overfitting occur? When should you have stopped training? Of course, this is merely the minimum that is required in your report.

   Extra credit opportunities include (but are not limited to) running on extra data sets, using other activation functions, using multiclass data sets, handling unspecified attribute values, and running experiments on more architectures and with more learning rates. As always, the amount of extra credit is commensurate with the level of extra effort and the quality of your report of the results.

   NOTE: When implementing EG, you need to be able to represent negative weights.

5. (5 pts) State how many hours you spent on each problem of this homework assignment.

Last modified 16 August 2011; please report problems to sscott AT cse.