CSCE 478/878 (Fall 2003) Homework 2

Assigned Monday, October 27
Due Monday, November 17 at 11:59:59 p.m.

When you hand in your results from this homework, you should submit the following, in separate files:

1. A single .tar.gz or .tar.Z file (make sure you use a UNIX-based compression program) called username.tar.gz where username is your username on cse. In this tar file, put:
   • Source code in C, C++, or Java (in plain text files).
   • A makefile and a README file facilitating compilation and running of your code (include a description of command line options). If we cannot easily re-create your experiments, you might not get full credit.
   • All your data and results (in plain text files).
2. A single .pdf file with your writeup of the results for all the homework problems, including the last problem. Only pdf will be accepted, and you should only submit one pdf file, with the name username.pdf, where username is your username on cse. Include all your plots in this file, as well as a detailed summary of your experimental setup, results, and conclusions. If you have several plots, you might put a few example ones in the main text and defer the rest to an appendix. Remember that the quality of your writeup strongly affects your grade. See the web page on ``Tips on Presenting Technical Material''.

1. (15 pts) Do Problem 4.5 on p. 124

2. (10 pts) Do Problem 5.3 on p. 152

3. (50 pts) Implement one of the following classifiers using the Lasso API:
   1. An artificial neural network (ANN) with one hidden layer. The input layer will have n nodes (i.e. there are n features per example) and the hidden layer will have k nodes. Here, n and k are parameters passed to the learner (i.e. do not hard-code these values into your program). The number of output nodes may be fixed at 1 if you wish, though if you plan to handle multi-class data, you should make this dynamic as well. Your ANN will be trained by the Backpropagation algorithm, either GD- or EG-based. (If you implement EG-based backpropagation, you will receive extra credit. Note that in this case you need to be able to represent negative weights.)

   2. A support vector machine (SVM) with the kernel of your choice. For simplicity, you may choose the polynomial kernel on slide 40 (on slide 40, the degree d of the kernel is 2; you may choose any fixed d that you wish), or any other from the SVM handout (e.g. Gaussian kernel). You need not worry about formulating the problem as a quadratic program, nor must you even try to maximize the margin. Instead, if you wish you may use the dual form of the standard Perceptron algorithm.

   You are to plug your classifier into Lasso and train and test on the web data that we provide you. There are three labelings of the data, so you are to run three different experiments, saving three different classifiers. Further, you are to run your ID3 implementation from Homework 1 on these same data sets. (If you do not have a fully functional implementation from Homework 1, you may use Chris Hammack's version instead.) Thus you will run six different experiments and you are to compute 95% confidence intervals for each experimental result.

   You are to submit a detailed, well-written report, with real conclusions and everything. In particular, you should answer the following questions for both your new classifier and ID3. Did training error go to 0? Did overfitting occur? Should you have stopped training early? Was there a statistically significant difference between the performance of ID3 and that of the ANN/SVM? Was there a statistically significant difference in performance of the same algorithm on different labelings of the data? What algorithm would you recommend for web page classification in general? Of course, this is merely the minimum that is required in your report.

   Extra credit opportunities include (but are not limited to) running on extra data sets, using other activation functions, using this multiclass data set, and running experiments on more ANN architectures/SVM kernels and/or with more learning rates. As always, the amount of extra credit is commensurate with the level of extra effort and the quality of your report of the results.

4. (5 pts) State how many hours you spent on each problem of this homework assignment. For CSCE 878 students, this includes the next problem.

   The following problem is only for students registered for CSCE 878. CSCE 478 students who do it will receive extra credit, but the amount will be less than the number of points indicated.

5. (20 pts) Run your classifier from Problem 3 on the three UCI data sets you used with ID3 on Homework 1, compute 95% confidence intervals for all runs (including those from Homework 1), and compare your SVM/ANN to ID3 with a paired t test. (Note that this requires running ID3 on new partitions of your UCI data. If you had problems with your ID3 implementation or had questionable results, you should either repair your implementation or use Chris Hammack's version.) For your experiments, you need to try at least two distinct ANN architectures or two SVM kernels. Similar to Problem 3, you are to submit a detailed report on your results.

Last modified 16 August 2011; please report problems to sscott AT cse.