CSCE 478/878 (Fall 2008) Homework 1

Assigned Monday, September 22
Due Friday, October 10 at 11:59 p.m.

When you hand in your results from this homework, you should submit the following, in separate files:

1. A single .tar.gz or .tar.Z file (make sure you use a UNIX-based compression program) called username.tar.gz where username is your username on cse. In this tar file, put:
   • Source code in the language of your choice (in plain text files).
   • A makefile and a README file facilitating compilation and running of your code (include a description of command line options). If we cannot easily re-create your experiments, you might not get full credit.
   • All your data and results (in plain text files).
2. A single .pdf file with your writeup of the results for all the homework problems, including the last problem. Only pdf will be accepted, and you should only submit one pdf file, with the name username.pdf, where username is your username on cse. Include all your plots in this file, as well as a detailed summary of your experimental setup, results, and conclusions. If you have several plots, you might put a few example ones in the main text and defer the rest to an appendix. Remember that the quality of your writeup strongly affects your grade. See the web page on ``Tips on Presenting Technical Material''.

Submit everything by the due date and time using the web-based handin program.

On this homework, you must work on your own and submit your own results written in your own words.

1. (40 pts) Implement the ID3 algorithm from Table 3.1 (p. 56). You will train and test your algorithm on three different data sets from the UCI Machine Learning Repository. You must use both the "Vote" and "Monks I" data sets. For the third data set, you may choose any that you wish, but you should note the following when making your selection.
   
   • You will use these same three data sets in future homeworks. Thus if your chosen third data set has more than two classes, while using them in ID3 will be no problem, in future homeworks you may have to adapt your binary classifiers to work with multiclass data. This is possible, but requires a little more work, which will also result in extra credit.
   • It is better to use larger data sets (though not enormous) so you can more easily split the data into training and testing (and validation, when needed; students in 878 should see Problem 5 below) sets. You will set aside at least 30 examples per set for testing (and validation). Thus if your data set only has e.g. 50 examples, very few are left over for training, and you may not get a very good result.
   • Beware of data sets with unspecified attribute values. If you opt for such sets and cleverly handle these cases, you will receive extra credit.

Let U₁, U₂, and U₃ be your three data sets. From each set U_i remove 30 examples, placing them in set T_i (T_i will serve as the test set for experiment i). We will refer to the set of examples left over in U_i as D_i, i.e. D_i is the set of examples in U_i that are not in T_i. For each pair (D_i, T_i), do the following.

1. Choose 5 numbers evenly between 10 and |D_i| = size of D_i. Call these numbers s₁, s₂, ... s₅.
2. For j = 1, ..., 5, uniformly at random select s_j examples from D_i without replacement (i.e. do not select the same example twice). Use these examples to learn a decision tree with ID3 and test it with the test set T_i. Repeat this process three times for each j and take the average. Thus you will run 15 experiments, generating 5 average error rates, one per value of j. (Note that all 15 tests are made on the same test set; this is important to allow comparisons between different runs.)
3. Using the 5 average error rates generated in the above loops, plot a curve of test error versus size of the training set.

Thus you will end up with three plots, one per U_i, each with 5 points.

You are to submit a detailed, well-written report, with real conclusions and everything. In particular, you should answer the following questions. How did increasing the training set size influence generalization error? Did overfitting occur? If not, can you push the learner to the point of overfitting? Why or why not? In your report, you should also discuss how you randomly selected the test sets T_i and how you subsampled D_i to get the training sets. From your report, the reader should be able to get enough information to repeat your experiments, and the reader should be convinced that your methods are sound, e.g. that your subsampling methods are sufficiently random. Refer to Numerical Recipes if you have questions about simulating random processes.

Extra credit opportunities for this problem include (but are not limited to) running on extra data sets, handling continuous-valued attributes, and handling unspecified attribute values. The amount of extra credit is commensurate with the level of extra effort and the quality of your report of the results.

2. (15 pts) Do Problem 2.4 on p. 48


3. (5 pts) Do Problem 3.2 on p. 77


4. (5 pts) State how many hours you spent on each problem of this homework assignment (for CSCE 878 students, this includes the next problem).

The following problem is only for students registered for CSCE 878. CSCE 478 students who do it will receive extra credit, but the amount will be less than the number of points indicated.

5. (20 pts) Rerun your ID3 experiments with rule post-pruning, testing on the same test sets as before. You will therefore need to set aside at least 30 of your training examples for validation during pruning. Generate the same plots as before to determine the decrease in test error that rule post-pruning yields. (To make your comparisons fair with the non-pruning case, you are advised to not train on the validation set in Problem 1, i.e. build the tree on the exact same set of examples for this problem and Problem 1.)

Last modified 16 August 2011; please report problems to sscott.