CSCE 478/878 (Fall 2012) Homework 1

Assigned Monday, September 10
Due Friday, September 28 Sunday, September 30 at 11:59 p.m.

When you hand in your results from this homework, you should submit the following, in separate files:

1. A single tarball or zip file called username.tar.gz or username.zip, where username is your username on cse. In this tar file, put:
   • Source code in the language of your choice (in plain text files).
   • A makefile and a README file facilitating compilation and running of your code (include a description of command line options). If we cannot easily re-create your experiments, you might not get full credit.
   • All your data and results (in plain text files).
2. A single .pdf file with your writeup of the results for all the homework problems. Only pdf will be accepted, and you should only submit one pdf file, with the name username.pdf, where username is your username on cse. Include all your plots in this file, as well as a detailed summary of your experimental setup, results, and conclusions. If you have several plots, you might put a few example ones in the main text and defer the rest to an appendix. Remember that the quality of your writeup strongly affects your grade. See the web page on ``Tips on Presenting Technical Material''.

Submit everything by the due date and time using the web-based handin program.

On this homework, you must work on your own and submit your own results written in your own words.

1. (10 pts) Do Problem 2.7 on p. 48.


2. (10 pts) Do Problem 3.1 on p. 77.


3. (10 pts) Do Problem 3.2 on p. 77. Show your work.


4. (40 pts) Implement the ID3 algorithm from Table 3.1 (p. 56). You will train and test your algorithm on three different data sets from the UCI Machine Learning Repository. You must use both the "Congressional Voting Records" and "Monks I" data sets. For the third data set, you may choose any that you wish, but you should note the following when making your selection.
   
   • You will use these same three data sets in future homeworks. Thus if your chosen third data set has more than two classes, while using them in ID3 will be no problem, in future homeworks you may have to adapt your binary classifiers to work with multiclass data. This is possible, but requires a little more work, which will also result in extra credit.
   • It is better to use larger data sets (though not enormous) so you can more easily split the data into training and testing (and validation, when needed; students in 878 should see Problem 5 below) sets. You will set aside at least 30 examples per set for testing (and validation). Thus if your data set only has e.g. 50 examples, very few are left over for training, and you may not get a very good result.
   • Beware of data sets with unspecified attribute values. If you opt for such sets and cleverly handle these cases, you will receive extra credit.

Let U₁, U₂, and U₃ be your three data sets. From each set U_i remove 30 examples, placing them in set T_i (T_i will serve as the test set for experiment i). We will refer to the set of examples left over in U_i as D_i, i.e. D_i is the set of examples in U_i that are not in T_i. Use D_i to train a decision tree with your ID3 implementation, and test that tree on T_i. Report the accuracies for each data set in your writeup.

You are to submit a detailed, well-written report, with real conclusions and everything. In particular, you should attempt to answer the following questions. How large of a tree did ID3 produce for each learning problem? Did overfitting occur? Would pruning the tree reduce overfitting? In your report, you should also discuss how you randomly selected the test sets T_i. From your report, the reader should be able to get enough information to repeat your experiments, and the reader should be convinced that your methods are sound, e.g. that your sampling methods are sufficiently random. Refer to Numerical Recipes if you have questions about simulating random processes.

Extra credit opportunities for this problem include (but are not limited to) running on extra data sets, varying the training set size to gauge overfitting, handling continuous-valued attributes, and handling unspecified attribute values. The amount of extra credit is commensurate with the level of extra effort and the quality of your report of the results.




The following problem is only for students registered for CSCE 878. CSCE 478 students who do it will receive extra credit, but the amount will be less than the number of points indicated.

5. (25 pts) Rerun your ID3 experiments with rule post-pruning, testing on the same data sets as before. You will therefore need to set aside at least 30 of your training examples for validation during pruning. Report accuracies as before to determine the decrease in test error that rule post-pruning yields. (To make your comparisons fair with the non-pruning case, you are advised to not train on the validation set in Problem 1, i.e. build the tree on the exact same set of examples for this problem and Problem 1.)

Last modified 26 September 2012; please report problems to sscott.