Form homework groups of size 2–3 and report the members of your group, including full names and cse usernames. This is to be submitted by Saturday, September 21, by 11:59pm (same due time as the hack-a-thon) via Piazza, under the "Team Membership" post.
In this part of the homework, your group will work with a variation of the MNIST dataset called Fashion-MNIST, which is meant to serve as a direct drop-in replacement for MNIST. You will take data as numpy arrays, partition it into separate training and testing sets, use this to train and evaluate your models, and report the results.
Your trained network will take as input a feature vector of dimension 784 (corresponding to the pixel values of 28×28 images), each an integer from 0–255. The class labels are in the following table.
| Label Value | Meaning |
| 0 | T-shirt/top |
| 1 | Trouser |
| 2 | Pullover |
| 3 | Dress |
| 4 | Coat |
| 5 | Sandal |
| 6 | Shirt |
| 7 | Sneaker |
| 8 | Bag |
| 9 | Ankle boot |
Here are some sample images:
Design and implement at least two architectures for this problem. You may vary the number and sizes of the layers, but you must use at least one hidden layer and you must use fully connected layers with ReLU or a variation (not convolutional nodes) for all hidden nodes and softmax for the output layer. You will measure loss with cross-entropy after mapping the class labels to one-hot vectors.
The data is on crane in the folder
/work/cse479/shared/homework/01The folder has two numpy files: fmnist_train_data.npy and fmnist_train_labels.npy.
The numpy array provides all the data available to you for this assignment. It is up to you to partition it into (1) a single training set and a single testing set, or (2) k subsets for k-fold cross validation. Option (1) is quicker and easier, but option (2) provides a more thorough analysis of your model, and might give you an edge in the competition. Whichever option you choose, you must ensure that you do not test on the training set.
For each of your architectures, you will use Adam to optimize on the training set and then you will test on the test set. For each training run, you will use at least two sets of hyperparameters. You must also choose a regularizer and evaluate your system's performance with and without it. Thus, you will perform at least 2×2×2=8 training runs. (You will run a factor of k times more if you do k-fold cross validation.)
You are to submit a detailed, well-written report, with conclusions that you can justify with your results. Your report should include a description of the learning problem (what is to be learned), a detailed description of your architecture, activation functions, and regularizer, the values of the hyperparameters you tested, and how you partitioned your data. All these design decisions must be justified. You should then describe your experimental results, including a confusion matrix and basic confidence intervals on the probability of generalization error, and draw conclusions from your results (conclusions must be justified by your presented results). In particular, you should discuss the impact that your hyperparameter settings and regularizer had on performance, including overfitting.
As part of your submission, you will include files representing your best model of the ones that you created and tested. After the deadline, we will evaluate each team's best model on a held-out data set (separate from the file you will access on crane). Bonus points will be awarded to the three teams with the best submitted models, as measured by classification accuracy. To help you determine how good your model is relative to your competitors, each night we will evaluate on our data set each team's submitted model and post the accuracies. These evaluations will begin on Monday, September 23, and will be done daily until the homework deadline.
Submission Requirements
To submit your program, you must submit the following three files:
- main.py: Code to specify and train the TensorFlow graph
- model.py: TensorFlow code that defines the network
- util.py: Helper functions
To sumit your model for the competition, you must submit the following three files:
- homework_1.data-00000-of-00001
- homework_1.index
- homework_1.meta
You should use this command to save the files:
saver.save(session, "homework_1")Your model must use the following two tensors:
- input_placeholder with shape=[None,784]
- output with shape=[None,10]
Also, submit a file called team.txt with your team name as you want it to appear in the competition standings.
Finally, you must submit your report as username1-username2.pdf, where username1 and username2 are your team members' user names on cse.
Tips & Tricks
- The Fashion-MNIST data is has integer values in the range 0–255.
You might find it useful to normalize these values to the real interval [0,1].
Note that, if you train your model on the normalized values, then you should do
the normalization inside your model. I.e., when we test your submitted models,
we will give it as input data in the original format of 0–255.
Normalization within the model can be done by
x = x / 255.0
where x is the tensor.