Status Reports

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours.

I ended up being unable to use the EfficientNet architecture because of a bug which I discussed in our weekly meeting this week. However, I retrained our colour and clothing type models using a balanced dataset and tried our the ResNet50,101, and 152 architecture. The ResNet152 performed best for clothing type classification and the ResNet101 performed the best for color classification, each reaching a validation accuracy of 65-75%. I also manually compiled a small dataset of 80 images(including at least 5 examples from each type and color class) which I used as a sanity check to ensure that our models performed well on the types of images that Style Sprout users would take. This spreadsheet includes the accuracy for the ResNet50 and ResNet152 versions of the models, and I will update it to include the accuracy for the ResNet101 after training is finished. 

I also did some troubleshooting of our usage model since it was only outputted the “sports” usage(which has now been switched to casual). My troubleshooting, which included running the 3 tests discussed in our weekly meeting, and I have determined that my next course of action is to try and determine the usage for less categories. This means determining only if clothing types like tops and skirts are formal or casual and ignoring clothing types like blazers and jeans, which are inherently either casual or formal. 

I also worked with Gabriella in person on Monday to test the uses until dirty feature. We had to make a couple of changes to our backend and frontend, which are located on our Github, but were able to successfully complete this feature. I also added the ability to display a variable # amount of pictures when an outfit is generated. Before, it only displayed a top and bottom, however now there is the ability to show outfits with 3 pieces of clothing(jacket, top, bottom), 2 pieces of clothing(jacket and dress, top and bottom), or 1 piece of clothing(dress). This is located on this branch on our Github.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

My progress is on schedule.

What deliverables do you hope to complete in the next week?

In the next week, I hope to finish the usage model and continue helping Gabriella with the backend/frontend. 

Verification Tests:

I tested the accuracy of our models in two ways. Our dataset was split into two parts, training and validation(training is 80% of the original, validation is the remaining 20%). When training each model, the validation dataset was used to measure the performance of the model. For the clothing type and color classification models, the validation accuracy ranged from 60-70% depending on the architecture that was used. Since each model is a 1-of-k classifier, where each image belongs to one of k classes, the validation accuracy is categorical accuracy, meaning # of correctly classified images/total correctly classified images. A correctly classified image is an image where the predicted class is the same as the actual class. 

I also tested the accuracy of our models using a small dataset of 80 images containing 5 examples of each of our 16 possible clothing types and about 6-10 images for each of our 12 colors. I manually compiled this dataset. I evaluated the predictions made by our models using categorical accuracy and a spreadsheet linked earlier in my status report. I haven’t finished checking the accuracy for the ResNet101 clothing type and color models since they are still training.

However, it is highly likely that none of our models will meet the 80% accuracy use case requirement we had for classification accuracy. We are supplementing this with the additional feature of having the user confirm/correct the labels produced for each piece of our user’s clothing, which should correct any incorrect labels to achieve a 100% accuracy for clothing classification in our database.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours.

This week, I worked on retraining our classification models, focusing on the clothing type classification model. I tried out the EfficientNet architecture, which from my research, outperforms ResNet while maintaining a low classification time on edge devices. I reprocessed our DeepFashion training data because it had been cropped incorrectly with this code. I then applied image augmentation to boost the training accuracy, the code is located here. I experimented with various combinations of image augmentation and architecture, and improved the performance from last week by 20% to 70% by using the EfficientNetB4. 

While training models, I tried to find an alternative to training on my local device. I looked into CMU provided machines, however they reset every night which would interfere with the training. I also looked into using Google Colab, however the free version would not let me upload a zip file of my training data. Also, the free version runtimes are limited to around 2 hours, and even sped-up training on a GPU would take longer than that. I also signed up for a free trial for Google Cloud to utilize Colab Enterprise, however it still would not allow me to upload my training data and runtimes were limited to 17 hours, which is shorter than the training time. I will ask more about alternatives to local training during my meeting next week.

I also worked on the frontend. I added functionality to the Settings button which allows the user to input how many times a piece of clothing can be worn until it becomes dirty. This included adding a popup when the Settings button is pressed, adding an integer input field to the popup, and adding an API call when the input is entered. I also added a basic API outline to the backend code which will be updated next week with functionality of changing the setting on the backend. The frontend code is located on this branch and the backend code is located here.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

My progress is on schedule.

What deliverables do you hope to complete in the next week?

Next week, I hope to start retraining the usage model. I also hope to finish the functionality for the settings button. 

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours.

This week, I finished drawing the background image of our app for the frontend. It looks like this:

I also began testing our clothing type, color, and usage models with various pictures of pieces from my own closet as well as online secondhand stores like Depop. I chose to source some pictures from Depop as we want to test our model with pieces from all possible classes. However, we don’t own enough clothing from certain categories(orange, blazers, rompers). However, I realized that we would be able to find pieces of clothing from these categories on online secondhand shops where people would take well-lit pictures of their own clothes to sell which would help supplement our testing data. It was during this testing that I realized that our usage and clothing type models struggled to achieve high accuracies in practice. For example, our model classified the below image as beige, casual, and a dress which is correct.

However, clothing type and usage classification were frequently incorrect. Our models classified a tank top as a blazer and pants as a top, while always outputting a casual usage type even for formal clothing. I realized that this was due to an incorrect training data set. I had initially misunderstood what having a “balanced” dataset means and thought it meant that you need to have enough training data for each class past a threshold.

However, I realized that the amount of data present for each class should also be equal. The accuracy of my clothing type and usage models was inflated since some classes had much more data, like tshirts and casual clothing, while the color model was trained on a dataset where each class had a similar amount of data. Since the imbalance of my dataset was severe for the clothing type, with categories like Tshirts having thousands of examples and categories like blazers having just over 100, I had to source more.

I returned to using the DeepFashion dataset. This dataset was initially insufficient for us because it didn’t label images with all the attributes appropriate(most images were missing color labels). However, it does have many labeled clothing images which works for our application since we are only classifying by color. The updated training and processing code is on our Github. I trained the ResNet50, which reached 50% accuracy. I am also training the ResNet101, which has more layers/parameters, to try and achieve a higher accuracy. If the accuracy of ResNet101 is still insufficient, I will pivot to a different architecture like EfficientNet. 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

My progress is on schedule.

What deliverables do you hope to complete in the next week?

In the next week, I hope to retrain the clothing type and usage accuracies to be more accurate. I also hope to continue helping Gabriella with the frontend and helping Riley integrate the trained models onto the Jetson.