This week I worked on many things – getting frames from the RPi camera video stream to pass to the gesture classifier, collecting more images (through crowdsourcing on social media which was a very interesting experience – we have a total of around 1200 real images in our dataset I believe) to train the classifier, wrote a script to relabel the data easily and most importantly boosting the classifier’s validation accuracy from ~80% to 90.36% when Neeti last ran it yay!

I tried quite a few things to improve our accuracy:

• Adding more images to the old model – did not really help beyond a point, it would overfit on the training dataset and not increase beyond 81% at best
• Adding regularization to the old model to reduce the overfitting by adding Dropout
• Tweaking the hyperparameters of the old model (batch size, adaptive learning rate on keras, different error calculation algorithm (adam turned out to be the best idea), training:test ratio, etc) – this finally maxed out the classifier’s validation accuracy at around 85%
• I had an idea that the variations in the backgrounds of the images (plentiful since we got these images from SO many different people) was causing the validation accuracy to be low.  So my goal was to eliminate the background entirely in each image and only feed in the hand portion to the net.  I implemented the skin detection algorithm using online resources and figured out the correct HSV color thresholds that would work best.  This algorithm basically does a type of clustering by finding contours in the image based on the thresholds.  It ended up working really well – I checked that it worked on our images.
• I also thought that it might be a good idea to turn the result of the skindetector into full black and white using Otsu binarization (something I learned in a comp bio class last sem for counting cells in a microscope image) – black background and white hand to increase the uniformity among the images being fed in.
• However, the above two steps did not help with the old neural net model and it stayed maxed out, fluctuating between 80-85% on average.  I then decided that it would be worth it to try a different neural net architecture.
• I researched common CNN architectures used for industrial gesture classifiers and discovered an architecture called VGG16 that is apparently the best.  However it is also huge and ends up learning about 19 million weights total.  This kind of net is built for large scale data processing and run on powerful GPUs with high compute and large memory availability.  It is not possible to run in on a normal laptop/RPi.  That being said, it was still the best architecture available.  I discovered that using a subset of the VGG16 layers and reducing some node counts in the dense layers  but following a similar pattern was effective enough for our purpose.
• Finally, using this model + skindetector output (without otsu binarization which didn’t help much) on 923 images at the time gave a consistent validation accuracy of 87.88% over multiple runs.
• After adding more images today, Neeti ran it again and it hit 90%

Crowdsourcing was a wild experience.  Overall I think it has been a highly productive week and we are on track. 🙂

This week we met on Monday to research what to do about the microphones and finally ordered 4 cheaper omnidirectional ones.  We decided that making baffles around these would be a better idea than buying 4 super expensive cardioid singing mics.  On Wednesday we met with Professor Sullivan and Jens to go over our progress, SOW and Design Review Report.  On Friday we met again to discuss our individual progress’ so far and work together.  I worked this week on planning and writing the outline and pseudocode of the main control loop.

Overall, we seem to be on track to demo-ing manual mode by demo week.  Next week, once Neeti finishes the classifier, I can integrate that with the control loop.   It might be tricky to get outputs sent from the control loop running on the RPi to Simulink running on the laptop so that might not be ready for demo but I hope to figure out next week at least how that can happen.

This week we spent most of our time on our design review presentation, finalizing parts and design decisions regarding interface communications, etc.  We met a couple times last weekend and once during the week to get these done and help Neeti rehearse.  Yesterday we met again to chalk out our design review report and get some of the things we could do before receiving parts done.  I setup a git repo and made sure we could all push and commit from both Pis.  Then I did some research into datasets available for gesture detection and found these: https://lttm.dei.unipd.it/downloads/gesture/ and http://www.nlpr.ia.ac.cn/iva/yfzhang/datasets/egogesture.html.  Because our gestures aren’t readily available, we might need to manufacture our own (this might not be enough data to train a neural net, but it also might be since we only basically need two gestures).  In our peer feedback, some people brought up the point that CNN might not be the most accurate model to do gesture detection and so I emailed Professor Savvides who taught us Pattern Recognition Theory last semester and would be the best person to advise us on this.  Then, I worked with Shrutika on getting a TCP connection between the Pis working and processing keystrokes using Pygame.

I think we are a little behind schedule because we had aimed to get our manual mode working by spring break, but spring break is next week.  We thought that since we’d ordered our parts last Monday, we’d have them by Friday, but there seems to have been a slight delay in ordering.  We will probably be able to get our manual mode (without gesture detection) working by this Friday if our parts arrive by Wednesday.  If not, I’m hoping we get the parts by the end of the week and then we might be able to realistically get manual mode working by the end of the first week after spring break.

I have two onsites though and will be flying out this Wednesday.  I will try to work on our project (specifically the ML models) a bit during spring break since I’m back on March 10th.  I’m not too worried about us catching up on manual mode, but I do think we have a significant amount of experimentation to do with the microphones and that we should get started on the in parallel in order to save time.