Look who's talking!
This week I worked on many things – getting frames from the RPi camera video stream to pass to the gesture classifier, collecting more images (through crowdsourcing on social media which was a very interesting experience – we have a total of around 1200 real …
I have been playing with the microphones this week. Turns out that macOS Catalina doesn’t support Audacity, and then I tried to connect to an old laptop but the USB ports were broken, and had some other adaptor issues… also it’s super unclear if the …
This week was productive – as usual, everything took longer than expected, but we have been making significant progress.
Neeti and Gauri spent a lot of time this week on our gesture classifier. We realized that it’s more difficult to recognize gestures with the same number of fingers (thumbs up/thumbs down, point left/right), so we switched our gestures to the ‘ok’ symbol for right (middle finger, ring finger, pinky up), and the ‘L’ or ‘loser’ symbol for left (thumb and pointer) since these are clear different fingers. These gestures seemed to be better, but another problem was we were training with a Kaggle/LeapMotion dataset we found – however these images were taken with an infrared camera and were all of the same person, so they looked fairly different from the real images of a person that we want our device to respond. We tried:
… but these were unsuccessful. We are now creating our own set of data from images our friends and family, of which we have collected a pretty large set of images already (I guess everyone is really bored in quarantine…). If this doesn’t work we may have to change the CNN model we’re using.
On the hardware side of manual mode, Shrutika set up the raspberry pis on her home network and connected the raspberry pi cameras. We are able to capture images using OpenCV and the pi camera, and are working on getting a video stream frame by frame from the camera to feed the frames into the classifier. The output of the classifier and the animation are already connecting and working, so once the classifier works we should be done with manual mode.
Shrutika is working on setting up the microphones and reading from them today – we should have a clearer picture of where we are on automatic mode by tonight/tomorrow.
This week we spent a lot of time getting our manual mode ready for the demo! This meant that I was primarily working on the hand gesture classifier, Shrutika was working on animation, and Gauri was working on planning and integration. I spent Monday (04/06) …
Minor issues for manual mode so it took longer than expected, but still making significant progress. This week we decided to switch to just using pygame because it was much simpler to integrate, I already knew how to use it, and Matlab was causing a …
This week we spent time on the gesture classifier and animation for manual mode. We met multiple times to work remotely. I worked with Neeti on debugging, training and testing the gesture classifier with different online datasets. We discovered that the Kaggle dataset had a few issues which is why training on it did not help the classifier classify real images of gestures that we took of ourselves and fed in. The dataset (though large with ~4000 images) has images taken with an infrared camera and all these images are very similar and of the same person. So when we fed in images taken by us of the gestures against a solid background the classifier always predicted the same gesture.
We tried changing gestures to more obvious ones (with a clear difference in the number of fingers) – so we switched to “ok” and “L” for right and left. This did not help the classifier trained on the kaggle dataset. We tried feeding in x-ray filter images taken with PhotoBooth on the Mac and this did not work either. Another thing we tried was to train the classifier on an augmented dataset of the Kaggle images and also our own real images, but the proportion of Kaggle images was much larger and this did not help. We then attempted to use clustering techniques on the real images we fed in to separate background and foreground (using otsu binarization) but this was not super effective since the real images varied a lot. So finally for demo-ing the manual mode so far, we just fed in Kaggle images and passed the output to Shrutika’s pygame animation. This works.
We are attempting to fix the classifier now by training on a new dataset created with images taken by our friends and family. If this doesn’t work we may need to change the CNN model we are using. Shrutika tested that the camera + RPi work and I am now working on getting a video stream frame by frame from the camera to feed the frames intermittently into the classifier and make gesture predictions to send to the animation.
We had a reality check last night when we found out that we have only about 11 days till the final in-class demo. We are striving to accomplish as much as possible of our goals by then – this week will be intense. Shrutika is working on the mics now and will have a better idea of how that is going tonight. By tomorrow night, we want to have manual mode fully integrated and working and a clearer picture of the mics status for automatic mode.
This week we made a lot of progress on our manual mode – hopefully enough to have it semi-working by demo Wednesday! Neeti has been working on the neural net classifier for the hand gesture (thumbs up and thumbs down). She realized earlier this week …
This week was too busy for me personally due to midterms, assignments and moving to another apartment on campus, so I didn’t get to spend as much time as I would have liked to on the control loop. We met a couple times this week …
This week we all worked individually on our predetermined parts rather than meeting to discuss decisions.
On Monday, we met briefly to discuss our progress on our respective parts.
I spent the majority of this week working on the neural net classifier for the hand gesture recognition part of manual mode. I closely followed the tutorial that was linked in a previous status report. I used a Kaggle dataset that contained 20000 hand gesture images acquired from a leap motion sensor device that I downloaded locally. I was then able to traverse the directories and subdirectories to collect all the image paths in an array. After which, I read the images and performed some preprocessing on the images and created a numpy array of the image vectors. Finally, I used Keras and TensorFlow to train the neural net with 16000 of the images over 5 epochs with a 2-layer CNN and then tested it on 4000 of the images. The classifier was able to attain a 99% accuracy on the data from the dataset.
On Wednesday, we discussed the shortcomings of the current model and training – the accuracy was already 99.5% after one epoch, training the model took a lot of horsepower and an hour, and we realized we did not need to train the model on all 10 hand gestures provided in the dataset. We just needed to train it on the two we are using to signify the direction of rotation – thumbs up and thumbs down. Gauri also gave me input on how to save the model so we do not need to retrain it everytime we restart the device.
Since Wednesday, I modified the model to only train on 4000 images that pertain to the thumbs up and thumbs down gestures. I am also working on saving the model and creating a processing pipeline for test input from the camera modules.
This week we made a lot of progress on manual mode – Neeti was working on the gesture classifier and training the models, and I worked on creating the simulation and figuring out how we can hook it up to output from the sensors. Matlab …