Team Status Update for 04/11
This week was productive – as usual, everything took longer than expected, but we have been making significant progress.
Neeti and Gauri spent a lot of time this week on our gesture classifier. We realized that it’s more difficult to recognize gestures with the same number of fingers (thumbs up/thumbs down, point left/right), so we switched our gestures to the ‘ok’ symbol for right (middle finger, ring finger, pinky up), and the ‘L’ or ‘loser’ symbol for left (thumb and pointer) since these are clear different fingers. These gestures seemed to be better, but another problem was we were training with a Kaggle/LeapMotion dataset we found – however these images were taken with an infrared camera and were all of the same person, so they looked fairly different from the real images of a person that we want our device to respond. We tried:
- using an x-ray filter to make real images look more like the dataset
- using an augmented dataset with the existing set + our own images
- clustering techniques on the real images to separate background and foreground using otsu binarization
… but these were unsuccessful. We are now creating our own set of data from images our friends and family, of which we have collected a pretty large set of images already (I guess everyone is really bored in quarantine…). If this doesn’t work we may have to change the CNN model we’re using.
On the hardware side of manual mode, Shrutika set up the raspberry pis on her home network and connected the raspberry pi cameras. We are able to capture images using OpenCV and the pi camera, and are working on getting a video stream frame by frame from the camera to feed the frames into the classifier. The output of the classifier and the animation are already connecting and working, so once the classifier works we should be done with manual mode.
Shrutika is working on setting up the microphones and reading from them today – we should have a clearer picture of where we are on automatic mode by tonight/tomorrow.