The hardware for the wearable is completed and a testbed for the physical sensor configuration and MQTT sensor data streaming have been fully set up! The device reads from all 10 sensors in a round-robin fashion (to prevent IR interference), applies a EWMA filter to the sensor readings, and sends the data to MQTT to be received by a Python program or a website. The device can also send power information like battery percentage and voltage. After this was finished, we moved on to thinking about the webapp and gesture recognition while Edward works on polishing up the hardware.
For gesture recognition, we made a lot of progress since receiving the sensors. We began by just observing the general behavior of the sensors and brainstorming ideas on how we can map them to different gestures. We noticed a few mathematical characteristics of the zooming gestures that were different from the characteristics of a swipe, so we will most likely use those as our gesture identification. We have a few theories as to how we can distinguish those further, and those will be our next step.
For the Unity-Web Application part of our project, we started working on communication of data. In order to prepare for integration, we made sure that data could be streamed to our web application via the MQTT protocol. For now, we just sent over dummy data from a python script via the protocol above. Then we made sure that we can actually communicate smoothly with the Unity interface that is embedded onto the Django web app. Now we are continuing to work on how to translate the gesture data into more precise gestures applied to the model.
The next steps are to build out the actual wearable (buy wrist straps, create enclosure for hardware, attach hardware to strap, etc.). We also need to flesh out gesture recognition and make it more robust to the various kinds of ways to perform specific gestures like a swiping left and right or a pinching in and out.
Our biggest concern still remains to be the Jetson. However, since we know the connection between the device and our computers work, we at least have one avenue for communication. That will most likely be our contingency plan, but we’re still hoping that the Jetson will work for faster computation purposes. Our schedule still remains the same, and we’re planning on beginning integration over the next week.