Summary. I spent this week developing a first pass of our Computer Vision contact detection algorithm and the glove design. As a reminder, this algorithm is meant to create a “best guess” of which object the user may be potentially touching and/or holding. To this end, the algorithm should first detect all possible objects in the scene, their classified types, and positions in the frame; next, it should detect the position of the user’s hand (the glove); finally, it should provide an estimate for which object is closest to the glove.
Glove Design. I came up with an initial design for our glove, mostly as a proof of concept for tracking the glove (see below image). I taped a 1.7cm x 1.7cm arUco marker to a piece of cardboard, which is then attached to the back of the glove with a wire. This is far from ideal, and I will need to figure out a better way to attach these markers onto the fabric of the glove.
Contact Detection Algorithm. As discussed in our design report, we are detecting objects using a Single Shot Detector deep neural network; presently, we are utilizing a pre-trained model provided by the Jetson-Inference library, which can detect objects such as keyboards, computer mice, people, etc. We are detecting the glove using fiducial markers (arUco tags) that are printed and attached to the glove. Using the OpenCV library, we identify the position and orientation of these markers to generate a bounding area around the glove. The algorithm then decides which object is most likely being touched by its center’s proximity to this glove bounding area. Below are several images depicting a rudimentary glove design and the current implementation in practice. I have already identified some issues with the current implementation, such as the glove markers being hard to detect when the user moves their hand quickly producing motion blur. I will need to do more testing and iteration, as expected.
Schedule. As mentioned in our team report this week, we have made some changes to our schedule. For the computer vision component, my goal this week was to finish a first pass over the contact detection algorithm using a basic, circular bounding area shape and the pre-trained neural network model. This has been achieved. My plan for next week is to begin iterating on the design of the glove by determining optimal locations for the fiducial markers and how to best attach them to the fabric. One idea we have floating around is to 3D print a base that we might hand-paint and glue onto the glove. I also plan to work with Harry this week to begin the process of outputting the predicted contact objects over our communications protocol.
The rudimentary glove design with an attached arUco marker.
A keyboard (pink), computer mouse (orange) and glove are detected.
Potential contact with the mouse is detected.
Potential contact with the keyboard is detected.