This week I shifted focus to tap detection. Building on our existing logic that marks a fingertip as a tap candidate when its vertical motion meets thresholds for downward speed, total travel distance, and a clear stop, I changed how we handle cases where multiple fingers on the same hand satisfy these conditions in the same time window. Because one finger tapping often drags neighboring fingers with similar motion, we now treat all of them as candidates but select only the finger whose fingertip ends up at the lowest vertical position as the actual tap. I also added a visual highlight on the overlaid keyboard so that whenever a tap is mapped to a key, that key briefly lights up. This makes it easier for users to see what was recognized without relying solely on the text editor output.
We are currently behind schedule, as tap detection still needs significant improvement in reducing false positives. I am exploring additional thresholding and feature checks for correct taps, as well as the relationships between fingers during a tap, to further reduce errors. I will need to finish this and start testing early next week.
Next week, I plan to continue improving the tap detection algorithm, complete our testing, and integrate the parts of my subsystem that were developed after the last integration into the main system.
Over the course of the project, most of the new things I had to learn were about calibration and keyboard overlay geometry. For tap detection (which I only started exploring this past week), I briefly read about filtering techniques in a textbook, which was a bit beyond my current level but gave me intuition for possible future improvements. The main parts I worked on over the past two months start after we already have fingertip positions. I had to map our rectangular keyboard layout onto the skewed keyboard shape in the camera view and decide which key each fingertip position should correspond to. When the overlay was not drawing correctly or an error appeared, I relied on online resources, since I had little prior experience with these kinds of coordinate transforms or the canvas and rendering APIs. I then tweaked the mapping math and compared the overlay against the physical keyboard until the calibration and key mapping behaved the way we wanted. Overall, I learned these math and graphics concepts on my own, mainly through documentation, online examples, and trial and error.