Author: yileih

This week, I wired the pressure sensors on each finger and the resistors to the Arduino. I also figured out a way to fit all the parts on one hand by gluing a tiny breadboard (with wires connecting the sensors to the Arduino) onto an Arduino that can be strapped to one’s wrist with a rubber band. The wires, with the pressure sensors at the end, are wrapped around each finger, with the sensor on the fingertip secured by thin tape and a small sticker, to reduce the impact on the stability of hand detection. I added debounce to the detection algorithm so that holding a finger down slightly too long still counts as a single tap event, so it only prints once when the state changes. We tested the implementation with the newly added pressure sensors. Although we have more false positives now, the false negatives are significantly reduced. The main reason for the new false positives is that the wires and tape on the finger interfere with hand recognition, and the sticker under the fingertip affects fingertip detection. I will replace the stickers with clear tape to see whether fingertip detection is less affected. I also ran tap performance test across different lighting conditions and score thresholds.

We are on schedule. Adding the physical sensors is the additional part we decided on last week (a last minute fix for our not so sensitive tap detection).

Next week, we need to finish testing the new system that uses pressure sensors for tap detection and complete the final demo video and report. We will demo both versions: one with the pressure sensors and one with vision-based motion thresholding.

This week I shifted focus to tap detection. Building on our existing logic that marks a fingertip as a tap candidate when its vertical motion meets thresholds for downward speed, total travel distance, and a clear stop, I changed how we handle cases where multiple fingers on the same hand satisfy these conditions in the same time window. Because one finger tapping often drags neighboring fingers with similar motion, we now treat all of them as candidates but select only the finger whose fingertip ends up at the lowest vertical position as the actual tap. I also added a visual highlight on the overlaid keyboard so that whenever a tap is mapped to a key, that key briefly lights up. This makes it easier for users to see what was recognized without relying solely on the text editor output.

We are currently behind schedule, as tap detection still needs significant improvement in reducing false positives. I am exploring additional thresholding and feature checks for correct taps, as well as the relationships between fingers during a tap, to further reduce errors. I will need to finish this and start testing early next week.

Next week, I plan to continue improving the tap detection algorithm, complete our testing, and integrate the parts of my subsystem that were developed after the last integration into the main system.

Over the course of the project, most of the new things I had to learn were about calibration and keyboard overlay geometry. For tap detection (which I only started exploring this past week), I briefly read about filtering techniques in a textbook, which was a bit beyond my current level but gave me intuition for possible future improvements. The main parts I worked on over the past two months start after we already have fingertip positions. I had to map our rectangular keyboard layout onto the skewed keyboard shape in the camera view and decide which key each fingertip position should correspond to. When the overlay was not drawing correctly or an error appeared, I relied on online resources, since I had little prior experience with these kinds of coordinate transforms or the canvas and rendering APIs. I then tweaked the mapping math and compared the overlay against the physical keyboard until the calibration and key mapping behaved the way we wanted. Overall, I learned these math and graphics concepts on my own, mainly through documentation, online examples, and trial and error.

This week I added a straighten button that users can press after the keyboard overlay has frozen. During calibration, if the two index fingertips are at slightly different depths, the F-J line can be slightly tilted, and the overlay inherits that tilt. The straighten button keeps the same center and horizontal separation between F and J but snaps them to a perfectly horizontal line and recomputes the quadrilateral, so the keyboard becomes level without requiring a full recalibration. I also added a Shift toggle that controls how key labels are rendered on the overlay. By default, the keyboard shows the unshifted legends (number keys display 1-0, punctuation keys show their base symbols (-, =, [, ]), and the letter keys appear as capital letters (matching the physical Mac keyboard)). When the Shift button is enabled, the overlay keeps the letter keys the same but switches the number and symbol keys to their shifted characters (1 becomes ! and = becomes +). This ensures that the on-screen legends always match what would be typed on a real Mac keyboard, while giving us a simple way to emulate Shift (equivalent to Caps Lock in our design) without needing fingertip level modifier detection yet. After the overlay is integrated with tap detection, the text engine, when it receives a Shift (Caps Lock) keystroke, should act as the signal (replacing the temporary button) to change the overlay to the shifted keys. We are taking advantage of the virtual nature of the keyboard (the key labels can change easily) instead of shoving both shifted and unshifted symbols into a single key, which would be difficult to read.

Regarding verification for my part, most of the quantitative testing to verify that my component meets the design/use-case requirements needs to run on an integrated version. I’ve done a physical perspective test to ensure the keyboard overlay matches the physical Mac keyboard. I also ran a basic lighting test for the calibration process and the fingertip-to-key mapping under three different lighting conditions (30.3, 147.9, and 675.5 lux), which includes the 200-600 lux range we are aiming for. The key labels were correct and calibration ran smoothly. Quantitative accuracy and usability tests that checks if my component meet its design goals will be conducted later on the integrated version.

I’m a bit behind schedule on a few remaining details of the keyboard overlay, but we’ve now discussed the testing plan in more detail and can start conducting the accuracy test next week, which should put me back on schedule while I finish those smaller details.

Next week, I hope to finalize how we calculate accuracy for a 1-minute typing test (we will probably record and analyze every keypress to determine accuracy instead of relying only on the final number of characters typed) and start running the accuracy and usability tests. I also plan to change the calibration countdown so it only runs while the index fingers are visible. Right now it starts on first detection and keeps ticking even if the hands leave the frame, so when they come back the overlay freezes almost immediately. Instead, the countdown should reset when the hands disappear and restart when they are detected again. I might also add a feature to center the keyboard on the screen after calibration. I plan to add a way to adjust the key labels (and maybe the overlay color) to better fit different surface colors, lighting conditions (200-600 lux), and skin tones, so these factors don’t affect the visibility of the keyboard and thus the typing experience.

This week I focused on getting the keyboard overlay to closely match a physical Mac keyboard in perspective and on encoding that geometry into better user defaults. Using the phone stand, I set up a perspective test where I put a Mac keyboard in frame, mounted my phone with our app open at roughly a 60-degree angle, and rested my hands on the physical keyboard to run F/J calibration. I then tuned the Height and Top/Bottom sliders until the rendered overlay lined up with the real keyboard, set those tuned values as the new defaults, and adjusted the slider ranges so that the defaults sit roughly at the midpoint. While doing this, I realized my previous model only squeezed the keyboard horizontally, so I added a per-row vertical scale derived from the same top-bottom ratio: with 5 rows there are 4 steps from bottom to top, so I define a factor such that a^4=topShrink, so a=topShrink^(1/4), and the row heights become [1, a, a^2, a^3, a^4] for the number, QWERTY, ASDF, ZXCV, and modifier rows. This gives us a consistent perspective in both width and height. I tested this repeatedly with the physical keyboard and also refined the offsets between the letter key rows now that I have a physical reference. Below is a picture showing the overlay on the physical keyboard after the adjustments, you can see the overlay (specifically the number and letter keys) very closely matching the physical reference.

I am mostly on schedule for my part of the project in terms of overlay geometry and tap decision, since the perspective test and the new per-row vertical scaling are now working.

Next week, I want to hook my key-under-fingertip mapping up to Joyce’s tap detector so that instead of just labeling fingers, my module will take her tap events and return a concrete key label, which will then be fed into Hanning’s system so their component receives a stream of keypresses from my overlay. I am also considering adding a third slider that lets users adjust the rotation of the keyboard overlay (default 0 degree, horizontal), even though calibration already encodes tilt via the index fingers. In practice, it is easy for the two fingertips to be at slightly different depths, which leaves the keyboard a bit tilted, and right now the only fix is to recalibrate and carefully match fingertip depth. A small rotation slider would let users straighten the keyboard without redoing calibration once they are already happy with the size and position.

This week I made the virtual keyboard geometry match the Mac layout horizontally. I changed the layout math so that the 10 main keys in each typing row (number row, QWERTY row, A row, and Z row) all share the same column widths instead of being scaled differently per row. Before this change, rows with large peripheral keys (like tab, caps lock, or shift) would cause the letter keys in that row to shrink, so diagonals like “qaz” and “wsx” didn’t line up exactly. Now the letter/number block is fixed and the peripheral keys flex to fill the leftover space, which keeps the total row width the same. I also pulled the number row into this system so “1qaz,” “2wsx,” “3edc,” and so on are now consistent across all rows. I also updated the camera setup so that on phones and tablets we use the front camera by default instead of the back camera.

I am on schedule and finishing up the keyboard layout and key decision.

Next week I want to clean up the non-letter keys (the space row, including the arrows). I also want to use the phone/tablet stand that arrived to do a perspective test: put a printed Mac keyboard in frame, put the device on the stand, and see what top-bottom ratio and height make our rendered keyboard match the real one. From that, I can pick better defaults for users so they don’t have to drag the sliders every time (they would only adjust them when they want something different, with the default matching the real keyboard). Finally, I want to start integrating with Joyce’s part so that her tap detector uses my key under fingertip mapping now that the horizontal columns are actually correct.

This week, I added layout controls (live sliders for overall keyboard height and the top-to-bottom row ratio with a reset to the default settings). I also implemented per-finger key tracking with on-screen annotations: each fingertip (thumb through pinky) is mapped to the key it’s over in the Mac layout, and the active key label is rendered at the fingertip.

I am on schedule. The main integration challenge is aligning coordinate frames and timestamps with Joyce’s tap decision output.

Next week, I will tune the relative horizontal positions to match a Mac keyboard more precisely (I tried typing a word and the layout still felt slightly off). I will also add a small tap-testing utility for my component: on-screen buttons for each finger that, when clicked, simulate a tap and emit the key associated with the video frame at the click timestamp. My goal is to start integrating this path with Joyce’s detected taps by the end of next week. Now that the HTML build runs on our phones, several details render differently, I will fix those in the following week.

This week, I added all the remaining keys and organized them to resemble a Mac keyboard. I expanded the layout to include the full number row, punctuation, Return, Shift, Delete, Space, arrows, and the Mac modifiers.

Although I didn’t finish as much as I had hoped (I also intended to add a feature to adjust the keyboard’s vertical scale (height)), I am still roughly on schedule. The calibration and overlay are functionally complete enough that I can wrap those controls next week without slipping the overall plan. To stay on track, I’ll start with a basic scaling slider and polish it after integration.

Next week I hope to add a slider to adjust the keyboard height and another to adjust the top-bottom ratio. In parallel, I’ll start working on the tap-decision logic and outline a testing plan for my tap-decision component by itself. The goal is to validate my decision module independently, then integrate with the actual tap-detection signals that my teammate is building toward the end of week 7.

 

This week, I replaced the placeholder trapezoid with a QWERTY keyboard overlay, corrected calibration so the left and right index fingertips land exactly on the F/J key centers, ensured the video is mirrored but the labels are not, set the keyboard to the correct orientation relative to our intended hand orientation, and added a 10-second calibration window that automatically freezes the keyboard, with a recalibrate button to restart (the timer only counts down when it’s seeing both hands). 

I’m on schedule. I have made progress on calibration, defining the keyboard’s size and position from fingertip separation, and the AR overlay. 

Next week, I plan to add the remaining keys (number row, punctuation, and Shift, Enter, and Space), introduce tilt-aware horizontal and vertical scaling based on a fixed keyboard tilt (moving from “looks like perspective” to a consistent homography), and work on the image keyboard mapping (imagePointToKey function).

This week I built a file that integrates Joyce’s MediaPipe Hands component to auto-calibrate the keyboard from the left and right index fingertips (F/J) (a new method compared with last week’s four-tap approach) and renders a mirrored, perspective trapezoid outline aligned to those fingertips. I am on schedule for Tasks 1.2 and 1.3. Next week I plan to complete mapping keyboard positions to keys (Task 1.4) and finalize a basic calibration flow that locks the keyboard layout after a 10-second hold.