This week I added a straighten button that users can press after the keyboard overlay has frozen. During calibration, if the two index fingertips are at slightly different depths, the F-J line can be slightly tilted, and the overlay inherits that tilt. The straighten button keeps the same center and horizontal separation between F and J but snaps them to a perfectly horizontal line and recomputes the quadrilateral, so the keyboard becomes level without requiring a full recalibration. I also added a Shift toggle that controls how key labels are rendered on the overlay. By default, the keyboard shows the unshifted legends (number keys display 1-0, punctuation keys show their base symbols (-, =, [, ]), and the letter keys appear as capital letters (matching the physical Mac keyboard)). When the Shift button is enabled, the overlay keeps the letter keys the same but switches the number and symbol keys to their shifted characters (1 becomes ! and = becomes +). This ensures that the on-screen legends always match what would be typed on a real Mac keyboard, while giving us a simple way to emulate Shift (equivalent to Caps Lock in our design) without needing fingertip level modifier detection yet. After the overlay is integrated with tap detection, the text engine, when it receives a Shift (Caps Lock) keystroke, should act as the signal (replacing the temporary button) to change the overlay to the shifted keys. We are taking advantage of the virtual nature of the keyboard (the key labels can change easily) instead of shoving both shifted and unshifted symbols into a single key, which would be difficult to read.
Regarding verification for my part, most of the quantitative testing to verify that my component meets the design/use-case requirements needs to run on an integrated version. I’ve done a physical perspective test to ensure the keyboard overlay matches the physical Mac keyboard. I also ran a basic lighting test for the calibration process and the fingertip-to-key mapping under three different lighting conditions (30.3, 147.9, and 675.5 lux), which includes the 200-600 lux range we are aiming for. The key labels were correct and calibration ran smoothly. Quantitative accuracy and usability tests that checks if my component meet its design goals will be conducted later on the integrated version.
I’m a bit behind schedule on a few remaining details of the keyboard overlay, but we’ve now discussed the testing plan in more detail and can start conducting the accuracy test next week, which should put me back on schedule while I finish those smaller details.
Next week, I hope to finalize how we calculate accuracy for a 1-minute typing test (we will probably record and analyze every keypress to determine accuracy instead of relying only on the final number of characters typed) and start running the accuracy and usability tests. I also plan to change the calibration countdown so it only runs while the index fingers are visible. Right now it starts on first detection and keeps ticking even if the hands leave the frame, so when they come back the overlay freezes almost immediately. Instead, the countdown should reset when the hands disappear and restart when they are detected again. I might also add a feature to center the keyboard on the screen after calibration. I plan to add a way to adjust the key labels (and maybe the overlay color) to better fit different surface colors, lighting conditions (200-600 lux), and skin tones, so these factors don’t affect the visibility of the keyboard and thus the typing experience.