Category: Yilei’s Status Reports

This week, I added all the remaining keys and organized them to resemble a Mac keyboard. I expanded the layout to include the full number row, punctuation, Return, Shift, Delete, Space, arrows, and the Mac modifiers.

Although I didn’t finish as much as I had hoped (I also intended to add a feature to adjust the keyboard’s vertical scale (height)), I am still roughly on schedule. The calibration and overlay are functionally complete enough that I can wrap those controls next week without slipping the overall plan. To stay on track, I’ll start with a basic scaling slider and polish it after integration.

Next week I hope to add a slider to adjust the keyboard height and another to adjust the top-bottom ratio. In parallel, I’ll start working on the tap-decision logic and outline a testing plan for my tap-decision component by itself. The goal is to validate my decision module independently, then integrate with the actual tap-detection signals that my teammate is building toward the end of week 7.

 

This week, I replaced the placeholder trapezoid with a QWERTY keyboard overlay, corrected calibration so the left and right index fingertips land exactly on the F/J key centers, ensured the video is mirrored but the labels are not, set the keyboard to the correct orientation relative to our intended hand orientation, and added a 10-second calibration window that automatically freezes the keyboard, with a recalibrate button to restart (the timer only counts down when it’s seeing both hands). 

I’m on schedule. I have made progress on calibration, defining the keyboard’s size and position from fingertip separation, and the AR overlay. 

Next week, I plan to add the remaining keys (number row, punctuation, and Shift, Enter, and Space), introduce tilt-aware horizontal and vertical scaling based on a fixed keyboard tilt (moving from “looks like perspective” to a consistent homography), and work on the image keyboard mapping (imagePointToKey function).

This week I built a file that integrates Joyce’s MediaPipe Hands component to auto-calibrate the keyboard from the left and right index fingertips (F/J) (a new method compared with last week’s four-tap approach) and renders a mirrored, perspective trapezoid outline aligned to those fingertips. I am on schedule for Tasks 1.2 and 1.3. Next week I plan to complete mapping keyboard positions to keys (Task 1.4) and finalize a basic calibration flow that locks the keyboard layout after a 10-second hold.

This week I worked on Task 1.1, surface detection and plane fitting. I integrated OpenCV.js into our project and implemented the calibration flow where the user taps four corners of the typing surface. From these taps, I now compute the homography matrix that maps image pixels to the keyboard plane. My progress is on schedule. For next week, I plan to add a minimal overlay to visualize the calibration results and begin preparing the mapping function for Task 1.4.