Author: yileih

This week, I added layout controls (live sliders for overall keyboard height and the top-to-bottom row ratio with a reset to the default settings). I also implemented per-finger key tracking with on-screen annotations: each fingertip (thumb through pinky) is mapped to the key it’s over in the Mac layout, and the active key label is rendered at the fingertip.

I am on schedule. The main integration challenge is aligning coordinate frames and timestamps with Joyce’s tap decision output.

Next week, I will tune the relative horizontal positions to match a Mac keyboard more precisely (I tried typing a word and the layout still felt slightly off). I will also add a small tap-testing utility for my component: on-screen buttons for each finger that, when clicked, simulate a tap and emit the key associated with the video frame at the click timestamp. My goal is to start integrating this path with Joyce’s detected taps by the end of next week. Now that the HTML build runs on our phones, several details render differently, I will fix those in the following week.

This week, I added all the remaining keys and organized them to resemble a Mac keyboard. I expanded the layout to include the full number row, punctuation, Return, Shift, Delete, Space, arrows, and the Mac modifiers.

Although I didn’t finish as much as I had hoped (I also intended to add a feature to adjust the keyboard’s vertical scale (height)), I am still roughly on schedule. The calibration and overlay are functionally complete enough that I can wrap those controls next week without slipping the overall plan. To stay on track, I’ll start with a basic scaling slider and polish it after integration.

Next week I hope to add a slider to adjust the keyboard height and another to adjust the top-bottom ratio. In parallel, I’ll start working on the tap-decision logic and outline a testing plan for my tap-decision component by itself. The goal is to validate my decision module independently, then integrate with the actual tap-detection signals that my teammate is building toward the end of week 7.

This week, I replaced the placeholder trapezoid with a QWERTY keyboard overlay, corrected calibration so the left and right index fingertips land exactly on the F/J key centers, ensured the video is mirrored but the labels are not, set the keyboard to the correct orientation relative to our intended hand orientation, and added a 10-second calibration window that automatically freezes the keyboard, with a recalibrate button to restart (the timer only counts down when it’s seeing both hands). 

I’m on schedule. I have made progress on calibration, defining the keyboard’s size and position from fingertip separation, and the AR overlay. 

Next week, I plan to add the remaining keys (number row, punctuation, and Shift, Enter, and Space), introduce tilt-aware horizontal and vertical scaling based on a fixed keyboard tilt (moving from “looks like perspective” to a consistent homography), and work on the image keyboard mapping (imagePointToKey function).

This week I built a file that integrates Joyce’s MediaPipe Hands component to auto-calibrate the keyboard from the left and right index fingertips (F/J) (a new method compared with last week’s four-tap approach) and renders a mirrored, perspective trapezoid outline aligned to those fingertips. I am on schedule for Tasks 1.2 and 1.3. Next week I plan to complete mapping keyboard positions to keys (Task 1.4) and finalize a basic calibration flow that locks the keyboard layout after a 10-second hold.

This week I worked on Task 1.1, surface detection and plane fitting. I integrated OpenCV.js into our project and implemented the calibration flow where the user taps four corners of the typing surface. From these taps, I now compute the homography matrix that maps image pixels to the keyboard plane. My progress is on schedule. For next week, I plan to add a minimal overlay to visualize the calibration results and begin preparing the mapping function for Task 1.4.