Our project, HoloKeys, is a virtual keyboard system that lets you type on any flat surface without carrying a physical keyboard. Using only the camera on a phone or tablet, a browser-based app tracks your hands and maps each fingertip to a key in a virtual keyboard layout rendered on the device screen. After a brief auto-calibration step that aligns the overlay to the desk using the user’s index fingers on F and J, the keyboard locks in, and the labels for the keys under each fingertip are shown in real time so users can see exactly where they are typing.
Our original goal was a purely vision-based system that detects both fingertip positions and tap events from camera motion alone. In practice, tap detection using only computer vision was unreliable: the system often missed valid taps while still occasionally reacting to glitches. To address this, we redesigned HoloKeys into a hybrid system. Vision is still responsible for tracking fingertip locations, but tap timing can now be provided by thin-film pressure sensors attached to each fingertip and connected to an Arduino. When a finger presses down, the sensor spike triggers a tap event, and the app looks up which key that finger was on at that moment.
The current HoloKeys system supports two modes: a pure vision mode that works with just the camera, and a hardware-assisted mode that combines camera-based fingertip tracking with pressure-sensor tap detection. In our tests, the sensor-based mode dramatically reduces missed taps compared to the vision-only approach, making typing much more reliable.