The main risk is the reliability of hand detection using MediaPipe. When the palm is viewed nearly edge-on (appearing more like a line than a triangle), the detected position shakes significantly or the hands may not be detected at all, which threatens accurate fingertip tracking. To manage this, we are tilting the camera to improve hand visibility and applying temporal smoothing to stabilize landmark positions. We have also recently updated the design to incorporate vibration detection into the tap detection pipeline, since relying on vision alone can make it difficult to distinguish between hovering and actual keystrokes.
Part A public health, safety or welfare
Our product supports public welfare by improving accessibility and comfort in digital interaction. Because it operates entirely through a camera interface, it can benefit users who find it difficult to press down physical keys due to mobility, dexterity, or strength limitations. By requiring no additional hardware or forceful contact, the system provides a low-effort and inclusive way to input text. In terms of psychological well-being, all processing is performed locally on the user’s device, and no video frames or images are stored or transmitted. This protects personal privacy and reduces anxiety related to data security or surveillance. By combining accessibility with privacy-preserving design, the system enhances both the welfare and peace of mind of its users.
Part B social factors
Our virtual keyboard system directly addresses the growing social need for inclusive, portable, and accessible computing. In many educational and professional settings—such as shared classrooms, libraries, and public workspaces—users must often type on the go without carrying physical hardware, which may be costly, impractical, or socially disruptive. By enabling natural typing on any flat surface, our design reduces barriers for mobile students, low-income users without access to external peripherals. For example, a commuter can take notes on a tray table during a train ride, or a student with limited finger dexterity can type with adaptive finger placement during lectures. Socially, this technology supports a more equitable digital experience by removing dependency on specialized devices, promoting inclusivity in both educational and workplace contexts. Moreover, it also respects users’ privacy by running entirely on-device and not transmitting camera data to the cloud.
Part C economic factors
As a web app, HoloKeys meets the need for portable, hardware-free typing while minimizing costs for both users and providers. Users don’t buy peripherals or install native software, they simply open a URL. This shifts total cost of ownership away from hardware toward a service with negligible marginal cost. This lowers adoption barriers for students, travelers, and anyone for whom carrying a keyboard is impractical. Additionally, HoloKeys may modestly substitute for portable Bluetooth keyboards but is largely complementary to laptops; its primary use cases are phone-and-tablet-first contexts where a full laptop is unnecessary or inconvenient.
Part A was written by Joyce Zhu, part B was written by Hanning Wu and part C was written by Yilei Huang.