This week I explored a new hardware-assisted tap detection method for our virtual keyboard. Instead of relying solely on camera-based motion, we connected pressure-sensitive resistors to each fingertip and fed the signals into an Arduino. Initially, we could only see the pressure values in the Arduino Serial Monitor, and I wasn’t sure how to turn that serial output into something JavaScript could use. To solve this, I implemented a new module, pressure.js, that uses the Web Serial API to read the Arduino’s serial stream directly in the browser. The Arduino sends simple newline-delimited messages like “sensorIndex,state” (e.g., 1,tapped), and pressure.js parses each line into a structured event with handIndex, sensorIndex, fingerName, fingerIndex, and state.
Within pressure.js, I added helper logic to normalize different tap labels (tapped, tap, pressed, 1, etc.) into a single “tap” event and then invoke user-provided callbacks (onState and onTap). This lets the rest of our web framework treat Arduino pressure events exactly like our previous tap detection output: we can map fingerIndex to the existing Mediapipe fingertip landmarks and reuse the same downstream pipeline that maps fingertips to keys and updates the text editor. In other words, Arduino signals are now fully integrated as JavaScript events/variables instead of being stuck in the serial monitor.
Schedule change is turning my slack week into implementing this pressure sensor solution.