10/18
This week was fall break, so progress was limited due to reduced team activity and schedule constraints. However, substantial work was completed in the prior week.
During the previous week, I made significant contributions to the design report by drafting multiple core sections, including the Introduction, Use Case Requirements, Design Requirements, parts of the Trade Studies, Software Implementation Plan, Testing and Verification Plan, and Project Management. These sections now connect system objectives with measurable performance targets and a structured development timeline.
In parallel, I began exploring the computer vision subsystem for hand recognition, specifically investigating MediaPipe’s hand-tracking framework for chord detection. I also experimented with NumPy and PyAudio to handle audio synthesis and chord manipulation, testing how note frequencies can be programmatically generated and combined to emulate realistic guitar tones. These experiments are intended to validate the feasibility of the audio rendering pipeline before integrating with the gesture-recognition module.
Overall, my progress remains on schedule, and there have been no delays or blockers. The current focus aligns with our planned milestones for this stage of the project.
Next week, I plan to continue developing the PyAudio-based sound synthesis module to generate test samples and record prototype chord sequences. This will allow for preliminary evaluation of sound quality, latency, and responsiveness before hardware integration. We will also communicate with the School of Music participants to give feedback or potentially test out our progress so far.
10/4
This week, I completed my portion of the design presentation, finalized the content and visuals for the hardware design, and system overview sections. I also set up and tested the Veremin GitHub repository on my local machine to understand how the original codebase handled the computer vision and sound synthesis. After getting it to run locally, I experimented with the structure of the code to identify which components we could reuse for our implementation. I also started working on my portions of the design report that would be worked on with more details next week.
My progress is on schedule. Finishing the design presentation and testing the Veremin repo provided a foundation for the next stage of hardware integration and firmware development. No delays have occurred so far.
Next week, I plan to begin integrating the computer vision hand-recognition with the Flask server. I will also work on the design report further. If time permits, I will start developing the data visualization and calibration tools to support initial testing.
9/27
This week, I focused on refining our project concept and researching the computer vision component in depth. Together with Taj and Alexa, we finalized the idea of using a hybrid design: MediaPipe Hands for detecting one hand’s finger/joint positions and the IMU wristband with haptic pads for strumming. My personal contribution was conducting research into how MediaPipe Hands works, including its ability to track 21 hand landmarks in real time. I evaluated how finger bends could be mapped to different chords, making the system more expressive and customizable. I also compared this approach to full-arm tracking (PoseNet), noting that MediaPipe Hands provides finer granularity and better aligns with our accessibility goals.
In addition, I attended our meetings with Professor Mukherjee, mentor Belle, Professor Dueck, and John Cohn, and documented feedback. I researched integration tools, including TensorFlow.js, Web MIDI API, Web Audio API, Tone.js, and MQTT/WebSockets, based on the “veremin” codebase for connecting hand-tracking data with real-time audio synthesis. I also created the base codebase, forked from the “veremin” project, and took a deep dive into the code.
I also created a Figma prototype of what our final project might look like.
Our team is currently on schedule. By narrowing the scope to one hand under computer vision and combining it with our hardware strumming mechanism, we reduced project complexity while still maintaining an engaging use case. If any delays arise in integrating MediaPipe Hands with the audio pipeline, I will focus on building a minimal working prototype that detects just one or two gestures first, then expand.
By next week, I plan to implement a basic MediaPipe Hands demo that detects hand landmarks from webcam input, experiment with mapping one or two finger bends to simulated chord events, and draft a short requirements list outlining how the hand-tracking data will connect to Tone.js via the Web MIDI API.
9/20
This week, I focused on developing our proposal presentation and coordinating with Jocelyn, the representative from the School of Music. Together, we finalized the use case for our project and made adjustments to the design direction.
Instead of creating an air guitar with haptic pads and a strumming wristband, we decided to pivot toward building a video theremin (“veremin”) inspired by a pre-existing project. Our main innovation will be shifting from computer vision that tracks large hand movements to a system that tracks individual finger movements. As a possible hardware extension, users could press a button to change chords while moving their hands, producing a strumming effect.
The chosen use case centers on accessibility. For individuals with injuries or disabilities who may have limited hand mobility, interacting with haptic pads could be difficult. Our new design aims to lower these barriers, making it easier for them to create and enjoy music.
Our team is currently on schedule. By collaborating and having constant communication with Jocelyn, we can develop a product that can satisfy both an engineer’s point of view and a musician’s point of view.
By next week, I plan to: research further how our project can be achieved, complete the design presentation, and develop further details of our idea.