This week, our main progress came from a meeting with Jocelyn and John Cohn, the creator of the Veremin. Together, we refined both our use case and design direction. The updated use case centers on people with limited mobility who can still bend their fingers or touch two fingers together. Our design now involves one hand for chord selection (where customizable finger bends trigger specific chords detected by CV) and the other for strumming, detected via haptic pads on the thumb and pointer finger, along with a wristband for strumming motion.
After discussion, we decided that in addition to computer vision, we will also focus on IMU-based sensing. This approach is more inclusive since it allows users to position their hand or arm comfortably by their side rather than raising it into view of a camera for strumming. John also suggested that, in addition to hardware, we could expand the scope of our project through software: the Veremin is a MIDI controller, and MIDI could be used to map to different instrument sounds or even integrate with a software synthesizer to minimize coding. He also highlighted the potential of using frameworks like Posenet or OpenPose, importing pre-trained models via JavaScript/Node, and exploring devices such as Ultraleap for hand tracking. Beyond instrument emulation, he noted possibilities like digitizing singing and using gestures to modulate or autotune voices.
In parallel with these design discussions, we have been preparing our upcoming presentation. We are ensuring it incorporates insights from Dr. Dueck, John, and the full Music + AI class, while clearly articulating our workflow, technologies, and hardware. To improve from our previous presentation, we are emphasizing testable metrics and quantitative values that demonstrate feasibility and create a logical progression toward our MVP.
One of the most significant risks to our project is the reliability of computer vision for hand and finger tracking. MediaPipe or other CV frameworks may struggle with occlusion, lighting variability, or inconsistent landmark detection, which could reduce accuracy in chord selection. To mitigate this, our plan to combine CV with IMU-based sensing is the fallback, ensuring users can still play even if vision fails. Another risk is hardware integration: syncing haptic pads, IMU sensors, and CV data in real time could introduce latency. As a contingency, we will begin with a simplified prototype using only one sensing modality (e.g., IMU alone) and add complexity gradually, validating latency and stability at each step.
A was written by Taj, B was written by Alexa, and C was written by Lucy.
A: The product solution we are designing directly addresses public health, safety, and welfare by creating a musical interface that is inclusive for people with physical disabilities such as mild cerebral palsy, arthritis, Limb-Girdle Muscular Dystrophy, and early-stage ALS. These conditions often limit fine motor control, finger strength, or the ability to play traditional string instruments, which can negatively impact both physical and psychological well-being. By enabling users to make music through accessible gestures such as finger bends or simple strumming motions detected by IMUs, haptic pads, and computer vision, the system lowers barriers to participation in creative expression, supports psychological health by fostering joy and confidence, and provides a pathway for therapeutic engagement through music-making in a safe manner.
B: The instrument we are building address might address social factors like community around music. We hope that people will impaired ability to play traditional acoustic instruments can still feel like a contributing member of musical social groups. Potentially new social groups can form around non-traditional instrument that will make the music scene more diverse and inclusive.
C: Our project contributes to economic factors by offering a low-cost, accessible alternative to traditional adaptive instruments, which are often prohibitively expensive. By leveraging open-source software (MediaPipe, TensorFlow.js) and inexpensive hardware (IMU sensors, haptic pads), we reduce production costs and make the system more affordable for schools, therapy programs, and individual users. This broadens participation in music-making, expands potential markets in education and healthcare, and creates opportunities for scalable distribution without the need for specialized equipment.