This week, our team made strong progress on both the software and hardware fronts, keeping us on track for our upcoming meeting with the School of Music. On the software side, we focused on integrating the recorded vocal samples from last week’s session into synthesizable sound files that can be manipulated in code. This step brings us closer to creating a dynamic audio system where sound output can be modulated in real time based on sensor data and user interaction. As part of this process, we also began exploring how to connect microcontrollers to MIDI interfaces, referencing this CMU resource to better understand how our embedded system could communicate with standard digital instruments. This topic has been discussed frequently in class with the Music and Technology majors, and it presents a promising direction for future integration.
On the hardware and communication side, we identified and resolved a key issue preventing reliable data transmission between our ESP32-S3 microcontroller and the host PC. After testing, we determined that ESP-NOW is not a suitable protocol for this type of communication, so we are transitioning to Bluetooth Low Energy (BLE) as our new wireless solution. We are currently not using Wi-Fi due to persistent issues connecting to CMU Secure, but we are investigating the possibility of connecting through the CMU Devices network instead. Since this network only requires a registered MAC address and hostname, it may provide the stable connectivity we need for data streaming and control signals.
On the software side, we were able to use the vocal samples that we collected and start using different hand gestures to play chords in different modes (both the pre-programmed piano notes and the voices from a vocalist). Below are the videos of the software progress:
Link to video with notes, link to video with voice
Looking ahead to early next week, our team plans to conduct focused development sprints to assemble an initial end-to-end prototype in preparation for our meeting with the School of Music on Wednesday.