Implemented User Feedback Modes and Interaction Logic
This week I focused on designing and testing different visual feedback methods to guide users during key presses. The ESP32 hardware layer is now fully connected to the core software layer through Bluetooth communication, allowing the system to receive note data and control the LED indicators in real time.
Three interaction modes were implemented: Basic Mode, Beat Mode, and Play-Through Mode. These modes allow the system to guide the user through different learning experiences, ranging from simple key highlighting to full song playback visualization.
To improve usability, I implemented a timing tolerance system that evaluates whether a user presses a key within an acceptable time window relative to the metronome beat. This allows the system to determine whether a note is played correctly or incorrectly without requiring perfect timing, making the interaction more forgiving for beginners.
The visual feedback behavior was also refined. When a user presses the correct key, the corresponding LED turns green and gradually fades, indicating that the note was successfully played. If the user presses an incorrect key, the LED blinks red, signaling the mistake and preventing the note sequence from advancing. This feedback loop helps users quickly understand their errors while practicing.
Restructured Musical Timing Data
Another major improvement this week was restructuring the JSON format used to represent musical notes. Previously, notes were processed sequentially, but the new structure now includes explicit start time values for each note.
This change allows notes to activate based on their absolute timing positions, enabling the implementation of the Play-Through Mode, where LEDs illuminate the full sequence of keys according to the timing of the entire song. This significantly improves the system’s ability to represent more complex musical passages where multiple notes may occur simultaneously or overlap in time.
Implemented Metronome-Based Start Countdown
To help users prepare before the song begins, I implemented a three-beat countdown indicator using the LED system. Before playback starts, the LEDs blink three times in synchronization with the system’s metronome.
This visual countdown ensures that users can align themselves with the tempo before beginning to play, improving the overall learning experience and reducing confusion about when the song starts.
Status Report Requirement: Part C (Environmental Factors)
Environmental factors were considered in the design of LumiKeys with respect to material usage, energy consumption, and the lifespan of electronic components. The system is designed as a retrofit solution that can be installed on existing acoustic or digital pianos rather than requiring the production of a completely new instrument. By augmenting instruments that users already own, the project helps reduce electronic waste and minimizes the environmental impact associated with manufacturing new hardware.
Additionally, the hardware architecture was designed to use low-power components, such as the ESP32 microcontroller and addressable NeoPixel LEDs, which can operate efficiently with a single regulated power supply. The system also minimizes unnecessary processing by transmitting compact musical data through Bluetooth rather than requiring continuous high-bandwidth communication.
Another environmental consideration involves the durability and modularity of the hardware design. Components such as the FSR strips, multiplexer, and LED strip are connected in a modular manner so they can be repaired or replaced individually without discarding the entire system. This design approach extends the usable lifetime of the device and reduces electronic waste.
Overall, by focusing on retrofit compatibility, efficient power usage, and modular hardware components, the LumiKeys system aims to reduce its environmental footprint while providing an accessible tool for music learning.
Schedule
I am currently on schedule. Bluetooth communication between the core layer and the hardware layer is stable, and the LED feedback system is functioning correctly across all implemented modes. The restructuring of the JSON timing data also enables more complex playback functionality moving forward.
Deliverables for Next Week
Next week I plan to integrate the first octave FSR strips with the analog multiplexer so the ESP32 can begin detecting real key presses. I will set up a simplified prototype using one octave of LEDs and FSR sensors and test it using songs that only require a single octave.
During testing, I will measure the timing delay between LED activation and FSR detection to evaluate whether the system introduces any noticeable latency for the user. These tests will help verify that the sensing and LED feedback systems work reliably before scaling to the full keyboard.