Author: ghchang

Our final poster can be found using this link

https://docs.google.com/presentation/d/1Vzt34DuYDCTVtQU2lHRa8Gi4bRE4dUuA/edit?usp=sharing&ouid=109476918104877410554&rtpof=true&sd=true

This week I focused on refining the physical design and improving hardware reliability, with the goal of preparing the system for a clean and stable demo-ready build.

Housing Design and 3D Modeling

A major focus this week was designing the enclosure for the LumiKey system:

• Created a 3D model of the housing to contain the ESP32, wiring, LED chain, and FSR circuitry
• Iterated on dimensions to ensure proper alignment with piano keys and internal component spacing
• Considered accessibility for wiring, maintenance, and future modifications

This step is important for transitioning from an exposed prototype to a more polished and portable system.

Hardware Debugging and Signal Stability

I worked on identifying and resolving reliability issues in the hardware:

• Discovered that loose “hair-like” wires and exposed connections were causing intermittent LED failures and unstable behavior
• Re-soldered and reinforced connections to eliminate unintended contact between power, ground, and data lines
• Improved insulation and organization to reduce noise and prevent future short circuits

These fixes significantly improved overall system stability, especially during movement and handling.

FSR Calibration and Threshold Tuning

Another key area this week was refining how the FSR sensors behave when integrated with the piano:

• Adjusted threshold values to account for how FSR strips respond when placed under real piano keys
• Observed differences between free-space testing and actual mounted conditions
• Tuned press and release thresholds to reduce false positives and improve responsiveness

This ensures that the system accurately detects user input in realistic playing conditions.

Material Selection for FSR Interface

I also experimented with different materials placed on top of the FSR strips:

• Tested materials such as foam, cardboard, and thin rigid paper layers
  Evaluated based on force distribution, sensitivity, and consistency of readings
• Selected a thin cardboard-like material as it provided the best balance between responsiveness and stability

This layer plays a critical role in translating key presses into reliable sensor readings.

Next Steps
– Finalize and fabricate the enclosure design
– Integrate all components into the housing for full system assembly
– Continue long-duration testing for hardware reliability
– Validate system performance during continuous piano interaction

This week I focused on scaling and stabilizing the hardware subsystem for the LumiKey project, with the goal of transitioning from a small prototype to a full single-octave implementation that is reliable enough for validation and demo use.

Learning New Tools and Knowledge Acquisition

Since I worked primarily on the hardware components of the project, I had to learn how to interpret and apply technical documentation effectively. This included understanding the behavior of components such as FSR strips and LED drivers, as well as implementation details like bit shifting in the Adafruit LED libraries. To navigate this, I relied heavily on a combination of official documentation, online forums such as Reddit, and LLM based tools. These resources were especially helpful in quickly identifying relevant sections of large documentation sets, allowing me to focus on the specific details I needed rather than spending excessive time searching.

In addition to online resources, I adopted hands-on and collaborative learning strategies. I initially lacked confidence in soldering on a protoboard and troubleshooting hardware issues, but by consulting with TechSpark staff and seeking in person guidance, I was able to learn proper soldering techniques and best practices. This significantly reduced hardware related issues and improved the reliability of my system.

Through this process, I also developed practical debugging skills. For example, I was able to identify and resolve issues such as unintended short circuits caused by stray wires, which led to incorrect LED behavior during testing. Overall, combining targeted use of documentation, community resources, LLM assisted search, and hands-on mentorship allowed me to efficiently acquire the knowledge needed to design, implement, and debug the system.

System Expansion

A major milestone this week was expanding both sensing and visual output coverage:

• Increased FSR input channels from 12 to 24, enabling support for a full octave range
• Expanded LED output from 5 LEDs to 36 LEDs, allowing per-key visual guidance across the entire octave
• Ensured proper mapping between each FSR input and its corresponding LED output

This expansion moves the system closer to the intended use case of guiding real piano playing rather than demonstrating isolated functionality.

FSR Strip Validation and Debugging

A significant portion of the week was spent verifying the reliability of the FSR sensing system:

• Tested each FSR channel individually to ensure consistent and correct readings
• Identified and debugged issues related to inconsistent triggering and cross-channel interference
• Verified that each sensor responds appropriately to key presses and releases

This process was important to ensure that the system produces stable and interpretable input signals before integrating higher-level logic.

FSR Modification Experiments

I also explored the flexibility of the sensing hardware by experimenting with modifying the FSR strips:

• Tested whether FSR strips could be cut to custom lengths for better physical alignment with keys
• Found that the strips remain functional when cut within certain threshold regions, without breaking conductivity
• This result improves the adaptability of the system for different keyboard sizes and layouts

Hardware Integration and Wiring

To improve system robustness and prepare for enclosure, I worked on consolidating the hardware:

• Performed extensive wire management to reduce clutter and improve signal reliability
• Migrated connections onto a perfboard-based PCB, replacing loose wiring with a more stable structure
• Organized power, ground, and signal routing to support both LEDs and FSR inputs cleanly

This significantly improves durability and makes the system more suitable for packaging into a final enclosure.

Next Steps
Finalize enclosure design and integrate all components into a compact housing

Continue validating sensor reliability under continuous use
Integrate hardware with software pipeline for full system testing (input → processing → LED feedback)

Measure and report system-level latency and responsiveness

This week I focused on expanding and integrating the hardware subsystem of the LumiKey project to support more complete functionality and better interaction with the software layer. Building on the previous prototype, my goal was to increase system coverage and establish a more seamless connection between hardware inputs and the software interface.

Hardware Expansion

A major focus this week was scaling the hardware setup from a smaller prototype to a 10-key system. I expanded both the LED array and the FSR strip to support 10 keys, allowing for more realistic interaction and better representation of actual piano input. This required updating the wiring and ensuring that each sensor and LED channel functioned reliably when scaled.

Audio Feedback Integration

I implemented a sound feedback feature where each key press detected on the FSR strip triggers a corresponding sound played on the laptop. This enables immediate auditory feedback, making the system feel more interactive and closer to a real piano experience. The integration ensures that physical input on the hardware directly maps to sound output in real time.

Software and Hardware Interface

Another key development was integrating the software UI with the hardware system. I established a bidirectional interface where inputs from the hardware trigger responses in the software, and interactions from the UI can activate corresponding LEDs on the hardware. Specifically, pressing a button on the software UI now correctly triggers the associated hardware component, allowing for synchronized control between the two layers.

Next Steps

Next steps include improving the reliability of the hardware sensing and LED signaling under continuous use, as well as refining timing synchronization between hardware input, sound output, and UI feedback. I will also begin preparing the system for structured testing as part of the overall verification phase.

This week I focused on advancing the hardware and interaction system of LumiKey in preparation for the upcoming demo. Building on the existing software pipeline, the goal this week was to improve real-time responsiveness and enable meaningful user interaction through key detection and visual feedback.

LED Timing and Visualization Improvements

A major focus this week was implementing LED fading behavior to better represent timing and note progression. Instead of simple on/off signals, LEDs now fade in a controlled manner to indicate when a note is approaching, improving the user’s ability to anticipate timing.

This enhancement makes the system more intuitive during playback and aligns the visual guidance more closely with musical rhythm. The fading behavior was tested alongside the existing event stream to ensure it remains synchronized with playback.

FSR Integration and Multiplexer Expansion

Another key area of progress was expanding key detection using FSR strips integrated through multiplexers. By incorporating the multiplexer, the system can now handle input from multiple keys while minimizing the number of required GPIO pins on the ESP32.

This setup allows for scalable input sensing across a larger range of keys. Initial testing confirmed that multiple FSR inputs can be read and processed, enabling the system to detect user interaction across different notes.

Some challenges remain with signal consistency, including occasional false positives or missed presses, likely due to noise or variability in pressure sensing. To address this, I began refining threshold detection and improving signal stability, which will continue to be optimized.

Interaction Modes Implementation

This week I also implemented multiple interaction modes to support different learning experiences:

Basic Mode: The system waits for the user to press the correct key before advancing, enabling step-by-step learning.
Beat Mode: Notes are aligned with a consistent tempo, allowing users to play along with timing guidance.
Score System: A scoring mechanism was introduced to track user performance based on accuracy.

These modes allow the system to move beyond passive visualization and support active learning through user input.

Integration with Event System

All hardware interactions, including LED output and FSR input detection, were integrated with the existing note event system. This ensures that both visualization and user input are driven by the same underlying data, maintaining synchronization between software and hardware components.

This unified approach reduces the risk of timing drift and simplifies debugging and further development.

Schedule

The project remains on schedule and is progressing toward a fully integrated demo. This week’s work focused on hardware functionality and interaction, aligning with the planned transition to a complete interactive system.

Deliverables for Next Week

Next week I will focus on final testing and supporting the live demo. This includes improving the reliability of FSR input detection, further tuning LED timing behavior, and resolving any integration issues between hardware and the software pipeline.

I will also continue refining interaction modes and ensuring that the system provides a smooth and intuitive user experience during demonstrations.

This week primarily focused on continued development and debugging of the force-sensitive resistor (FSR) input system for the LumiKeys project. Progress was centered on stabilizing the detection of key presses across a smaller subset of the keyboard, specifically five keys, before scaling to the full system.

FSR Integration and Debugging

• Worked on refining the code for reading and interpreting signals from five FSR strips connected through the analog multiplexer.
• Focused on improving signal reliability and consistency, as initial readings showed noise and occasional false triggers.
• Tested different threshold values and filtering approaches in software to better distinguish intentional key presses from background noise.
• Continued debugging wiring and connection stability to ensure that each FSR input is correctly mapped and independently detected.

System Stability and Validation

• Conducted repeated tests on the five-key setup to verify that key presses are consistently detected and transmitted.
• Identified minor inconsistencies in readings, which are currently being investigated as potential issues related to wiring layout or power distribution.
• Maintained the goal of achieving a stable and reliable baseline before expanding to additional keys.

Schedule

I am slightly behind the ideal timeline due to extended debugging of the FSR inputs, but this step is critical to ensure reliability before scaling to the full keyboard system.

Deliverables for Next Week

• Finalize stable detection for the five FSR keys with minimal noise and false positives.
• Implement improved filtering or smoothing techniques if necessary to enhance signal accuracy.
• Begin scaling the working solution to additional keys once consistent performance is achieved.

This week focused mainly on wiring additional FSR strips to the 16-channel analog multiplexer and ensuring that each key press can be individually detected and registered. Currently, I’m encountering an issue where ghost signals appear from other keys. I suspect this may be caused either by the wires being routed too closely together or by a power stability issue. I also reviewed the firmware to determine whether the problem could be mitigated through software.

A significant portion of the week was also spent soldering the LEDs and FSR strips together for the hardware assembly.

Implemented User Feedback Modes and Interaction Logic

This week I focused on designing and testing different visual feedback methods to guide users during key presses. The ESP32 hardware layer is now fully connected to the core software layer through Bluetooth communication, allowing the system to receive note data and control the LED indicators in real time.

Three interaction modes were implemented: Basic Mode, Beat Mode, and Play-Through Mode. These modes allow the system to guide the user through different learning experiences, ranging from simple key highlighting to full song playback visualization.

To improve usability, I implemented a timing tolerance system that evaluates whether a user presses a key within an acceptable time window relative to the metronome beat. This allows the system to determine whether a note is played correctly or incorrectly without requiring perfect timing, making the interaction more forgiving for beginners.

The visual feedback behavior was also refined. When a user presses the correct key, the corresponding LED turns green and gradually fades, indicating that the note was successfully played. If the user presses an incorrect key, the LED blinks red, signaling the mistake and preventing the note sequence from advancing. This feedback loop helps users quickly understand their errors while practicing.

Restructured Musical Timing Data

Another major improvement this week was restructuring the JSON format used to represent musical notes. Previously, notes were processed sequentially, but the new structure now includes explicit start time values for each note.

This change allows notes to activate based on their absolute timing positions, enabling the implementation of the Play-Through Mode, where LEDs illuminate the full sequence of keys according to the timing of the entire song. This significantly improves the system’s ability to represent more complex musical passages where multiple notes may occur simultaneously or overlap in time.

Implemented Metronome-Based Start Countdown

To help users prepare before the song begins, I implemented a three-beat countdown indicator using the LED system. Before playback starts, the LEDs blink three times in synchronization with the system’s metronome.

This visual countdown ensures that users can align themselves with the tempo before beginning to play, improving the overall learning experience and reducing confusion about when the song starts.

Status Report Requirement: Part C (Environmental Factors)

Environmental factors were considered in the design of LumiKeys with respect to material usage, energy consumption, and the lifespan of electronic components. The system is designed as a retrofit solution that can be installed on existing acoustic or digital pianos rather than requiring the production of a completely new instrument. By augmenting instruments that users already own, the project helps reduce electronic waste and minimizes the environmental impact associated with manufacturing new hardware.

Additionally, the hardware architecture was designed to use low-power components, such as the ESP32 microcontroller and addressable NeoPixel LEDs, which can operate efficiently with a single regulated power supply. The system also minimizes unnecessary processing by transmitting compact musical data through Bluetooth rather than requiring continuous high-bandwidth communication.

Another environmental consideration involves the durability and modularity of the hardware design. Components such as the FSR strips, multiplexer, and LED strip are connected in a modular manner so they can be repaired or replaced individually without discarding the entire system. This design approach extends the usable lifetime of the device and reduces electronic waste.

Overall, by focusing on retrofit compatibility, efficient power usage, and modular hardware components, the LumiKeys system aims to reduce its environmental footprint while providing an accessible tool for music learning.

Schedule

I am currently on schedule. Bluetooth communication between the core layer and the hardware layer is stable, and the LED feedback system is functioning correctly across all implemented modes. The restructuring of the JSON timing data also enables more complex playback functionality moving forward.

Deliverables for Next Week

Next week I plan to integrate the first octave FSR strips with the analog multiplexer so the ESP32 can begin detecting real key presses. I will set up a simplified prototype using one octave of LEDs and FSR sensors and test it using songs that only require a single octave.

During testing, I will measure the timing delay between LED activation and FSR detection to evaluate whether the system introduces any noticeable latency for the user. These tests will help verify that the sensing and LED feedback systems work reliably before scaling to the full keyboard.

This week focused on exploring and implementing communication methods between the ESP32 and the software layer, while continuing hardware integration for the LumiKeys system. Specifically, I focused on evaluating different connection methods, integrating LED control with the ESP32, and prototyping communication between the ESP32 and a Python-based interface.

Hardware Integration and Communication Prototyping

• Researched and evaluated multiple communication approaches for connecting the ESP32 to a computer and software layer, including Bluetooth and MQTT-based web services.
• Selected Bluetooth as the primary method for the proof of concept due to its simplicity, low setup overhead, and suitability for short-range communication.
• Integrated NeoPixel LEDs with the ESP32 and implemented basic LED control using the Arduino IDE to verify that LEDs can be addressed and controlled reliably.
• Investigated methods for transmitting note and color data between Python and the ESP32, including the use of structured data formats such as JSON for communication.
• Focused on prototyping the connection between the ESP32 and the Python layer to establish a stable communication workflow before full hardware assembly.

System Planning and Preparation

• Planned the transition from prototyping to full assembly by identifying components that will need to be ordered once the communication architecture and hardware configuration are finalized.
• Prepared for scaling from breadboard prototyping to a complete physical model after validating communication and control workflows.

Schedule

I am currently on schedule, as communication methods have been evaluated, LED control has been successfully integrated with the ESP32, and the architecture for Python-to-ESP32 communication has been defined.

Deliverables for Next Week

• Fully integrate Bluetooth communication between the Python program and the ESP32.
• Develop the first prototype demonstrating LED control through Bluetooth commands sent from Python, including color coding and structured message formats.Test transmission of force-sensitive resistor (FSR) input data to the Python layer and verify that key presses are detected and recorded reliably.

Part B – Social FactorsThe LumiKeys system is designed with consideration of social factors, particularly accessibility to music education and differences in how people learn in modern social environments. Many beginner piano learners rely on online tutorials or self-guided learning due to limited access to formal lessons, which may be influenced by cultural, geographic, or financial constraints. By providing real-time visual guidance and note verification on any existing piano, LumiKeys supports learners who may not have access to private instruction or structured music programs, helping to reduce barriers to participation in music as a social and cultural activity.

In addition, the system is intended to support collaborative and informal learning settings, where friends, family members, or online communities share music learning experiences. Because the device is portable and designed to work with standard acoustic pianos, it can be used in homes, schools, and shared spaces without requiring specialized instruments. This flexibility allows the product to fit into diverse social contexts and encourages music learning as a shared and engaging activity rather than an isolated or resource-intensive one.