Team D7 LumiKeys

This is the link to our final report!

Our final poster can be found using this link

https://docs.google.com/presentation/d/1Vzt34DuYDCTVtQU2lHRa8Gi4bRE4dUuA/edit?usp=sharing&ouid=109476918104877410554&rtpof=true&sd=true

This week I focused on expanding system scale, improving usability, and stabilizing both software and hardware through extensive integration work. The primary efforts included expanding the system to support 36 keys, co-designing a physical casing for hardware wiring, and implementing a large set of software refinements aimed at making the system more user-friendly and robust.

Software and Hardware Integration

• Expanded the system from partial coverage to 36-key integration, including testing and debugging across the full range.
  Worked on software debugging related to real-time hardware interaction as
• key coverage increased.
• Co-designed a 3D-printed casing to organize wiring and improve physical stability during testing and demonstrations.
• Tested the expanded system extensively to ensure stable playback, LED behavior, and key detection.

UI and Playback System Improvements

• Renewed the user interface to improve clarity and usability:
  • Consolidated playback controls into clear mode-specific buttons (Playback, No-Beat, Beat).
  • Removed developer-oriented terms, internal states, and debug data from the user-facing UI.
  • Replaced abstract piano visuals with more realistic piano key representations.
• Enabled clicking individual bars in the piano roll to trigger sound playback.
• Implemented session saving for uploaded sheet music.
• Added an interactive tutorial to guide new users.

Musical Logic and Bug Fixes

• Implemented key range shifting to ensure music fits within the supported three-octave range.
• Fixed score evaluation logic to exclude notes outside the supported range.
  Improved metronome behavior:
• Added countdown-based metronome start.
• Adjusted metronome beats based on musical rhythm (e.g., waltz).
• Fixed multiple bugs, including:
  • Playback slider not resetting when uploading a new sheet.
  • Duplicate piano roll creation when pressing the space bar mid-playback.
  • Visual inconsistencies in piano key illustrations.

Schedule

I am currently on schedule. While this week involved a high volume of debugging and refinement, these efforts significantly improved system stability, usability, and scalability, and align well with the planned transition toward final validation and presentation.

Deliverables for Next Week

• Continue validation testing across the full 36-key range.
• Refine hardware setup and casing based on testing feedback.
• Conduct focused usability testing of the updated UI and tutorial.
• Prepare the system for final demonstrations and evaluations.

This week I focused on refining the physical design and improving hardware reliability, with the goal of preparing the system for a clean and stable demo-ready build.

Housing Design and 3D Modeling

A major focus this week was designing the enclosure for the LumiKey system:

• Created a 3D model of the housing to contain the ESP32, wiring, LED chain, and FSR circuitry
• Iterated on dimensions to ensure proper alignment with piano keys and internal component spacing
• Considered accessibility for wiring, maintenance, and future modifications

This step is important for transitioning from an exposed prototype to a more polished and portable system.

Hardware Debugging and Signal Stability

I worked on identifying and resolving reliability issues in the hardware:

• Discovered that loose “hair-like” wires and exposed connections were causing intermittent LED failures and unstable behavior
• Re-soldered and reinforced connections to eliminate unintended contact between power, ground, and data lines
• Improved insulation and organization to reduce noise and prevent future short circuits

These fixes significantly improved overall system stability, especially during movement and handling.

FSR Calibration and Threshold Tuning

Another key area this week was refining how the FSR sensors behave when integrated with the piano:

• Adjusted threshold values to account for how FSR strips respond when placed under real piano keys
• Observed differences between free-space testing and actual mounted conditions
• Tuned press and release thresholds to reduce false positives and improve responsiveness

This ensures that the system accurately detects user input in realistic playing conditions.

Material Selection for FSR Interface

I also experimented with different materials placed on top of the FSR strips:

• Tested materials such as foam, cardboard, and thin rigid paper layers
  Evaluated based on force distribution, sensitivity, and consistency of readings
• Selected a thin cardboard-like material as it provided the best balance between responsiveness and stability

This layer plays a critical role in translating key presses into reliable sensor readings.

Next Steps
– Finalize and fabricate the enclosure design
– Integrate all components into the housing for full system assembly
– Continue long-duration testing for hardware reliability
– Validate system performance during continuous piano interaction

Project Risks and Mitigation

As the system moves into deeper hardware integration, a primary risk is instability arising from real-time interaction between software playback logic and physical hardware. While individual software and hardware components function correctly in isolation, integration can expose timing mismatches, communication errors, or inconsistent behavior during live playback. This risk is being addressed through extensive debugging and repeated end-to-end testing using the fully integrated 13-key hardware setup.

Another key risk is inaccurate or inconsistent hardware response during real-world use, particularly related to LED triggering and key press detection thresholds. Variability in sensor readings or environmental factors could lead to incorrect guidance or missed inputs. This is being mitigated by testing the system on an actual keyboard and tuning activation thresholds based on observed behavior, rather than relying solely on simulated or theoretical values.

A further risk is that new software features introduced during integration—such as additional playback sound options and enhanced playback controls—could introduce unintended side effects or regressions. To manage this risk, each new feature is tested alongside hardware playback to ensure it does not interfere with timing, visualization, or LED output.

Design Changes

This week did not introduce major architectural changes, but several implementation-level refinements were made to support full system integration and testing. Software logic was adjusted to better handle real-time hardware communication and improve robustness during playback.

In addition, new user-facing software features were added, including support for multiple playback sound options and a fully functional sliding playback bar. These changes improve usability and user control while preserving the existing event-driven playback architecture.

The overall design continues to emphasize modularity, allowing hardware integration issues to be debugged without impacting the core conversion or visualization pipelines.

Schedule Update

The project remains on schedule. Full integration of 13 keys represents a significant milestone, and the team is now focused on stabilizing the system through debugging and validation. Progress aligns with the planned transition from integration into broader system validation.

Validation Plan

Validation this week focused on early system-level testing enabled by full software–hardware integration. The team conducted repeated end-to-end tests to ensure that software-generated musical events correctly drive LED output and playback on the physical keyboard.

Testing included verifying key activation thresholds, observing LED behavior during real playback, and confirming that timing remains consistent across different playback scenarios. These tests helped identify integration issues that were not visible in software-only testing and guided subsequent fixes.

Additional validation work confirmed that new playback features—such as sound selection and the sliding playback bar—function correctly during live hardware playback.

Demonstrated Progress

This week marked a major step forward in system maturity, with successful integration and testing of 13 hardware keys connected to the full software pipeline. The system was tested on an actual keyboard, demonstrating that LumiKey can operate under real-world conditions rather than controlled demo setups.

Through extensive debugging and testing, the team improved system stability and usability while adding new playback features. This progress positions the project well for continued validation and final demonstrations, with a functioning end-to-end system that reflects real user interaction.

This week I focused on stabilizing the software–hardware integration and validating the system through extensive real-world testing. With full integration of 13 keys completed, most of the work centered on debugging integration issues, testing the system on an actual keyboard, and making software improvements to enhance playback and usability.

Software and Hardware Integration Debugging

• Debugged software issues related to hardware integration following full integration of 13 keys.
• Identified and resolved errors related to event transmission, timing behavior, and key activation.
• Refined software logic to improve robustness during real-time playback.
• Conducted repeated integration tests to ensure stable communication between the software stack and hardware components.

Physical Keyboard Testing

• Tested the system on an actual keyboard to validate real-world behavior.
• Tuned and verified key activation thresholds to ensure accurate and reliable LED guidance.
• Performed extensive end-to-end testing to confirm that playback, visualization, and hardware output remain synchronized during execution.
• Used real keyboard testing to uncover edge cases not visible in software-only testing.

Software Feature Improvements

• Added support for multiple playback sound options, allowing users to switch between different playback modes.
• Implemented a fully functional sliding video-style playback bar to improve user interaction and playback control.
• Made additional usability-focused software refinements based on testing observations.

New Tools, Knowledge, and Learning Strategies

As the project progressed into full software–hardware integration and real-world testing, it became necessary to learn additional tools and system-level debugging techniques. In particular, I gained hands-on experience debugging real-time interactions between software playback logic and physical hardware, including timing-sensitive LED control and key activation thresholds on an actual keyboard. This required a deeper understanding of how software assumptions translate (or fail to translate) into physical behavior.

I also learned more about diagnosing integration failures across language and system boundaries, especially when tracking down intermittent errors caused by data formatting, timing mismatches, or hardware constraints. Supporting new playback features, such as multiple sound options and a sliding playback bar, further required learning UI-state synchronization techniques and refining event-driven playback logic.

To acquire this knowledge, I primarily relied on informal learning strategies. These included reading documentation and source code, consulting online forum discussions and issue threads related to similar hardware–software integration problems, watching short technical videos for targeted concepts, and using iterative experimentation through testing and debugging. Repeated hands-on testing—especially on the physical keyboard—was a key learning strategy, as it exposed issues that were not apparent in isolated software tests and helped reinforce practical understanding of system behavior.

Schedule

I am currently on schedule. Full 13-key integration and successful real-world testing represent a major milestone, and the system is becoming increasingly stable through continued debugging and refinement.

Deliverables for Next Week

• Continue debugging remaining edge cases in software–hardware integration.
• Expand testing to additional musical inputs and tempos.
• Refine UI behavior and playback controls based on user interaction.
• Begin preparing the system for broader demonstrations and evaluations.

This week I focused on scaling and stabilizing the hardware subsystem for the LumiKey project, with the goal of transitioning from a small prototype to a full single-octave implementation that is reliable enough for validation and demo use.

Learning New Tools and Knowledge Acquisition

Since I worked primarily on the hardware components of the project, I had to learn how to interpret and apply technical documentation effectively. This included understanding the behavior of components such as FSR strips and LED drivers, as well as implementation details like bit shifting in the Adafruit LED libraries. To navigate this, I relied heavily on a combination of official documentation, online forums such as Reddit, and LLM based tools. These resources were especially helpful in quickly identifying relevant sections of large documentation sets, allowing me to focus on the specific details I needed rather than spending excessive time searching.

In addition to online resources, I adopted hands-on and collaborative learning strategies. I initially lacked confidence in soldering on a protoboard and troubleshooting hardware issues, but by consulting with TechSpark staff and seeking in person guidance, I was able to learn proper soldering techniques and best practices. This significantly reduced hardware related issues and improved the reliability of my system.

Through this process, I also developed practical debugging skills. For example, I was able to identify and resolve issues such as unintended short circuits caused by stray wires, which led to incorrect LED behavior during testing. Overall, combining targeted use of documentation, community resources, LLM assisted search, and hands-on mentorship allowed me to efficiently acquire the knowledge needed to design, implement, and debug the system.

System Expansion

A major milestone this week was expanding both sensing and visual output coverage:

• Increased FSR input channels from 12 to 24, enabling support for a full octave range
• Expanded LED output from 5 LEDs to 36 LEDs, allowing per-key visual guidance across the entire octave
• Ensured proper mapping between each FSR input and its corresponding LED output

This expansion moves the system closer to the intended use case of guiding real piano playing rather than demonstrating isolated functionality.

FSR Strip Validation and Debugging

A significant portion of the week was spent verifying the reliability of the FSR sensing system:

• Tested each FSR channel individually to ensure consistent and correct readings
• Identified and debugged issues related to inconsistent triggering and cross-channel interference
• Verified that each sensor responds appropriately to key presses and releases

This process was important to ensure that the system produces stable and interpretable input signals before integrating higher-level logic.

FSR Modification Experiments

I also explored the flexibility of the sensing hardware by experimenting with modifying the FSR strips:

• Tested whether FSR strips could be cut to custom lengths for better physical alignment with keys
• Found that the strips remain functional when cut within certain threshold regions, without breaking conductivity
• This result improves the adaptability of the system for different keyboard sizes and layouts

Hardware Integration and Wiring

To improve system robustness and prepare for enclosure, I worked on consolidating the hardware:

• Performed extensive wire management to reduce clutter and improve signal reliability
• Migrated connections onto a perfboard-based PCB, replacing loose wiring with a more stable structure
• Organized power, ground, and signal routing to support both LEDs and FSR inputs cleanly

This significantly improves durability and makes the system more suitable for packaging into a final enclosure.

Next Steps
Finalize enclosure design and integrate all components into a compact housing

Continue validating sensor reliability under continuous use
Integrate hardware with software pipeline for full system testing (input → processing → LED feedback)

Measure and report system-level latency and responsiveness