Posted on

Team Status Report for April 25

Project Risks and Mitigation

As the system scales from partial to wider keyboard coverage, a key risk is increased complexity in both hardware wiring and software mapping. Expanding to 36 keys introduces more points of failure, including incorrect key-to-LED mappings, inconsistent physical placement, and greater sensitivity to configuration errors. This risk is being mitigated by introducing a configurable key-to-strip mapping system, allowing the software to adapt to arbitrary LED strip placement rather than assuming a fixed layout.

Another risk is usability degradation as more features are added. As the system matured, earlier developer-oriented UI elements and controls became confusing or overwhelming for end users. This risk is being actively addressed through a UI redesign that removes developer-only information, consolidates controls, and replaces abstract visual elements with more intuitive representations (e.g., piano keys instead of generic bars).

A further risk lies in musical correctness and user trust. Features such as metronome behavior, rhythm handling (e.g., waltz timing), and score evaluation must align with musical expectations. Incorrect rhythm or inclusion of out-of-range notes could confuse users. This is being mitigated by refining rhythm-aware metronome logic and explicitly filtering notes outside the supported three-octave range.

On the hardware side, physical durability and wiring organization pose a risk as the system becomes more complex. To address this, the team co-designed a 3D-printed casing to safely house wiring and improve reliability during repeated setup and demonstrations.

Design Changes

This week included several significant design refinements focused on scalability, usability, and system polish.

The system was expanded to support 36 keys, along with software support for shifting note ranges so that music fits within the available three-octave span. A flexible key-to-LED-strip mapping mechanism was added, allowing LED strips to be attached in arbitrary positions and configured through software.

The user interface was substantially redesigned to improve clarity and usability. Playback controls were consolidated into clear mode-specific buttons (Playback, No-Beat, Beat), replacing earlier toggles and developer-oriented states. Preparation and internal-state indicators were removed, developer-only data was hidden behind a development mode, and the piano roll and keyboard visuals were updated to resemble actual piano keys. Users can now click directly on piano roll bars to trigger sound, improving interactivity and learnability.

Several functional improvements and bug fixes were also completed, including session saving for uploaded sheets, a tutorial system for first-time users, metronome improvements (countdown start and rhythm-aware beats), playback slider reset fixes, visual polish on piano key illustrations, and fixes for issues such as duplicate piano roll creation and incorrect score evaluation.

Schedule Update

The project remains on schedule. While this week involved a high volume of refinements and fixes, these changes represent planned convergence work as the system transitions from integration toward final usability and validation. No major schedule adjustments are required.

Validation Plan

Validation this week focused on confirming that recent expansions and refinements work cohesively as a system. The team tested the full 36-key setup with physical hardware, verified that configurable key mappings behave correctly, and ensured that note shifting produces musically sensible output within the supported range.

UI validation included testing all playback modes, verifying correct metronome behavior across different rhythms, confirming that session saving and reloading behaves correctly, and ensuring that interactive elements (piano roll clicking, playback slider, mode buttons) work consistently.

Hardware validation included checking that the new casing supports stable wiring and that expanded key coverage remains reliable during extended playback and testing sessions.

Demonstrated Progress

This week marked a major step toward a polished, user-ready system. LumiKey now supports a substantially expanded key range, a cleaner and more intuitive user interface, flexible hardware configuration, and improved musical correctness. The addition of a 3D-printed casing further demonstrates readiness for real-world use and repeated demonstrations.

Overall, the project has moved beyond core functionality into refinement, scalability, and usability, positioning LumiKey well for final validation and presentation as a cohesive guided piano learning system.

Unit and System Testing

At the unit level, we carried out targeted tests on individual subsystems to verify correctness and performance:

Scan pipeline tests: Compared generated note events against labeled reference outputs to verify full event correctness and pitch-level accuracy.
MusicXML validation tests: Ensured generated outputs were structurally valid and could be parsed reliably after cleanup.
UI responsiveness tests: Verified timing using the application refresh interval (100 ms) and visualizer scheduler (50 ms), confirming compliance with the ≤200 ms requirement.
Pipeline consistency tests: Repeated runs on the same inputs to confirm deterministic and stable outputs.

At the system level, we conducted end-to-end tests to evaluate the integrated system:

End-to-end pipeline testing: Verified the full flow from sheet music input through scan, OMR, event generation, and playback produced correct and usable outputs.
Expanded keyboard testing (36 keys): Confirmed correct key-to-LED mapping and reliable behavior across the full range.
UI interaction testing: Tested playback modes, tempo adjustments, and user interactions to ensure consistent and responsive behavior.
Hardware integration testing: Verified stable FSR sensing, LED actuation, and communication during continuous operation.

Findings and Design Changes

Analysis of test results led to several key findings and corresponding design improvements:

Scan reliability issues: Complex or high-resolution inputs caused failures or inconsistent outputs. This led to restructuring the preprocessing pipeline to normalize and resize inputs before OMR.
Incorrect pitch ranges: OMR occasionally produced unrealistic note octaves. A correction step was added to constrain notes to a valid piano range.
Scalability challenges (36 keys): Fixed mappings did not generalize to larger setups. A configurable key-to-LED mapping system was introduced.
UI usability issues: Developer-oriented controls reduced clarity for users. The interface was redesigned with simplified controls and more intuitive visuals.
Hardware noise and wiring complexity: Increased sensor count introduced instability. Sensor strips were moved onto a PCB and a protective casing was added to improve reliability.

Posted on

Chris Oh’s Status Report for April 25

This week I focused on expanding system scale, improving usability, and stabilizing both software and hardware through extensive integration work. The primary efforts included expanding the system to support 36 keys, co-designing a physical casing for hardware wiring, and implementing a large set of software refinements aimed at making the system more user-friendly and robust.

Software and Hardware Integration

  • Expanded the system from partial coverage to 36-key integration, including testing and debugging across the full range.
    Worked on software debugging related to real-time hardware interaction as
  • key coverage increased.
  • Co-designed a 3D-printed casing to organize wiring and improve physical stability during testing and demonstrations.
  • Tested the expanded system extensively to ensure stable playback, LED behavior, and key detection.

UI and Playback System Improvements

  • Renewed the user interface to improve clarity and usability:
    • Consolidated playback controls into clear mode-specific buttons (Playback, No-Beat, Beat).
    • Removed developer-oriented terms, internal states, and debug data from the user-facing UI.
    • Replaced abstract piano visuals with more realistic piano key representations.
  • Enabled clicking individual bars in the piano roll to trigger sound playback.
  • Implemented session saving for uploaded sheet music.
  • Added an interactive tutorial to guide new users.

Musical Logic and Bug Fixes

  • Implemented key range shifting to ensure music fits within the supported three-octave range.
  • Fixed score evaluation logic to exclude notes outside the supported range.
    Improved metronome behavior:
  • Added countdown-based metronome start.
  • Adjusted metronome beats based on musical rhythm (e.g., waltz).
  • Fixed multiple bugs, including:
    • Playback slider not resetting when uploading a new sheet.
    • Duplicate piano roll creation when pressing the space bar mid-playback.
    • Visual inconsistencies in piano key illustrations.

Schedule

I am currently on schedule. While this week involved a high volume of debugging and refinement, these efforts significantly improved system stability, usability, and scalability, and align well with the planned transition toward final validation and presentation.

Deliverables for Next Week

  • Continue validation testing across the full 36-key range.
  • Refine hardware setup and casing based on testing feedback.
  • Conduct focused usability testing of the updated UI and tutorial.
  • Prepare the system for final demonstrations and evaluations.
Posted on

Team Status Report for April 18

Project Risks and Mitigation

As the system moves into deeper hardware integration, a primary risk is instability arising from real-time interaction between software playback logic and physical hardware. While individual software and hardware components function correctly in isolation, integration can expose timing mismatches, communication errors, or inconsistent behavior during live playback. This risk is being addressed through extensive debugging and repeated end-to-end testing using the fully integrated 13-key hardware setup.

Another key risk is inaccurate or inconsistent hardware response during real-world use, particularly related to LED triggering and key press detection thresholds. Variability in sensor readings or environmental factors could lead to incorrect guidance or missed inputs. This is being mitigated by testing the system on an actual keyboard and tuning activation thresholds based on observed behavior, rather than relying solely on simulated or theoretical values.

A further risk is that new software features introduced during integration—such as additional playback sound options and enhanced playback controls—could introduce unintended side effects or regressions. To manage this risk, each new feature is tested alongside hardware playback to ensure it does not interfere with timing, visualization, or LED output.

Design Changes

This week did not introduce major architectural changes, but several implementation-level refinements were made to support full system integration and testing. Software logic was adjusted to better handle real-time hardware communication and improve robustness during playback.

In addition, new user-facing software features were added, including support for multiple playback sound options and a fully functional sliding playback bar. These changes improve usability and user control while preserving the existing event-driven playback architecture.

The overall design continues to emphasize modularity, allowing hardware integration issues to be debugged without impacting the core conversion or visualization pipelines.

Schedule Update

The project remains on schedule. Full integration of 13 keys represents a significant milestone, and the team is now focused on stabilizing the system through debugging and validation. Progress aligns with the planned transition from integration into broader system validation.

Validation Plan

Validation this week focused on early system-level testing enabled by full software–hardware integration. The team conducted repeated end-to-end tests to ensure that software-generated musical events correctly drive LED output and playback on the physical keyboard.

Testing included verifying key activation thresholds, observing LED behavior during real playback, and confirming that timing remains consistent across different playback scenarios. These tests helped identify integration issues that were not visible in software-only testing and guided subsequent fixes.

Additional validation work confirmed that new playback features—such as sound selection and the sliding playback bar—function correctly during live hardware playback.

Demonstrated Progress

This week marked a major step forward in system maturity, with successful integration and testing of 13 hardware keys connected to the full software pipeline. The system was tested on an actual keyboard, demonstrating that LumiKey can operate under real-world conditions rather than controlled demo setups.

Through extensive debugging and testing, the team improved system stability and usability while adding new playback features. This progress positions the project well for continued validation and final demonstrations, with a functioning end-to-end system that reflects real user interaction.

Posted on

Chris Oh’s Status Report for April 18

This week I focused on stabilizing the software–hardware integration and validating the system through extensive real-world testing. With full integration of 13 keys completed, most of the work centered on debugging integration issues, testing the system on an actual keyboard, and making software improvements to enhance playback and usability.

Software and Hardware Integration Debugging

  • Debugged software issues related to hardware integration following full integration of 13 keys.
  • Identified and resolved errors related to event transmission, timing behavior, and key activation.
  • Refined software logic to improve robustness during real-time playback.
  • Conducted repeated integration tests to ensure stable communication between the software stack and hardware components.

Physical Keyboard Testing

  • Tested the system on an actual keyboard to validate real-world behavior.
  • Tuned and verified key activation thresholds to ensure accurate and reliable LED guidance.
  • Performed extensive end-to-end testing to confirm that playback, visualization, and hardware output remain synchronized during execution.
  • Used real keyboard testing to uncover edge cases not visible in software-only testing.

Software Feature Improvements

  • Added support for multiple playback sound options, allowing users to switch between different playback modes.
  • Implemented a fully functional sliding video-style playback bar to improve user interaction and playback control.
  • Made additional usability-focused software refinements based on testing observations.

New Tools, Knowledge, and Learning Strategies

As the project progressed into full software–hardware integration and real-world testing, it became necessary to learn additional tools and system-level debugging techniques. In particular, I gained hands-on experience debugging real-time interactions between software playback logic and physical hardware, including timing-sensitive LED control and key activation thresholds on an actual keyboard. This required a deeper understanding of how software assumptions translate (or fail to translate) into physical behavior.

I also learned more about diagnosing integration failures across language and system boundaries, especially when tracking down intermittent errors caused by data formatting, timing mismatches, or hardware constraints. Supporting new playback features, such as multiple sound options and a sliding playback bar, further required learning UI-state synchronization techniques and refining event-driven playback logic.

To acquire this knowledge, I primarily relied on informal learning strategies. These included reading documentation and source code, consulting online forum discussions and issue threads related to similar hardware–software integration problems, watching short technical videos for targeted concepts, and using iterative experimentation through testing and debugging. Repeated hands-on testing—especially on the physical keyboard—was a key learning strategy, as it exposed issues that were not apparent in isolated software tests and helped reinforce practical understanding of system behavior.

Schedule

I am currently on schedule. Full 13-key integration and successful real-world testing represent a major milestone, and the system is becoming increasingly stable through continued debugging and refinement.

Deliverables for Next Week

  • Continue debugging remaining edge cases in software–hardware integration.
  • Expand testing to additional musical inputs and tempos.
  • Refine UI behavior and playback controls based on user interaction.
  • Begin preparing the system for broader demonstrations and evaluations.
Posted on

Chris Oh’s Status Report for April 4

This week I focused on integrating the software and hardware components of the system and stabilizing the end-to-end pipeline. The main effort involved connecting the playback and visualization software to the hardware layer, debugging integration issues, and resolving errors encountered during real-world execution. In parallel, I worked on fully integrating the OMR pipeline with the conversion layer and began addressing a Java-side error affecting OMR processing.

Software and Hardware Integration

  • Integrated the playback and visualization software with the hardware layer to enable end-to-end system execution.
  • Verified that processed musical events are correctly transmitted to the hardware during playback.
  • Encountered intermittent failures and runtime errors during integration, primarily due to timing, data formatting, and interface mismatches.
    Spent time debugging and resolving these issues to improve system stability and reliability.
  • Continued iterative testing to ensure that software-hardware communication behaves consistently during playback.

OMR and Conversion Layer Integration

  • Fully integrated the OMR pipeline with the conversion layer, enabling direct flow from sheet music input to processed musical events.
  • Verified that OMR outputs are correctly parsed and transformed into the system’s internal event representation.
  • Encountered a Java-related error within the OMR pipeline that affects certain inputs.
  • Began debugging the Java error and investigating its root cause to restore reliable OMR processing.

Schedule

I am currently on schedule. While integration exposed several failures and errors, this work is expected at this stage and is helping stabilize the system. Progress continues toward a fully integrated end-to-end pipeline.

Deliverables for Next Week

  • Resolve remaining software-hardware integration errors and improve system robustness.
  • Fix the Java error in the OMR pipeline and verify reliable sheet music processing.
  • Continue end-to-end testing from OMR input through playback and visualization.
  • Begin performance and stability testing under different tempos and input conditions.
Posted on

Chris Oh’s Status Report for March 28

This week I focused on integrating the software and hardware components of the system and stabilizing the end-to-end pipeline. The main effort involved connecting the playback and visualization software to the hardware layer, debugging integration issues, and resolving errors encountered during real-world execution. In parallel, I worked on fully integrating the OMR pipeline with the conversion layer and began addressing a Java-side error affecting OMR processing.

Software and Hardware Integration

  • Integrated the playback and visualization software with the hardware layer to enable end-to-end system execution.
  • Verified that processed musical events are correctly transmitted to the hardware during playback.
  • Encountered intermittent failures and runtime errors during integration, primarily due to timing, data formatting, and interface mismatches.
    Spent time debugging and resolving these issues to improve system stability and reliability.
  • Continued iterative testing to ensure that software-hardware communication behaves consistently during playback.

OMR and Conversion Layer Integration

  • Fully integrated the OMR pipeline with the conversion layer, enabling direct flow from sheet music input to processed musical events.
  • Verified that OMR outputs are correctly parsed and transformed into the system’s internal event representation.
  • Encountered a Java-related error within the OMR pipeline that affects certain inputs.
  • Began debugging the Java error and investigating its root cause to restore reliable OMR processing.

Schedule

I am currently on schedule. While integration exposed several failures and errors, this work is expected at this stage and is helping stabilize the system. Progress continues toward a fully integrated end-to-end pipeline.

Deliverables for Next Week

  • Resolve remaining software-hardware integration errors and improve system robustness.
  • Fix the Java error in the OMR pipeline and verify reliable sheet music processing.
  • Continue end-to-end testing from OMR input through playback and visualization.
  • Begin performance and stability testing under different tempos and input conditions.
Posted on

Team Status Report for March 21

Project Risks and Mitigation

A key ongoing risk on the software side is maintaining tight synchronization between playback events and visual feedback during sheet music visualization. If visual updates lag behind or drift from playback timing, users may receive misleading guidance, reducing the effectiveness of the system. This risk is especially relevant as the visualization layer becomes more tightly integrated with playback.

Fortunately, this risk is being actively and effectively managed in the current design. To mitigate this risk, the visualization system is driven directly by the existing playback event stream rather than introducing a separate timing mechanism. All computationally expensive processing remains in the preprocessing stage, and the visualization layer operates only on precomputed event data. This approach minimizes runtime overhead and reduces the likelihood of timing jitter during playback.

Another ongoing risk is variability in OMR accuracy across different input conditions, such as skewed images, shadows, or lower-quality photos. Inconsistent MusicXML output can propagate downstream and affect both event generation and visualization accuracy. This risk is being managed by expanding the preprocessing pipeline and introducing a benchmark dataset to evaluate performance across varied inputs. Additionally, validation and cleaning of MusicXML ensures that only structurally consistent outputs are passed to the Conversion Layer.

A related risk is pipeline fragility when handling edge cases, such as corrupted files, missing OMR outputs, or malformed MusicXML. These failures could disrupt the user experience if not handled properly. To mitigate this, the system now includes explicit error reporting at each stage of the pipeline, along with targeted tests for invalid inputs. This ensures failures are predictable, easier to debug, and do not silently propagate through the system.

As the system moves toward hardware integration, there is also a risk that mismatches between the software-generated event format and the ESP32 expectations could introduce timing or parsing issues. This is being addressed by clearly defining the event payload structure and validating serialization before integration, ensuring that communication between software and hardware remains consistent and reliable.

Design Changes

This week, the team implemented the sheet music visualization component and integrated it with the playback pipeline. The visualization layer now renders musical events in real time and highlights active notes based on playback position. This change improves system observability and usability by allowing users to visually verify that playback behavior matches the intended musical structure.

The visualization was designed to interface cleanly with the existing event-processing architecture, avoiding changes to the core timing logic. While this adds additional UI complexity, the design preserves modular boundaries by keeping visualization logic separate from event generation and hardware communication.

Schedule Update

There have been no changes to the project schedule. Development remains on track, and progress continues to align with the milestones outlined in the Gantt chart (Gantt).

Posted on

Chris Oh’s Status Report for March 21

This week I focused on implementing the sheet music visualization component and integrating it with the existing playback pipeline. The main goal was to move the visualizer from a conceptual framework into a functional system that can display musical information in sync with playback.

Sheet Music Visualization Implementation

  • Implemented the core sheet music visualization pipeline, enabling musical events to be rendered visually during playback.
  • Added logic to map parsed musical events to visual elements, allowing notes to be displayed in their correct temporal order.
  • Integrated playback position tracking so the visualizer updates in real time as the music progresses.
  • Implemented basic note highlighting behavior to reflect currently active notes during playback.
  • Ensured the visualization system cleanly interfaces with the existing event processing and playback logic.

Playback Integration

  • Connected the sheet music visualizer to the playback system so visual updates are driven directly by playback timing.
  • Refined event timing handling to keep visual transitions aligned with note-on and note-off events.
    Verified that visualization remains stable across different tempos and musical passages.

Schedule

I am currently on schedule. The sheet music visualizer has moved from a prototype framework to a working implementation, and it is now integrated with the playback pipeline. The project continues to make steady progress toward a cohesive end-to-end system.

Deliverables for Next Week

  • Improve visual clarity and layout of the sheet music display.
  • Add support for additional musical elements (e.g., rests or measure boundaries).
  • Continue testing synchronization between playback, visualization, and hardware output.
  • Begin refining the visualizer UI based on usability considerations.
Posted on

Chris Oh’s Status Report for Feb 28

This week I focused on implementing and testing the conversion layer pipeline and adding an audio preview feature to support early validation of the system. The goal was to ensure that sheet music can successfully be converted into event data and previewed before hardware integration. In addition, I improved the pipeline to support direct MusicXML uploads so the system can be tested without relying on the OMR stage.

Conversion Layer Implementation

  • Completed implementation of the functionality to convert parsed MusicXML scores into structured note events.
  • Added a sanity check tool that verifies MusicXML parsing and event generation by loading a test score and printing basic statistics such as number of notes and generated events.
  • Improved robustness of the conversion module so it can run independently from the repository root, which simplifies development and testing.
  • Extended the pipeline to support direct MusicXML and MXL file uploads, allowing users to bypass the OMR stage when testing the system with known sheet music files.

Audio Playback and Testing

  • Implemented a lightweight audio synthesis system that generates simple sine-wave playback from MIDI note values. This allows the generated event data to be previewed without needing hardware.
  • Added support for chords and basic attack/release envelopes to reduce clicking artifacts during playback.
  • Implemented an event playback session system that supports play, pause, and stop controls while tracking elapsed playback time.
  • Integrated the playback system into the Streamlit interface as an Audio Preview feature. Users can now upload sheet music, generate events, and preview the resulting audio with adjustable BPM.
  • Added playback progress visualization and improved session state management so playback behaves correctly within Streamlit’s execution model.

Status Report 4 Requirements: Part A

From a global perspective, LumiKey addresses a common barrier in music education: the difficulty of learning to read sheet music and translate it into correct key presses on a piano. Many beginners around the world have access to inexpensive keyboards but lack access to formal instruction or structured learning tools. LumiKey aims to reduce this barrier by visually guiding users to the correct keys using LED indicators driven by sheet music input. Because the system relies on widely available hardware components and open-source software, it has the potential to be used by learners in many different environments, including schools, homes, and self-study settings where formal music instruction may not be available.

Schedule

I am currently on schedule. The conversion layer pipeline is now functional and can generate note events from MusicXML files. The addition of the audio preview feature also provides a useful way to validate the correctness of event generation before integrating with the hardware layer.

Deliverables for Next Week

  • Connect the event generation pipeline to the LED hardware interface.
  • Begin testing event timing synchronization between software playback and LED output.
  • Continue improving robustness of the conversion pipeline with additional sheet music examples.
  • Perform integration testing of the Streamlit interface with the full software pipeline.
Posted on

Chris Oh’s Status Report for Feb 21

This week I focused on transitioning from architecture design into concrete implementation. I set up the project repository, implemented the foundational conversion layer logic, and built a placeholder Streamlit-based UI that connects to the processing pipeline.

Software Implementation Progress

  • Initialized and structured the main project repository to reflect the modular architecture defined previously (UI/Core, Conversion, Hardware Interface).
  • Ensured clear separation of responsibilities so each layer can be developed and tested independently.
  • Implemented core functionality of the conversion layer, including:
    • MusicXML parsing and note extraction
    • MIDI to 88-key index mapping logic
    • Raw Event IR construction
    • Chord grouping for simultaneous note events
  • Established a functional end-to-end pipeline from MusicXML input to structured Raw Event IR output.
  • Integrated a Streamlit-based UI (skeleton UI; not hooked with backend yet) that supports:
    • File upload (MusicXML input)
    • Mode selection (Beat Mode / No-Beat Mode)
    • BPM input field
    • Play/Pause controls
    • Real-time display of processed event data fordebugging and validation

System Integration and Testing

  • Tested the conversion pipeline using sample sheet music files to verify:
    • Correct note extraction
    • Accurate key index mapping
    • Proper grouping of simultaneous notes
    • Stable modular boundaries between layers
  • The system is a prototype that partially supports a working vertical slice from file input to structured event output through the UI.

Schedule

I am currently on schedule. The repository structure is established, the basic conversion logic is implemented, and the UI layer is sketched out. The project is now in active prototyping and ready for further feature expansion.

Deliverables for Next Week

  • Complete Beat Mode quantization logic.
  • Finalize JSON payload structure for hardware transmission.