Posted on

Gene Chang’s Status Report for March 28

This week I focused on advancing the hardware and interaction system of LumiKey in preparation for the upcoming demo. Building on the existing software pipeline, the goal this week was to improve real-time responsiveness and enable meaningful user interaction through key detection and visual feedback.

LED Timing and Visualization Improvements

A major focus this week was implementing LED fading behavior to better represent timing and note progression. Instead of simple on/off signals, LEDs now fade in a controlled manner to indicate when a note is approaching, improving the user’s ability to anticipate timing.

This enhancement makes the system more intuitive during playback and aligns the visual guidance more closely with musical rhythm. The fading behavior was tested alongside the existing event stream to ensure it remains synchronized with playback.

FSR Integration and Multiplexer Expansion

Another key area of progress was expanding key detection using FSR strips integrated through multiplexers. By incorporating the multiplexer, the system can now handle input from multiple keys while minimizing the number of required GPIO pins on the ESP32.

This setup allows for scalable input sensing across a larger range of keys. Initial testing confirmed that multiple FSR inputs can be read and processed, enabling the system to detect user interaction across different notes.

Some challenges remain with signal consistency, including occasional false positives or missed presses, likely due to noise or variability in pressure sensing. To address this, I began refining threshold detection and improving signal stability, which will continue to be optimized.

Interaction Modes Implementation

This week I also implemented multiple interaction modes to support different learning experiences:

Basic Mode: The system waits for the user to press the correct key before advancing, enabling step-by-step learning.
Beat Mode: Notes are aligned with a consistent tempo, allowing users to play along with timing guidance.
Score System: A scoring mechanism was introduced to track user performance based on accuracy.

These modes allow the system to move beyond passive visualization and support active learning through user input.

Integration with Event System

All hardware interactions, including LED output and FSR input detection, were integrated with the existing note event system. This ensures that both visualization and user input are driven by the same underlying data, maintaining synchronization between software and hardware components.

This unified approach reduces the risk of timing drift and simplifies debugging and further development.

Schedule

The project remains on schedule and is progressing toward a fully integrated demo. This week’s work focused on hardware functionality and interaction, aligning with the planned transition to a complete interactive system.

Deliverables for Next Week

Next week I will focus on final testing and supporting the live demo. This includes improving the reliability of FSR input detection, further tuning LED timing behavior, and resolving any integration issues between hardware and the software pipeline.

I will also continue refining interaction modes and ensuring that the system provides a smooth and intuitive user experience during demonstrations.

Posted on

Team Status Report for March 28

Project Risks and Mitigation

A key focus this week was reducing risks related to system stability during the upcoming demo. One major risk is inconsistent pipeline behavior across different inputs, such as skewed images, uneven lighting, or PDFs. This is being managed by refining preprocessing, selecting reliable demo inputs, and ensuring consistent outputs across repeated runs. Another risk is timing synchronization between playback, visualization, and hardware output. To address this, both playback and visualization continue to be driven from the same event stream, avoiding separate timing systems and reducing the chance of drift. Pipeline robustness is also a concern when handling edge cases like missing OMR outputs or malformed MusicXML. This was mitigated by improving error handling so the system clearly identifies which stage fails instead of failing silently. As hardware integration approaches, there is also a risk of mismatches between software-generated events and ESP32 expectations, which is being addressed by keeping the event format consistent and validating outputs before use.

On the hardware side, a key risk is unreliable input detection from the FSR strips, including missed key presses or false trigger signals. These issues may arise from electrical noise, signal interference, or variability in pressure sensing. This is being mitigated by improving signal stability through better wiring organization, consistent grounding, and isolating sensor lines where possible. In addition, software-level filtering and threshold calibration are being applied to distinguish intentional key presses from noise, ensuring more consistent and reliable input detection across all keys.

Design Changes

Design changes this week focused on improving stability and usability rather than adding new features. The preprocessing and scan pipeline were refined to handle a wider range of inputs more consistently, especially under varying lighting and image conditions. Error handling was improved across the pipeline to provide clearer feedback when failures occur. The playback and visualization components were also tested and slightly adjusted to ensure they remain synchronized and stable when running together. These changes improve overall system reliability without increasing complexity.

No changes made in the hardware components.

Schedule Update

The project remains on schedule and is ready for the upcoming demo. This week’s work focused on stabilization and integration, which aligns with the planned transition from development to demonstration and system validation.

Demonstrated Progress

The team successfully stabilized the full software pipeline for demo use. The system can now reliably process sheet music inputs, run the OMR pipeline, generate note events, and pass them into the playback and visualization system. Playback and visualization now operate together more consistently, allowing users to follow musical progress during playback. Improved error handling ensures that issues can be quickly identified and addressed during live demonstrations, resulting in a more reliable and demo-ready system.

Posted on

Simon Lee’s Status Report for March 28

This week I focused on preparing the LumiKey system for the upcoming demo by improving the stability and reliability of the full software pipeline. Building on last week’s work validating the end-to-end flow, the goal this week was to make sure the system behaves consistently under demo conditions and handles errors more gracefully.

Pipeline Stability Improvements

A major focus this week was improving the consistency of the full pipeline: image/PDF input → preprocessing → OMR → MusicXML cleaning → conversion → note events. While the pipeline was already functional, I worked on making sure it produces stable outputs across repeated runs and different input types.

I refined preprocessing parameters to better handle difficult cases such as skewed images, uneven lighting, and PDFs. I also made adjustments to ensure that intermediate outputs passed between stages are consistently formatted, which helped reduce unexpected parsing or conversion issues. These changes improved the overall reliability of the pipeline when tested on the benchmark dataset.

Debugging and Error Handling

To better prepare for demo scenarios, I improved error handling throughout the pipeline. The system now clearly reports which stage fails (preprocessing, OMR, MusicXML cleaning, or parsing) instead of failing silently. This makes it easier to diagnose issues quickly during live demonstrations.

I also addressed several edge cases discovered during testing, including:

  • OMR producing no output file
  • Invalid or empty MusicXML files
  • Inconsistent outputs from certain image inputs

By handling these cases explicitly, the system is now more robust and less likely to crash or produce confusing results during the demo.

End-to-End Testing for Demo Inputs

I continued testing the pipeline using the benchmark dataset and focused on selecting a subset of reliable demo inputs. These inputs were chosen based on consistent performance across preprocessing, OMR, and conversion stages.

I also verified that the outputs from the pipeline align with the expected structure of note events, ensuring they can be used directly by the playback system and hardware components.

Integration Readiness

In preparation for demo integration, I ensured that the event output format remains consistent and compatible with the existing playback and hardware systems. This helps ensure smooth communication between the software pipeline and the ESP32 during the demo.

I also worked with the team to verify that the full flow—from sheet music input to event generation—can be demonstrated clearly and reliably.

Schedule

The project remains on schedule and ready for the upcoming demo. The software pipeline is now stable enough to support live demonstrations, and remaining work is focused on final integration and polishing.

Deliverables for Next Week (Demo Execution and Final Refinements)

Next week I will focus on supporting the demo on Monday and Wednesday and addressing any issues that arise during live testing. After the demo, I plan to continue refining the pipeline based on observed edge cases, especially improving robustness for more diverse inputs.

I will also continue working with the team on deeper integration with the hardware system and identifying any improvements needed for timing, accuracy, or user experience as we move toward the final system.

Posted on

Chris Oh’s Status Report for March 28

This week I focused on integrating the software and hardware components of the system and stabilizing the end-to-end pipeline. The main effort involved connecting the playback and visualization software to the hardware layer, debugging integration issues, and resolving errors encountered during real-world execution. In parallel, I worked on fully integrating the OMR pipeline with the conversion layer and began addressing a Java-side error affecting OMR processing.

Software and Hardware Integration

  • Integrated the playback and visualization software with the hardware layer to enable end-to-end system execution.
  • Verified that processed musical events are correctly transmitted to the hardware during playback.
  • Encountered intermittent failures and runtime errors during integration, primarily due to timing, data formatting, and interface mismatches.
    Spent time debugging and resolving these issues to improve system stability and reliability.
  • Continued iterative testing to ensure that software-hardware communication behaves consistently during playback.

OMR and Conversion Layer Integration

  • Fully integrated the OMR pipeline with the conversion layer, enabling direct flow from sheet music input to processed musical events.
  • Verified that OMR outputs are correctly parsed and transformed into the system’s internal event representation.
  • Encountered a Java-related error within the OMR pipeline that affects certain inputs.
  • Began debugging the Java error and investigating its root cause to restore reliable OMR processing.

Schedule

I am currently on schedule. While integration exposed several failures and errors, this work is expected at this stage and is helping stabilize the system. Progress continues toward a fully integrated end-to-end pipeline.

Deliverables for Next Week

  • Resolve remaining software-hardware integration errors and improve system robustness.
  • Fix the Java error in the OMR pipeline and verify reliable sheet music processing.
  • Continue end-to-end testing from OMR input through playback and visualization.
  • Begin performance and stability testing under different tempos and input conditions.
Posted on

Gene Chang’s Status Report for March 21

This week primarily focused on continued development and debugging of the force-sensitive resistor (FSR) input system for the LumiKeys project. Progress was centered on stabilizing the detection of key presses across a smaller subset of the keyboard, specifically five keys, before scaling to the full system.

FSR Integration and Debugging

  • Worked on refining the code for reading and interpreting signals from five FSR strips connected through the analog multiplexer.
  • Focused on improving signal reliability and consistency, as initial readings showed noise and occasional false triggers.
  • Tested different threshold values and filtering approaches in software to better distinguish intentional key presses from background noise.
  • Continued debugging wiring and connection stability to ensure that each FSR input is correctly mapped and independently detected.

System Stability and Validation

  • Conducted repeated tests on the five-key setup to verify that key presses are consistently detected and transmitted.
  • Identified minor inconsistencies in readings, which are currently being investigated as potential issues related to wiring layout or power distribution.
  • Maintained the goal of achieving a stable and reliable baseline before expanding to additional keys.

Schedule

I am slightly behind the ideal timeline due to extended debugging of the FSR inputs, but this step is critical to ensure reliability before scaling to the full keyboard system.

Deliverables for Next Week

  • Finalize stable detection for the five FSR keys with minimal noise and false positives.
  • Implement improved filtering or smoothing techniques if necessary to enhance signal accuracy.
  • Begin scaling the working solution to additional keys once consistent performance is achieved.
Posted on

Team Status Report for March 21

Project Risks and Mitigation

A key ongoing risk on the software side is maintaining tight synchronization between playback events and visual feedback during sheet music visualization. If visual updates lag behind or drift from playback timing, users may receive misleading guidance, reducing the effectiveness of the system. This risk is especially relevant as the visualization layer becomes more tightly integrated with playback.

Fortunately, this risk is being actively and effectively managed in the current design. To mitigate this risk, the visualization system is driven directly by the existing playback event stream rather than introducing a separate timing mechanism. All computationally expensive processing remains in the preprocessing stage, and the visualization layer operates only on precomputed event data. This approach minimizes runtime overhead and reduces the likelihood of timing jitter during playback.

Another ongoing risk is variability in OMR accuracy across different input conditions, such as skewed images, shadows, or lower-quality photos. Inconsistent MusicXML output can propagate downstream and affect both event generation and visualization accuracy. This risk is being managed by expanding the preprocessing pipeline and introducing a benchmark dataset to evaluate performance across varied inputs. Additionally, validation and cleaning of MusicXML ensures that only structurally consistent outputs are passed to the Conversion Layer.

A related risk is pipeline fragility when handling edge cases, such as corrupted files, missing OMR outputs, or malformed MusicXML. These failures could disrupt the user experience if not handled properly. To mitigate this, the system now includes explicit error reporting at each stage of the pipeline, along with targeted tests for invalid inputs. This ensures failures are predictable, easier to debug, and do not silently propagate through the system.

As the system moves toward hardware integration, there is also a risk that mismatches between the software-generated event format and the ESP32 expectations could introduce timing or parsing issues. This is being addressed by clearly defining the event payload structure and validating serialization before integration, ensuring that communication between software and hardware remains consistent and reliable.

Design Changes

This week, the team implemented the sheet music visualization component and integrated it with the playback pipeline. The visualization layer now renders musical events in real time and highlights active notes based on playback position. This change improves system observability and usability by allowing users to visually verify that playback behavior matches the intended musical structure.

The visualization was designed to interface cleanly with the existing event-processing architecture, avoiding changes to the core timing logic. While this adds additional UI complexity, the design preserves modular boundaries by keeping visualization logic separate from event generation and hardware communication.

Schedule Update

There have been no changes to the project schedule. Development remains on track, and progress continues to align with the milestones outlined in the Gantt chart (Gantt).

Posted on

Simon Lee’s Status Report for March 21

This week I focused on validating and stabilizing the full end-to-end LumiKey software pipeline. Building on last week’s work where the Scan Layer was integrated with the OMR engine and connected to the existing Conversion Layer, the goal this week was to ensure that the entire flow – from image or PDF input to note event generation – works reliably and is ready for the interim demo.

End-to-End Pipeline Validation

A major focus this week was testing the complete pipeline: image/PDF input → preprocessing → OMR → MusicXML cleaning → conversion → note events. Since the Conversion Layer was already implemented by another team member, my work focused on verifying that the Scan Layer output integrates correctly and consistently produces valid inputs for downstream processing.

To support this, I created a small benchmark dataset of sheet music inputs, including clean images, skewed photos, shadowed images, and PDFs. This allowed me to test how well the system performs across realistic conditions. I also began evaluating outputs based on whether the pipeline succeeds, whether parsing errors occur, and whether the generated events are reasonable in terms of structure and count.

Evaluation and Debugging Improvements

I added a simple evaluation workflow that runs the pipeline across the benchmark set and reports results such as success/failure and basic output consistency. This helped identify weak points in preprocessing and OMR where certain inputs fail or produce inconsistent results.

In addition, I improved debugging visibility by ensuring that failures are tied to specific stages of the pipeline. Instead of failing silently, the system now makes it clearer whether an issue occurs during preprocessing, OMR execution, MusicXML cleaning, or parsing. This makes it much easier to diagnose problems and iterate quickly.

Additional Testing and Edge Case Handling

I added several targeted tests to improve robustness of the Scan Layer and pipeline. These include handling invalid inputs such as corrupted images, malformed PDFs, missing OMR outputs, and invalid MusicXML files. These tests ensure that the system fails in a predictable and controlled way rather than crashing or producing unusable results.

The goal of this work is to make the pipeline stable enough for demo conditions, where unexpected inputs or edge cases may occur.

Preparation for Hardware Integration

With the software pipeline now functioning end-to-end, I began preparing for integration with the hardware system. This includes defining and documenting the event format generated by the pipeline, which will be used to communicate note timing and pitch information to the ESP32.

This step is important to ensure that the software output aligns with the expectations of the embedded system, particularly for timing accuracy and LED control. It also sets up the next phase of development, where software-generated note events will drive real hardware behavior.

Schedule

The project remains on schedule and aligned with the interim demo timeline. According to the planned system pipeline and milestones , the current focus on integration and validation is consistent with reaching an MVP before the demo. The full software pipeline is now operational, and remaining work is focused on stability and hardware integration.

Deliverables for Next Week (Interim Demo Preparation)

Next week I will focus on final preparation for the interim demo. This includes refining the pipeline to ensure consistent behavior on demo inputs, improving error handling for a smoother user experience, and working with the team to connect the software output to the ESP32 hardware.

The goal is to demonstrate a complete working system where a user uploads sheet music and receives corresponding LED guidance on the keyboard. I will also continue refining preprocessing and validation steps based on any issues observed during integration testing.

Posted on

Chris Oh’s Status Report for March 21

This week I focused on implementing the sheet music visualization component and integrating it with the existing playback pipeline. The main goal was to move the visualizer from a conceptual framework into a functional system that can display musical information in sync with playback.

Sheet Music Visualization Implementation

  • Implemented the core sheet music visualization pipeline, enabling musical events to be rendered visually during playback.
  • Added logic to map parsed musical events to visual elements, allowing notes to be displayed in their correct temporal order.
  • Integrated playback position tracking so the visualizer updates in real time as the music progresses.
  • Implemented basic note highlighting behavior to reflect currently active notes during playback.
  • Ensured the visualization system cleanly interfaces with the existing event processing and playback logic.

Playback Integration

  • Connected the sheet music visualizer to the playback system so visual updates are driven directly by playback timing.
  • Refined event timing handling to keep visual transitions aligned with note-on and note-off events.
    Verified that visualization remains stable across different tempos and musical passages.

Schedule

I am currently on schedule. The sheet music visualizer has moved from a prototype framework to a working implementation, and it is now integrated with the playback pipeline. The project continues to make steady progress toward a cohesive end-to-end system.

Deliverables for Next Week

  • Improve visual clarity and layout of the sheet music display.
  • Add support for additional musical elements (e.g., rests or measure boundaries).
  • Continue testing synchronization between playback, visualization, and hardware output.
  • Begin refining the visualizer UI based on usability considerations.
Posted on

Gene Chang’s Status Report for March 14

This week focused mainly on wiring additional FSR strips to the 16-channel analog multiplexer and ensuring that each key press can be individually detected and registered. Currently, I’m encountering an issue where ghost signals appear from other keys. I suspect this may be caused either by the wires being routed too closely together or by a power stability issue. I also reviewed the firmware to determine whether the problem could be mitigated through software.

A significant portion of the week was also spent soldering the LEDs and FSR strips together for the hardware assembly.