Posted on

Simon Lee’s Status Report for April 18

This week I focused on executing the verification and validation plans defined previously and analyzing the results to evaluate the reliability of the Scan Layer and full LumiKey pipeline. Building on the testing framework outlined last week, my goal was to run the planned tests, identify failure patterns, and refine the system based on observed issues. In addition, I began preparing for the final presentation next week by ensuring the system is stable and clearly demonstrable.

Verification Results for Scan Layer

I ran the full pipeline on the benchmark dataset of sheet music inputs, which included clean images, skewed photos, shadowed images, and PDFs. For each input, I recorded whether valid MusicXML was produced and tracked the stage at which failures occurred.

The main results observed were:

  • The pipeline achieved a high success rate on clean images and PDFs, but performance dropped on more challenging inputs such as skewed or poorly lit photos.
  • Most failures occurred during the OMR stage, particularly when preprocessing did not fully normalize lighting or alignment.
  • Preprocessing and XML cleaning stages were generally stable, with fewer failures compared to OMR.
  • Output consistency was strong for repeated runs on the same input, indicating that the pipeline behaves deterministically.

Based on these results, I identified preprocessing improvements as a key area for increasing overall reliability, particularly for handling lighting variation and skew. The measured success rate is close to the target of 90 percent for clean and moderate inputs, but still needs improvement for more difficult cases.

Validation Results for Full System

At the system level, I evaluated whether the pipeline produces usable outputs for guided piano playback. For selected test pieces with known expected outputs, I compared generated note events against trusted references.

The results showed that:

  • For successful OMR outputs, the generated note events were generally correct in pitch and ordering.
  • Playback and visualization components were able to use the generated event stream without errors.
  • In cases where OMR output was incomplete or incorrect, the resulting note events were also affected.

These results confirm that the system behaves correctly when the Scan Layer succeeds, and that improving early-stage reliability is important for overall system performance.

Refinements Based on Testing

Based on the testing results, I made several refinements to improve system stability:

  • Adjusted preprocessing parameters to better handle skewed and unevenly lit images.
  • Improved error handling to detect and report incomplete OMR outputs.
    Strengthened logging to better track failure cases and support debugging.

These changes improved performance on several previously failing inputs and made the pipeline more stable overall.

Final Presentation Preparation

In preparation for the final presentation next week, I focused on ensuring that the system performs reliably on selected demo inputs. I identified a set of stable sheet music examples that consistently produce correct outputs and verified that they work end to end with playback and visualization.

I also reviewed the pipeline flow and key components so that I can clearly explain how the Scan Layer integrates with the rest of the system. This includes being able to describe preprocessing, OMR, and how outputs are used for playback and hardware interaction.

Schedule

The project remains on schedule. The system has progressed from testing into refinement and presentation readiness, which aligns with the final project timeline.

Deliverables for Next Week

Next week I will finalize preparation for the presentation and support the live demos on Monday and Wednesday. I will also address any last-minute issues that arise during testing and ensure the pipeline remains stable under demo conditions.

Additional Reflection

Throughout the development and testing of the Scan Layer, I learned several new tools and concepts that were necessary to complete the project. One major area was working with the Audiveris OMR engine, including how to execute it programmatically, handle different output formats, and debug cases where it failed to produce usable results. I also developed a deeper understanding of MusicXML and how it is structured and validated before being used by the rest of the system.

Another important area was image preprocessing using OpenCV. I learned how to apply techniques such as adaptive thresholding, illumination normalization, skew correction, and morphological filtering to improve the quality of sheet music inputs. These techniques were essential for making the OMR stage more reliable.

I also gained experience working with PDF processing tools such as pypdfium2 to support PDF inputs and integrate them into the same pipeline as images. In addition, I improved my understanding of building and debugging multi-stage pipelines, including how to design tests, track failures by stage, and analyze system performance using structured metrics.

To learn these tools and concepts, I used a combination of strategies. I referred to official documentation for libraries such as OpenCV and music21 to understand how functions work and how they should be used. I also used online resources such as Stack Overflow and GitHub discussions to troubleshoot specific issues. In many cases, I wrote small test scripts to experiment with different approaches and observe their behavior. I also watched short tutorials to quickly understand new concepts before applying them directly to the project.

Overall, these learning strategies allowed me to quickly acquire and apply new technical skills while working across different parts of the system.

Posted on

Gene Chang’s Status Report for April 4

This week I focused on expanding and integrating the hardware subsystem of the LumiKey project to support more complete functionality and better interaction with the software layer. Building on the previous prototype, my goal was to increase system coverage and establish a more seamless connection between hardware inputs and the software interface.

Hardware Expansion

A major focus this week was scaling the hardware setup from a smaller prototype to a 10-key system. I expanded both the LED array and the FSR strip to support 10 keys, allowing for more realistic interaction and better representation of actual piano input. This required updating the wiring and ensuring that each sensor and LED channel functioned reliably when scaled.

Audio Feedback Integration

I implemented a sound feedback feature where each key press detected on the FSR strip triggers a corresponding sound played on the laptop. This enables immediate auditory feedback, making the system feel more interactive and closer to a real piano experience. The integration ensures that physical input on the hardware directly maps to sound output in real time.

Software and Hardware Interface

Another key development was integrating the software UI with the hardware system. I established a bidirectional interface where inputs from the hardware trigger responses in the software, and interactions from the UI can activate corresponding LEDs on the hardware. Specifically, pressing a button on the software UI now correctly triggers the associated hardware component, allowing for synchronized control between the two layers.

Next Steps

Next steps include improving the reliability of the hardware sensing and LED signaling under continuous use, as well as refining timing synchronization between hardware input, sound output, and UI feedback. I will also begin preparing the system for structured testing as part of the overall verification phase.

Posted on

Chris Oh’s Status Report for April 4

This week I focused on integrating the software and hardware components of the system and stabilizing the end-to-end pipeline. The main effort involved connecting the playback and visualization software to the hardware layer, debugging integration issues, and resolving errors encountered during real-world execution. In parallel, I worked on fully integrating the OMR pipeline with the conversion layer and began addressing a Java-side error affecting OMR processing.

Software and Hardware Integration

  • Integrated the playback and visualization software with the hardware layer to enable end-to-end system execution.
  • Verified that processed musical events are correctly transmitted to the hardware during playback.
  • Encountered intermittent failures and runtime errors during integration, primarily due to timing, data formatting, and interface mismatches.
    Spent time debugging and resolving these issues to improve system stability and reliability.
  • Continued iterative testing to ensure that software-hardware communication behaves consistently during playback.

OMR and Conversion Layer Integration

  • Fully integrated the OMR pipeline with the conversion layer, enabling direct flow from sheet music input to processed musical events.
  • Verified that OMR outputs are correctly parsed and transformed into the system’s internal event representation.
  • Encountered a Java-related error within the OMR pipeline that affects certain inputs.
  • Began debugging the Java error and investigating its root cause to restore reliable OMR processing.

Schedule

I am currently on schedule. While integration exposed several failures and errors, this work is expected at this stage and is helping stabilize the system. Progress continues toward a fully integrated end-to-end pipeline.

Deliverables for Next Week

  • Resolve remaining software-hardware integration errors and improve system robustness.
  • Fix the Java error in the OMR pipeline and verify reliable sheet music processing.
  • Continue end-to-end testing from OMR input through playback and visualization.
  • Begin performance and stability testing under different tempos and input conditions.
Posted on

Team Status Report for April 4

Project Risks and Mitigation

As the project enters the validation phase, a key risk is that the system may not perform reliably under real user conditions, even if individual components work correctly. In particular, variability in sheet music inputs such as skewed images, shadows, or low-quality photos may affect the overall system output. This is being addressed by planning end-to-end validation tests using a range of realistic inputs to ensure the system behaves consistently beyond controlled demo cases.

Another important risk is maintaining synchronization between playback, visualization, and hardware output. While this was stabilized for the demo, validation will now confirm that timing remains consistent during full system operation. This is being managed by continuing to drive playback and visualization from the same event stream and testing the system over longer runs and different inputs.

A further risk is that errors from earlier stages may still impact the final user experience, even if they are detected. To mitigate this, the team is focusing on ensuring that failures are clearly communicated and that the system either produces usable output or fails in a predictable and understandable way.

On the hardware side, there is still a risk of inconsistent key detection from the FSR sensors. This will be evaluated during system-level validation by testing how well hardware input aligns with software-generated events, with filtering and calibration used to improve reliability.

Design Changes

This week did not introduce major architectural changes, but the focus shifted toward preparing the system for validation. Minor improvements were made to logging and output consistency so that system behavior can be more easily observed and evaluated during testing.

The system design continues to emphasize modularity, allowing individual components to remain independent while supporting full system validation. This ensures that issues identified during validation can be traced back to specific parts of the system.

Schedule Update

The project remains on schedule. The team has transitioned from demo preparation into the validation phase, which aligns with the planned development timeline. No major schedule changes are required.

Validation Plan

Validation will focus on evaluating the full system from a user perspective, ensuring that LumiKey works as intended for guided piano learning. The team plans to run several end-to-end tests that simulate real usage of the system.

The first set of tests will evaluate the complete user workflow, where a user uploads sheet music and the system processes it through the full pipeline to generate LED guidance and playback. The goal is to confirm that the system operates without failure and produces usable results.

The second set of tests will evaluate correctness of musical guidance. For selected test pieces with known outputs, the team will compare the generated note events and LED behavior against expected notes. This ensures that the system is guiding users correctly rather than simply producing output.

The third set of tests will focus on timing and synchronization, verifying that playback, visualization, and LED output remain aligned. This includes checking that the system responds within the required latency and that musical timing is preserved during playback.

The fourth set of tests will evaluate robustness under different input conditions, including clean PDFs, skewed images, and lower-quality photos. The goal is to ensure the system either produces correct results or fails in a controlled and understandable way.

Finally, hardware integration validation will confirm that the event data generated by the software is correctly interpreted by the ESP32, that LEDs display the correct notes, and that key press detection aligns with expected inputs.

The results of these validation tests will be used to determine whether the system meets the use-case requirements, particularly in terms of correctness, timing, and usability.

Demonstrated Progress

The team has successfully transitioned from demo readiness to a system capable of supporting full validation. The software pipeline remains stable and is able to process sheet music inputs, generate note events, and drive playback and visualization components.

In addition, the team has defined a clear validation strategy focused on real user behavior rather than individual components. This represents an important step toward ensuring that LumiKey not only functions technically but also meets its intended purpose as a guided music learning system.

Posted on

Simon Lee’s Status Report for April 4

This week I focused on transitioning from demo preparation into the verification and validation phase of the LumiKey project. Building on the previous work where the scan pipeline was stabilized for the demo, my goal this week was to define concrete testing procedures and evaluation metrics to measure whether the Scan Layer and full pipeline meet the design and use-case requirements.

Verification Planning for Scan Layer

For my subsystem, verification will focus on evaluating the performance and reliability of the Scan Layer, including preprocessing, OMR execution, and MusicXML cleaning. I plan to use a benchmark dataset of 15 to 25 sheet music inputs that include a mix of clean images, skewed photos, shadowed images, and PDFs to reflect realistic usage conditions.

The specific tests I plan to run include:

  • Running the full pipeline on each input and recording whether valid MusicXML is successfully produced.
  • Logging the exact stage where failures occur, such as preprocessing, OMR execution, or XML cleaning, to identify bottlenecks.
  • Measuring output consistency by running the same input multiple times and verifying that the resulting MusicXML and note events remain stable.
  • Comparing detected note counts and structural properties against expected values for selected test files.
  • Tracking error types such as missing OMR output, malformed XML, or parsing failures.

For analysis, I will compute the overall pipeline success rate across the dataset, with a target of at least 90 percent successful conversions. I will also analyze failure distributions by stage to determine whether issues are concentrated in preprocessing, OMR, or post-processing. Consistency checks will help ensure the pipeline behaves deterministically rather than producing variable outputs.

Validation Planning for Full System

At the system level, validation will focus on whether the full pipeline produces correct and usable outputs for the intended application of guided piano learning. This involves confirming that the generated note events accurately reflect the musical structure and can be used by the playback system.

The planned validation steps include:

  • Selecting 3 to 5 representative pieces with known correct outputs.
  • Comparing generated note events against trusted references such as manually verified MusicXML or expected MIDI sequences.
  • Evaluating correctness by checking pitch accuracy, note ordering, and basic timing relationships.
  • Verifying that the generated event stream integrates correctly with the playback and visualization system without errors or misalignment.
  • Observing whether the output is usable in practice, meaning that playback behavior matches the intended musical structure.

The main validation criteria will be whether the system produces correct and usable note events for known inputs and whether those outputs integrate cleanly into the rest of the system.

Next Steps for Testing

These tests have not yet been executed, so the next step is to implement evaluation scripts that automate running the pipeline on the dataset and collecting results. I will also improve logging so each test clearly reports success, failure, and failure reason.

After running the tests, I will analyze patterns in failure cases and identify areas where preprocessing, OMR integration, or XML cleaning need further refinement. This will allow targeted improvements rather than ad hoc debugging.

Schedule

The project remains on schedule. The system has transitioned from demo readiness into structured verification and validation, which aligns with the planned development timeline.

Deliverables for Next Week

Next week I will execute the planned verification tests on the benchmark dataset and analyze results against the defined success criteria. Based on these results, I will refine preprocessing and OMR handling to improve reliability. I will also continue validating system-level behavior by ensuring that the generated outputs integrate correctly with playback, visualization, and hardware components.

Posted on

Gene Chang’s Status Report for March 28

This week I focused on advancing the hardware and interaction system of LumiKey in preparation for the upcoming demo. Building on the existing software pipeline, the goal this week was to improve real-time responsiveness and enable meaningful user interaction through key detection and visual feedback.

LED Timing and Visualization Improvements

A major focus this week was implementing LED fading behavior to better represent timing and note progression. Instead of simple on/off signals, LEDs now fade in a controlled manner to indicate when a note is approaching, improving the user’s ability to anticipate timing.

This enhancement makes the system more intuitive during playback and aligns the visual guidance more closely with musical rhythm. The fading behavior was tested alongside the existing event stream to ensure it remains synchronized with playback.

FSR Integration and Multiplexer Expansion

Another key area of progress was expanding key detection using FSR strips integrated through multiplexers. By incorporating the multiplexer, the system can now handle input from multiple keys while minimizing the number of required GPIO pins on the ESP32.

This setup allows for scalable input sensing across a larger range of keys. Initial testing confirmed that multiple FSR inputs can be read and processed, enabling the system to detect user interaction across different notes.

Some challenges remain with signal consistency, including occasional false positives or missed presses, likely due to noise or variability in pressure sensing. To address this, I began refining threshold detection and improving signal stability, which will continue to be optimized.

Interaction Modes Implementation

This week I also implemented multiple interaction modes to support different learning experiences:

Basic Mode: The system waits for the user to press the correct key before advancing, enabling step-by-step learning.
Beat Mode: Notes are aligned with a consistent tempo, allowing users to play along with timing guidance.
Score System: A scoring mechanism was introduced to track user performance based on accuracy.

These modes allow the system to move beyond passive visualization and support active learning through user input.

Integration with Event System

All hardware interactions, including LED output and FSR input detection, were integrated with the existing note event system. This ensures that both visualization and user input are driven by the same underlying data, maintaining synchronization between software and hardware components.

This unified approach reduces the risk of timing drift and simplifies debugging and further development.

Schedule

The project remains on schedule and is progressing toward a fully integrated demo. This week’s work focused on hardware functionality and interaction, aligning with the planned transition to a complete interactive system.

Deliverables for Next Week

Next week I will focus on final testing and supporting the live demo. This includes improving the reliability of FSR input detection, further tuning LED timing behavior, and resolving any integration issues between hardware and the software pipeline.

I will also continue refining interaction modes and ensuring that the system provides a smooth and intuitive user experience during demonstrations.

Posted on

Team Status Report for March 28

Project Risks and Mitigation

A key focus this week was reducing risks related to system stability during the upcoming demo. One major risk is inconsistent pipeline behavior across different inputs, such as skewed images, uneven lighting, or PDFs. This is being managed by refining preprocessing, selecting reliable demo inputs, and ensuring consistent outputs across repeated runs. Another risk is timing synchronization between playback, visualization, and hardware output. To address this, both playback and visualization continue to be driven from the same event stream, avoiding separate timing systems and reducing the chance of drift. Pipeline robustness is also a concern when handling edge cases like missing OMR outputs or malformed MusicXML. This was mitigated by improving error handling so the system clearly identifies which stage fails instead of failing silently. As hardware integration approaches, there is also a risk of mismatches between software-generated events and ESP32 expectations, which is being addressed by keeping the event format consistent and validating outputs before use.

On the hardware side, a key risk is unreliable input detection from the FSR strips, including missed key presses or false trigger signals. These issues may arise from electrical noise, signal interference, or variability in pressure sensing. This is being mitigated by improving signal stability through better wiring organization, consistent grounding, and isolating sensor lines where possible. In addition, software-level filtering and threshold calibration are being applied to distinguish intentional key presses from noise, ensuring more consistent and reliable input detection across all keys.

Design Changes

Design changes this week focused on improving stability and usability rather than adding new features. The preprocessing and scan pipeline were refined to handle a wider range of inputs more consistently, especially under varying lighting and image conditions. Error handling was improved across the pipeline to provide clearer feedback when failures occur. The playback and visualization components were also tested and slightly adjusted to ensure they remain synchronized and stable when running together. These changes improve overall system reliability without increasing complexity.

No changes made in the hardware components.

Schedule Update

The project remains on schedule and is ready for the upcoming demo. This week’s work focused on stabilization and integration, which aligns with the planned transition from development to demonstration and system validation.

Demonstrated Progress

The team successfully stabilized the full software pipeline for demo use. The system can now reliably process sheet music inputs, run the OMR pipeline, generate note events, and pass them into the playback and visualization system. Playback and visualization now operate together more consistently, allowing users to follow musical progress during playback. Improved error handling ensures that issues can be quickly identified and addressed during live demonstrations, resulting in a more reliable and demo-ready system.

Posted on

Simon Lee’s Status Report for March 28

This week I focused on preparing the LumiKey system for the upcoming demo by improving the stability and reliability of the full software pipeline. Building on last week’s work validating the end-to-end flow, the goal this week was to make sure the system behaves consistently under demo conditions and handles errors more gracefully.

Pipeline Stability Improvements

A major focus this week was improving the consistency of the full pipeline: image/PDF input → preprocessing → OMR → MusicXML cleaning → conversion → note events. While the pipeline was already functional, I worked on making sure it produces stable outputs across repeated runs and different input types.

I refined preprocessing parameters to better handle difficult cases such as skewed images, uneven lighting, and PDFs. I also made adjustments to ensure that intermediate outputs passed between stages are consistently formatted, which helped reduce unexpected parsing or conversion issues. These changes improved the overall reliability of the pipeline when tested on the benchmark dataset.

Debugging and Error Handling

To better prepare for demo scenarios, I improved error handling throughout the pipeline. The system now clearly reports which stage fails (preprocessing, OMR, MusicXML cleaning, or parsing) instead of failing silently. This makes it easier to diagnose issues quickly during live demonstrations.

I also addressed several edge cases discovered during testing, including:

  • OMR producing no output file
  • Invalid or empty MusicXML files
  • Inconsistent outputs from certain image inputs

By handling these cases explicitly, the system is now more robust and less likely to crash or produce confusing results during the demo.

End-to-End Testing for Demo Inputs

I continued testing the pipeline using the benchmark dataset and focused on selecting a subset of reliable demo inputs. These inputs were chosen based on consistent performance across preprocessing, OMR, and conversion stages.

I also verified that the outputs from the pipeline align with the expected structure of note events, ensuring they can be used directly by the playback system and hardware components.

Integration Readiness

In preparation for demo integration, I ensured that the event output format remains consistent and compatible with the existing playback and hardware systems. This helps ensure smooth communication between the software pipeline and the ESP32 during the demo.

I also worked with the team to verify that the full flow—from sheet music input to event generation—can be demonstrated clearly and reliably.

Schedule

The project remains on schedule and ready for the upcoming demo. The software pipeline is now stable enough to support live demonstrations, and remaining work is focused on final integration and polishing.

Deliverables for Next Week (Demo Execution and Final Refinements)

Next week I will focus on supporting the demo on Monday and Wednesday and addressing any issues that arise during live testing. After the demo, I plan to continue refining the pipeline based on observed edge cases, especially improving robustness for more diverse inputs.

I will also continue working with the team on deeper integration with the hardware system and identifying any improvements needed for timing, accuracy, or user experience as we move toward the final system.

Posted on

Chris Oh’s Status Report for March 28

This week I focused on integrating the software and hardware components of the system and stabilizing the end-to-end pipeline. The main effort involved connecting the playback and visualization software to the hardware layer, debugging integration issues, and resolving errors encountered during real-world execution. In parallel, I worked on fully integrating the OMR pipeline with the conversion layer and began addressing a Java-side error affecting OMR processing.

Software and Hardware Integration

  • Integrated the playback and visualization software with the hardware layer to enable end-to-end system execution.
  • Verified that processed musical events are correctly transmitted to the hardware during playback.
  • Encountered intermittent failures and runtime errors during integration, primarily due to timing, data formatting, and interface mismatches.
    Spent time debugging and resolving these issues to improve system stability and reliability.
  • Continued iterative testing to ensure that software-hardware communication behaves consistently during playback.

OMR and Conversion Layer Integration

  • Fully integrated the OMR pipeline with the conversion layer, enabling direct flow from sheet music input to processed musical events.
  • Verified that OMR outputs are correctly parsed and transformed into the system’s internal event representation.
  • Encountered a Java-related error within the OMR pipeline that affects certain inputs.
  • Began debugging the Java error and investigating its root cause to restore reliable OMR processing.

Schedule

I am currently on schedule. While integration exposed several failures and errors, this work is expected at this stage and is helping stabilize the system. Progress continues toward a fully integrated end-to-end pipeline.

Deliverables for Next Week

  • Resolve remaining software-hardware integration errors and improve system robustness.
  • Fix the Java error in the OMR pipeline and verify reliable sheet music processing.
  • Continue end-to-end testing from OMR input through playback and visualization.
  • Begin performance and stability testing under different tempos and input conditions.