Final Video
Simon Lee’s Status Report for April 25
This week I focused on finalizing and stabilizing the Scan Layer and overall software pipeline in preparation for the demo. As I began testing more complex sheet music inputs, especially larger files and higher-resolution PDFs, I encountered issues where the system would produce unreliable or blank outputs. My main goal this week was to resolve these issues and ensure consistent behavior across a wider range of inputs.
Pipeline Refinement for Complex Inputs
While testing more complex sheet music, I found that certain inputs caused failures in the OMR stage. In particular, large or high-resolution files sometimes exceeded what the OMR engine could handle, and lower-quality inputs occasionally failed to produce usable results.
To address this, I restructured parts of the Scan Layer pipeline to better balance input quality and processing reliability. I introduced normalization steps for both image and PDF inputs before they are passed into the OMR stage, ensuring that inputs fall within a more consistent and OMR-friendly range. This included resizing inputs to appropriate resolutions and standardizing preprocessing behavior across different file types.
I also improved how the system handles cases where the OMR output is incorrect but not completely missing. In some cases, notes were misinterpreted in unrealistic pitch ranges, which caused incorrect playback. To mitigate this, I added a correction step that adjusts clearly invalid note ranges back into a realistic piano range. This ensures that even imperfect OMR outputs remain usable for downstream playback.
These changes significantly improved the consistency of both image and PDF processing and reduced cases where the pipeline produced no output.
Improved Robustness and User Feedback
In addition to backend improvements, I added better user feedback during the scan process. The system now previews uploaded images and PDFs before processing, allowing users to confirm that the correct file was selected. This helps prevent user error and improves overall usability.
I also ensured that failures are more clearly handled, so that if the pipeline encounters an issue, it does not silently fail but instead provides more visible feedback.
Verification and Stability
After these updates, I reran the verification suite to ensure the pipeline remained stable. The tests passed successfully, confirming that the changes did not introduce regressions and that the system continues to meet the required correctness and performance criteria.
Testing across a wider range of inputs also gave more confidence that the system behaves reliably under more realistic conditions, not just ideal cases.
Demo Preparation
For demo preparation, I tested the full pipeline multiple times using a variety of inputs to ensure consistent behavior. I identified a set of reliable demo cases and verified that they produce correct outputs and smooth playback.
I also reviewed the full system flow and ensured I am prepared to explain how the Scan Layer integrates with the rest of the pipeline, as well as the key design decisions and trade-offs.
Schedule
The project is on schedule and ready for the demo. The system is now more robust, handles a wider range of inputs, and has been verified against key requirements.
Deliverables for Demo
For the demo, the system will show the full pipeline from sheet music input to LED guidance and playback. I will focus on demonstrating system reliability, correctness of note generation, and responsiveness, while also being prepared to discuss the challenges encountered and how they were resolved.
Simon Lee’s Status Report for April 18
This week I focused on executing the verification and validation plans defined previously and analyzing the results to evaluate the reliability of the Scan Layer and full LumiKey pipeline. Building on the testing framework outlined last week, my goal was to run the planned tests, identify failure patterns, and refine the system based on observed issues. In addition, I began preparing for the final presentation next week by ensuring the system is stable and clearly demonstrable.
Verification Results for Scan Layer
I ran the full pipeline on the benchmark dataset of sheet music inputs, which included clean images, skewed photos, shadowed images, and PDFs. For each input, I recorded whether valid MusicXML was produced and tracked the stage at which failures occurred.
The main results observed were:
- The pipeline achieved a high success rate on clean images and PDFs, but performance dropped on more challenging inputs such as skewed or poorly lit photos.
- Most failures occurred during the OMR stage, particularly when preprocessing did not fully normalize lighting or alignment.
- Preprocessing and XML cleaning stages were generally stable, with fewer failures compared to OMR.
- Output consistency was strong for repeated runs on the same input, indicating that the pipeline behaves deterministically.
Based on these results, I identified preprocessing improvements as a key area for increasing overall reliability, particularly for handling lighting variation and skew. The measured success rate is close to the target of 90 percent for clean and moderate inputs, but still needs improvement for more difficult cases.
Validation Results for Full System
At the system level, I evaluated whether the pipeline produces usable outputs for guided piano playback. For selected test pieces with known expected outputs, I compared generated note events against trusted references.
The results showed that:
- For successful OMR outputs, the generated note events were generally correct in pitch and ordering.
- Playback and visualization components were able to use the generated event stream without errors.
- In cases where OMR output was incomplete or incorrect, the resulting note events were also affected.
These results confirm that the system behaves correctly when the Scan Layer succeeds, and that improving early-stage reliability is important for overall system performance.
Refinements Based on Testing
Based on the testing results, I made several refinements to improve system stability:
- Adjusted preprocessing parameters to better handle skewed and unevenly lit images.
- Improved error handling to detect and report incomplete OMR outputs.
Strengthened logging to better track failure cases and support debugging.
These changes improved performance on several previously failing inputs and made the pipeline more stable overall.
Final Presentation Preparation
In preparation for the final presentation next week, I focused on ensuring that the system performs reliably on selected demo inputs. I identified a set of stable sheet music examples that consistently produce correct outputs and verified that they work end to end with playback and visualization.
I also reviewed the pipeline flow and key components so that I can clearly explain how the Scan Layer integrates with the rest of the system. This includes being able to describe preprocessing, OMR, and how outputs are used for playback and hardware interaction.
Schedule
The project remains on schedule. The system has progressed from testing into refinement and presentation readiness, which aligns with the final project timeline.
Deliverables for Next Week
Next week I will finalize preparation for the presentation and support the live demos on Monday and Wednesday. I will also address any last-minute issues that arise during testing and ensure the pipeline remains stable under demo conditions.
Additional Reflection
Throughout the development and testing of the Scan Layer, I learned several new tools and concepts that were necessary to complete the project. One major area was working with the Audiveris OMR engine, including how to execute it programmatically, handle different output formats, and debug cases where it failed to produce usable results. I also developed a deeper understanding of MusicXML and how it is structured and validated before being used by the rest of the system.
Another important area was image preprocessing using OpenCV. I learned how to apply techniques such as adaptive thresholding, illumination normalization, skew correction, and morphological filtering to improve the quality of sheet music inputs. These techniques were essential for making the OMR stage more reliable.
I also gained experience working with PDF processing tools such as pypdfium2 to support PDF inputs and integrate them into the same pipeline as images. In addition, I improved my understanding of building and debugging multi-stage pipelines, including how to design tests, track failures by stage, and analyze system performance using structured metrics.
To learn these tools and concepts, I used a combination of strategies. I referred to official documentation for libraries such as OpenCV and music21 to understand how functions work and how they should be used. I also used online resources such as Stack Overflow and GitHub discussions to troubleshoot specific issues. In many cases, I wrote small test scripts to experiment with different approaches and observe their behavior. I also watched short tutorials to quickly understand new concepts before applying them directly to the project.
Overall, these learning strategies allowed me to quickly acquire and apply new technical skills while working across different parts of the system.
Team Status Report for April 4
Project Risks and Mitigation
As the project enters the validation phase, a key risk is that the system may not perform reliably under real user conditions, even if individual components work correctly. In particular, variability in sheet music inputs such as skewed images, shadows, or low-quality photos may affect the overall system output. This is being addressed by planning end-to-end validation tests using a range of realistic inputs to ensure the system behaves consistently beyond controlled demo cases.
Another important risk is maintaining synchronization between playback, visualization, and hardware output. While this was stabilized for the demo, validation will now confirm that timing remains consistent during full system operation. This is being managed by continuing to drive playback and visualization from the same event stream and testing the system over longer runs and different inputs.
A further risk is that errors from earlier stages may still impact the final user experience, even if they are detected. To mitigate this, the team is focusing on ensuring that failures are clearly communicated and that the system either produces usable output or fails in a predictable and understandable way.
On the hardware side, there is still a risk of inconsistent key detection from the FSR sensors. This will be evaluated during system-level validation by testing how well hardware input aligns with software-generated events, with filtering and calibration used to improve reliability.
Design Changes
This week did not introduce major architectural changes, but the focus shifted toward preparing the system for validation. Minor improvements were made to logging and output consistency so that system behavior can be more easily observed and evaluated during testing.
The system design continues to emphasize modularity, allowing individual components to remain independent while supporting full system validation. This ensures that issues identified during validation can be traced back to specific parts of the system.
Schedule Update
The project remains on schedule. The team has transitioned from demo preparation into the validation phase, which aligns with the planned development timeline. No major schedule changes are required.
Validation Plan
Validation will focus on evaluating the full system from a user perspective, ensuring that LumiKey works as intended for guided piano learning. The team plans to run several end-to-end tests that simulate real usage of the system.
The first set of tests will evaluate the complete user workflow, where a user uploads sheet music and the system processes it through the full pipeline to generate LED guidance and playback. The goal is to confirm that the system operates without failure and produces usable results.
The second set of tests will evaluate correctness of musical guidance. For selected test pieces with known outputs, the team will compare the generated note events and LED behavior against expected notes. This ensures that the system is guiding users correctly rather than simply producing output.
The third set of tests will focus on timing and synchronization, verifying that playback, visualization, and LED output remain aligned. This includes checking that the system responds within the required latency and that musical timing is preserved during playback.
The fourth set of tests will evaluate robustness under different input conditions, including clean PDFs, skewed images, and lower-quality photos. The goal is to ensure the system either produces correct results or fails in a controlled and understandable way.
Finally, hardware integration validation will confirm that the event data generated by the software is correctly interpreted by the ESP32, that LEDs display the correct notes, and that key press detection aligns with expected inputs.
The results of these validation tests will be used to determine whether the system meets the use-case requirements, particularly in terms of correctness, timing, and usability.
Demonstrated Progress
The team has successfully transitioned from demo readiness to a system capable of supporting full validation. The software pipeline remains stable and is able to process sheet music inputs, generate note events, and drive playback and visualization components.
In addition, the team has defined a clear validation strategy focused on real user behavior rather than individual components. This represents an important step toward ensuring that LumiKey not only functions technically but also meets its intended purpose as a guided music learning system.
Simon Lee’s Status Report for April 4
This week I focused on transitioning from demo preparation into the verification and validation phase of the LumiKey project. Building on the previous work where the scan pipeline was stabilized for the demo, my goal this week was to define concrete testing procedures and evaluation metrics to measure whether the Scan Layer and full pipeline meet the design and use-case requirements.
Verification Planning for Scan Layer
For my subsystem, verification will focus on evaluating the performance and reliability of the Scan Layer, including preprocessing, OMR execution, and MusicXML cleaning. I plan to use a benchmark dataset of 15 to 25 sheet music inputs that include a mix of clean images, skewed photos, shadowed images, and PDFs to reflect realistic usage conditions.
The specific tests I plan to run include:
- Running the full pipeline on each input and recording whether valid MusicXML is successfully produced.
- Logging the exact stage where failures occur, such as preprocessing, OMR execution, or XML cleaning, to identify bottlenecks.
- Measuring output consistency by running the same input multiple times and verifying that the resulting MusicXML and note events remain stable.
- Comparing detected note counts and structural properties against expected values for selected test files.
- Tracking error types such as missing OMR output, malformed XML, or parsing failures.
For analysis, I will compute the overall pipeline success rate across the dataset, with a target of at least 90 percent successful conversions. I will also analyze failure distributions by stage to determine whether issues are concentrated in preprocessing, OMR, or post-processing. Consistency checks will help ensure the pipeline behaves deterministically rather than producing variable outputs.
Validation Planning for Full System
At the system level, validation will focus on whether the full pipeline produces correct and usable outputs for the intended application of guided piano learning. This involves confirming that the generated note events accurately reflect the musical structure and can be used by the playback system.
The planned validation steps include:
- Selecting 3 to 5 representative pieces with known correct outputs.
- Comparing generated note events against trusted references such as manually verified MusicXML or expected MIDI sequences.
- Evaluating correctness by checking pitch accuracy, note ordering, and basic timing relationships.
- Verifying that the generated event stream integrates correctly with the playback and visualization system without errors or misalignment.
- Observing whether the output is usable in practice, meaning that playback behavior matches the intended musical structure.
The main validation criteria will be whether the system produces correct and usable note events for known inputs and whether those outputs integrate cleanly into the rest of the system.
Next Steps for Testing
These tests have not yet been executed, so the next step is to implement evaluation scripts that automate running the pipeline on the dataset and collecting results. I will also improve logging so each test clearly reports success, failure, and failure reason.
After running the tests, I will analyze patterns in failure cases and identify areas where preprocessing, OMR integration, or XML cleaning need further refinement. This will allow targeted improvements rather than ad hoc debugging.
Schedule
The project remains on schedule. The system has transitioned from demo readiness into structured verification and validation, which aligns with the planned development timeline.
Deliverables for Next Week
Next week I will execute the planned verification tests on the benchmark dataset and analyze results against the defined success criteria. Based on these results, I will refine preprocessing and OMR handling to improve reliability. I will also continue validating system-level behavior by ensuring that the generated outputs integrate correctly with playback, visualization, and hardware components.
Interim Demo (Updated Gantt Chart + Presentation Slides)
Team Status Report for March 28
Project Risks and Mitigation
A key focus this week was reducing risks related to system stability during the upcoming demo. One major risk is inconsistent pipeline behavior across different inputs, such as skewed images, uneven lighting, or PDFs. This is being managed by refining preprocessing, selecting reliable demo inputs, and ensuring consistent outputs across repeated runs. Another risk is timing synchronization between playback, visualization, and hardware output. To address this, both playback and visualization continue to be driven from the same event stream, avoiding separate timing systems and reducing the chance of drift. Pipeline robustness is also a concern when handling edge cases like missing OMR outputs or malformed MusicXML. This was mitigated by improving error handling so the system clearly identifies which stage fails instead of failing silently. As hardware integration approaches, there is also a risk of mismatches between software-generated events and ESP32 expectations, which is being addressed by keeping the event format consistent and validating outputs before use.
On the hardware side, a key risk is unreliable input detection from the FSR strips, including missed key presses or false trigger signals. These issues may arise from electrical noise, signal interference, or variability in pressure sensing. This is being mitigated by improving signal stability through better wiring organization, consistent grounding, and isolating sensor lines where possible. In addition, software-level filtering and threshold calibration are being applied to distinguish intentional key presses from noise, ensuring more consistent and reliable input detection across all keys.
Design Changes
Design changes this week focused on improving stability and usability rather than adding new features. The preprocessing and scan pipeline were refined to handle a wider range of inputs more consistently, especially under varying lighting and image conditions. Error handling was improved across the pipeline to provide clearer feedback when failures occur. The playback and visualization components were also tested and slightly adjusted to ensure they remain synchronized and stable when running together. These changes improve overall system reliability without increasing complexity.
No changes made in the hardware components.
Schedule Update
The project remains on schedule and is ready for the upcoming demo. This week’s work focused on stabilization and integration, which aligns with the planned transition from development to demonstration and system validation.
Demonstrated Progress
The team successfully stabilized the full software pipeline for demo use. The system can now reliably process sheet music inputs, run the OMR pipeline, generate note events, and pass them into the playback and visualization system. Playback and visualization now operate together more consistently, allowing users to follow musical progress during playback. Improved error handling ensures that issues can be quickly identified and addressed during live demonstrations, resulting in a more reliable and demo-ready system.
Simon Lee’s Status Report for March 28
This week I focused on preparing the LumiKey system for the upcoming demo by improving the stability and reliability of the full software pipeline. Building on last week’s work validating the end-to-end flow, the goal this week was to make sure the system behaves consistently under demo conditions and handles errors more gracefully.
Pipeline Stability Improvements
A major focus this week was improving the consistency of the full pipeline: image/PDF input → preprocessing → OMR → MusicXML cleaning → conversion → note events. While the pipeline was already functional, I worked on making sure it produces stable outputs across repeated runs and different input types.
I refined preprocessing parameters to better handle difficult cases such as skewed images, uneven lighting, and PDFs. I also made adjustments to ensure that intermediate outputs passed between stages are consistently formatted, which helped reduce unexpected parsing or conversion issues. These changes improved the overall reliability of the pipeline when tested on the benchmark dataset.
Debugging and Error Handling
To better prepare for demo scenarios, I improved error handling throughout the pipeline. The system now clearly reports which stage fails (preprocessing, OMR, MusicXML cleaning, or parsing) instead of failing silently. This makes it easier to diagnose issues quickly during live demonstrations.
I also addressed several edge cases discovered during testing, including:
- OMR producing no output file
- Invalid or empty MusicXML files
- Inconsistent outputs from certain image inputs
By handling these cases explicitly, the system is now more robust and less likely to crash or produce confusing results during the demo.
End-to-End Testing for Demo Inputs
I continued testing the pipeline using the benchmark dataset and focused on selecting a subset of reliable demo inputs. These inputs were chosen based on consistent performance across preprocessing, OMR, and conversion stages.
I also verified that the outputs from the pipeline align with the expected structure of note events, ensuring they can be used directly by the playback system and hardware components.
Integration Readiness
In preparation for demo integration, I ensured that the event output format remains consistent and compatible with the existing playback and hardware systems. This helps ensure smooth communication between the software pipeline and the ESP32 during the demo.
I also worked with the team to verify that the full flow—from sheet music input to event generation—can be demonstrated clearly and reliably.
Schedule
The project remains on schedule and ready for the upcoming demo. The software pipeline is now stable enough to support live demonstrations, and remaining work is focused on final integration and polishing.
Deliverables for Next Week (Demo Execution and Final Refinements)
Next week I will focus on supporting the demo on Monday and Wednesday and addressing any issues that arise during live testing. After the demo, I plan to continue refining the pipeline based on observed edge cases, especially improving robustness for more diverse inputs.
I will also continue working with the team on deeper integration with the hardware system and identifying any improvements needed for timing, accuracy, or user experience as we move toward the final system.
Simon Lee’s Status Report for March 21
This week I focused on validating and stabilizing the full end-to-end LumiKey software pipeline. Building on last week’s work where the Scan Layer was integrated with the OMR engine and connected to the existing Conversion Layer, the goal this week was to ensure that the entire flow – from image or PDF input to note event generation – works reliably and is ready for the interim demo.
End-to-End Pipeline Validation
A major focus this week was testing the complete pipeline: image/PDF input → preprocessing → OMR → MusicXML cleaning → conversion → note events. Since the Conversion Layer was already implemented by another team member, my work focused on verifying that the Scan Layer output integrates correctly and consistently produces valid inputs for downstream processing.
To support this, I created a small benchmark dataset of sheet music inputs, including clean images, skewed photos, shadowed images, and PDFs. This allowed me to test how well the system performs across realistic conditions. I also began evaluating outputs based on whether the pipeline succeeds, whether parsing errors occur, and whether the generated events are reasonable in terms of structure and count.
Evaluation and Debugging Improvements
I added a simple evaluation workflow that runs the pipeline across the benchmark set and reports results such as success/failure and basic output consistency. This helped identify weak points in preprocessing and OMR where certain inputs fail or produce inconsistent results.
In addition, I improved debugging visibility by ensuring that failures are tied to specific stages of the pipeline. Instead of failing silently, the system now makes it clearer whether an issue occurs during preprocessing, OMR execution, MusicXML cleaning, or parsing. This makes it much easier to diagnose problems and iterate quickly.
Additional Testing and Edge Case Handling
I added several targeted tests to improve robustness of the Scan Layer and pipeline. These include handling invalid inputs such as corrupted images, malformed PDFs, missing OMR outputs, and invalid MusicXML files. These tests ensure that the system fails in a predictable and controlled way rather than crashing or producing unusable results.
The goal of this work is to make the pipeline stable enough for demo conditions, where unexpected inputs or edge cases may occur.
Preparation for Hardware Integration
With the software pipeline now functioning end-to-end, I began preparing for integration with the hardware system. This includes defining and documenting the event format generated by the pipeline, which will be used to communicate note timing and pitch information to the ESP32.
This step is important to ensure that the software output aligns with the expectations of the embedded system, particularly for timing accuracy and LED control. It also sets up the next phase of development, where software-generated note events will drive real hardware behavior.
Schedule
The project remains on schedule and aligned with the interim demo timeline. According to the planned system pipeline and milestones , the current focus on integration and validation is consistent with reaching an MVP before the demo. The full software pipeline is now operational, and remaining work is focused on stability and hardware integration.
Deliverables for Next Week (Interim Demo Preparation)
Next week I will focus on final preparation for the interim demo. This includes refining the pipeline to ensure consistent behavior on demo inputs, improving error handling for a smoother user experience, and working with the team to connect the software output to the ESP32 hardware.
The goal is to demonstrate a complete working system where a user uploads sheet music and receives corresponding LED guidance on the keyboard. I will also continue refining preprocessing and validation steps based on any issues observed during integration testing.