This week I focused on transitioning from demo preparation into the verification and validation phase of the LumiKey project. Building on the previous work where the scan pipeline was stabilized for the demo, my goal this week was to define concrete testing procedures and evaluation metrics to measure whether the Scan Layer and full pipeline meet the design and use-case requirements.
Verification Planning for Scan Layer
For my subsystem, verification will focus on evaluating the performance and reliability of the Scan Layer, including preprocessing, OMR execution, and MusicXML cleaning. I plan to use a benchmark dataset of 15 to 25 sheet music inputs that include a mix of clean images, skewed photos, shadowed images, and PDFs to reflect realistic usage conditions.
The specific tests I plan to run include:
- Running the full pipeline on each input and recording whether valid MusicXML is successfully produced.
- Logging the exact stage where failures occur, such as preprocessing, OMR execution, or XML cleaning, to identify bottlenecks.
- Measuring output consistency by running the same input multiple times and verifying that the resulting MusicXML and note events remain stable.
- Comparing detected note counts and structural properties against expected values for selected test files.
- Tracking error types such as missing OMR output, malformed XML, or parsing failures.
For analysis, I will compute the overall pipeline success rate across the dataset, with a target of at least 90 percent successful conversions. I will also analyze failure distributions by stage to determine whether issues are concentrated in preprocessing, OMR, or post-processing. Consistency checks will help ensure the pipeline behaves deterministically rather than producing variable outputs.
Validation Planning for Full System
At the system level, validation will focus on whether the full pipeline produces correct and usable outputs for the intended application of guided piano learning. This involves confirming that the generated note events accurately reflect the musical structure and can be used by the playback system.
The planned validation steps include:
- Selecting 3 to 5 representative pieces with known correct outputs.
- Comparing generated note events against trusted references such as manually verified MusicXML or expected MIDI sequences.
- Evaluating correctness by checking pitch accuracy, note ordering, and basic timing relationships.
- Verifying that the generated event stream integrates correctly with the playback and visualization system without errors or misalignment.
- Observing whether the output is usable in practice, meaning that playback behavior matches the intended musical structure.
The main validation criteria will be whether the system produces correct and usable note events for known inputs and whether those outputs integrate cleanly into the rest of the system.
Next Steps for Testing
These tests have not yet been executed, so the next step is to implement evaluation scripts that automate running the pipeline on the dataset and collecting results. I will also improve logging so each test clearly reports success, failure, and failure reason.
After running the tests, I will analyze patterns in failure cases and identify areas where preprocessing, OMR integration, or XML cleaning need further refinement. This will allow targeted improvements rather than ad hoc debugging.
Schedule
The project remains on schedule. The system has transitioned from demo readiness into structured verification and validation, which aligns with the planned development timeline.
Deliverables for Next Week
Next week I will execute the planned verification tests on the benchmark dataset and analyze results against the defined success criteria. Based on these results, I will refine preprocessing and OMR handling to improve reliability. I will also continue validating system-level behavior by ensuring that the generated outputs integrate correctly with playback, visualization, and hardware components.