Team D7 LumiKeys

This week I focused on integrating the Scan Layer preprocessing pipeline with the Optical Music Recognition (OMR) stage and building tools to validate that the preprocessing output produces reliable symbolic music data. My goal was to move the system from an isolated preprocessing pipeline to a functional end-to-end scan workflow that converts sheet music images into structured MusicXML data.

Integrated Audiveris OMR Execution

The main milestone this week was connecting the preprocessing pipeline directly to the Audiveris OMR engine. Previously the system stopped after producing normalized images, but it can now automatically pass the cleaned image output into Audiveris and generate MusicXML files. I implemented this integration using Python subprocess calls so the pipeline can invoke Audiveris programmatically and capture the resulting MusicXML output.

Once this was working, I tested the system on multiple sheet music samples to verify that the preprocessing improvements from last week were producing images that Audiveris could reliably interpret. I also validated the generated MusicXML by importing the results into MuseScore, which made it easier to visually compare the recognized notes against the original sheet music. This proof-of-concept confirmed that the preprocessing stage is producing OMR-compatible images and that the overall scan pipeline is technically feasible.

Implemented MusicXML Validation and Cleaning

After OMR execution, I began implementing a validation and cleanup stage for the generated MusicXML. OMR output can occasionally include inconsistent metadata, missing attributes, or formatting irregularities that can break downstream parsing. To address this, I used the music21 library to parse the generated MusicXML files and enforce a consistent structure.

This stage verifies that measures are correctly defined, removes unnecessary metadata, and ensures that note and duration information can be safely extracted by the Conversion Layer. The goal is to guarantee that the output from the Scan Layer is not only readable but also structurally reliable for the rest of the LumiKey pipeline.

Built Visualization Tools for Scan Layer Processing

To better understand how preprocessing affects OMR performance, I developed visualization scripts that highlight the structural features extracted from sheet music images. These tools overlay detected staff lines, barlines, and candidate noteheads on top of the processed image, making it easier to confirm that the system is correctly identifying musical structure before OMR is executed.

These visualizations also helped refine several preprocessing parameters, such as thresholding behavior and morphological filters used to isolate staff lines. In addition to helping with debugging, the visualization outputs will also be useful for illustrating the Scan Layer architecture in our design report.

Status Report 4 Requirements: Part B

Cultural factors were considered in the design of LumiKey because music education is strongly shaped by cultural traditions, learning practices, and accessibility within different communities. Many cultures place significant value on learning traditional instruments such as the piano, yet access to formal instruction and specialized digital instruments can vary widely depending on economic and regional circumstances. By designing LumiKey as a retrofit system that works with existing acoustic or digital pianos, the project respects the cultural importance of traditional instruments while lowering barriers to entry for learners. The system also avoids requiring proprietary hardware or software ecosystems, allowing it to be adapted for use in diverse educational settings, including home practice, community music programs, and classrooms across different cultural contexts.

Schedule

I am currently on schedule. The Scan Layer is now capable of running the full pipeline from image input through preprocessing and OMR execution. The remaining work in this stage primarily involves refining MusicXML cleanup and validating the pipeline on a larger dataset of sheet music images.

Deliverables for Next Week

Next week I plan to finalize the MusicXML validation logic and begin implementing the Conversion Layer that transforms symbolic music data into the time-indexed event representation used by the embedded system. I will also run the full scan pipeline on a larger dataset of sheet music images to evaluate pitch extraction accuracy and ensure the system can consistently meet our ≥90% correctness target. Additionally, I will work with the rest of the team to begin connecting the software pipeline to the ESP32 communication layer so recognized notes can be transmitted to the hardware for LED playback.

This week I focused entirely on designing and implementing the Scan Layer preprocessing pipeline for LumiKey. My goal was to build a robust system that converts raw phone photos of sheet music into normalized, OMR-ready images suitable for Audiveris. I built, refined, tested, and structured a modular preprocessing system that is tuned for phone photo conditions as well as PDF uploads.

Rebuilt and Stabilized the Full Normalization Pipeline

Instead of keeping preprocessing as a collection of loosely connected image operations, I reorganized it into a clearer, more structured pipeline. I separated configuration, image processing steps, quality checks, and utility functions into different files so the system is easier to understand and modify. The main preprocessing function now returns a structured result that includes whether the scan passed, relevant metrics, and specific failure reasons, rather than just showing intermediate debug images. This makes the behavior much more transparent and easier to improve.

I also improved the page detection step by tightening how contours are filtered and adding fallback logic when the page can’t be confidently detected. After perspective correction, I normalize the image resolution and ensure the page is upright before continuing. These changes make the preprocessing output more consistent and reliable across different phone photos.

Replaced Basic Contrast Enhancement with Illumination

Last week I was using CLAHE to improve contrast, but this week I replaced that with a better lighting correction method that works more reliably for phone photos. Instead of just boosting contrast, I now estimate the background lighting and normalize the image to reduce shadows and uneven brightness. This made a noticeable difference when testing on real photos taken under indoor lighting.

I also improved the black-and-white conversion step. I’m now using adaptive thresholding, and if the result looks clearly wrong (for example, too much black or too much white), the system automatically falls back to a different thresholding method. This helps prevent cases where the sheet music becomes unreadable due to poor lighting.

Finally, I refined the deskew step so that the image is only rotated when the detected angle is reasonable. Previously, large or incorrect skew estimates could make the image worse. Now, the system avoids over-correcting and instead flags the image if it’s too distorted.

Implemented Formal Quality Gate and Testing Framework

The biggest improvement this week was adding a proper quality check stage. Before, I was mainly checking whether the image was blurry, whereas now the system checks multiple things: whether the image is too blurry, whether there’s too much glare, whether the black-and-white conversion looks reasonable, and whether the page is still slightly rotated after correction. If something looks wrong, the preprocessing step doesn’t just fail silently, but rather provides reasons for what went wrong. This makes it much easier to understand and improve weak spots, and it also lets us track whether we’re meeting the ≥90% usable scan goal.

I also added automated tests using pytest to make sure key parts of the pipeline behave consistently, such as blur detection and skew correction. On top of that, I built a simple script that runs preprocessing over an entire folder of phone photos, saves the cleaned outputs, and logs which ones failed and why. Instead of manually eyeballing results, I can now evaluate performance in a more systematic and measurable way.

Schedule

I am on schedule. The preprocess stage has been evolving into a structured, modular, and testable system as thought out by the team from a partially implemented prototype. Remaining work primarily involves larger-scale testing and integrating with the OMR stage.

Deliverables for Next Week

Next week, I plan to connect the preprocessing pipeline directly to Audiveris so the entire scan process can run automatically from image input to MusicXML output. I’ll also begin implementing the MusicXML validation and cleaning stage to make sure the OMR output is structurally sound before it moves into the conversion layer. In addition, I want to run preprocessing on a larger batch of real phone photos to measure our actual usable scan rate and fine-tune parameters as needed to confidently meet the ≥90% target.

This week was mainly about stepping back and rethinking the scan subsystem so it’s realistic and achievable within our timeline. After looking more closely at what it would take to build a fully custom OMR pipeline, I decided to revise the architecture to something more modular and practical. Instead of implementing staff detection and symbol recognition from scratch, I restructured the scan layer into three parts: preprocessing, third-party OMR execution, and postprocessing. This change reduces risk and gives us more flexibility to focus on higher-impact features later.

Planning & Architecture Refinement

A large part of the week went into refining the scan pipeline design and updating our system diagrams to reflect that change. I clarified how image normalization feeds into the OMR engine, and how the raw MusicXML output will be validated and cleaned before being used downstream.

This week I clarified the following pipeline stages:

• Image normalization and geometric correction
• Quality-based input filtering
• OMR engine execution
• MusicXML parsing and structural validation

During this process, I also worked on completing the design presentation with my teammates in refining our user requirements, design requirements, solution approach, and our overall schedule and approach.

Initial Normalization Implementation

I began implementing the preprocessing stage using OpenCV in Python. So far, I’ve put together the basic structure of the normalization pipeline, including:

• Detecting the page contour and applying perspective warping
• Estimating and correcting skew using Hough line detection
• Applying contrast enhancement (CLAHE)
• Adding preliminary blur detection using Laplacian variance

The pipeline is partially implemented and producing intermediate debug outputs so I can see how each stage behaves. I’m still tuning parameters and testing on different sheet music images to make sure the behavior is consistent.

Schedule

I’m currently on schedule. The architecture change actually reduces complexity and makes the scan layer more manageable moving forward. There are no major blockers right now.

Deliverables for Next Week

Next week I plan to connect the normalization output to the OMR execution step and start integrating basic MusicXML parsing and validation. I’ll also continue refining the preprocessing parameters using additional test images.

Part A

On the topic of public health, safety, and overall welfare, the system we are building is designed to be safe, low-risk and supportive for users. Specifically on the safety side, the system operates using low-voltage microcontroller hardware and LED components which minimizes electrical hazard. The bluetooth communication between the hardware and software also follows standard protocols and is not exposed to high-power components. From a health and well-being perspective, the system is designed to create a positive learning experience where feedback is structured and is not overwhelming for the user. The guided playing modes are intended to give users more freedom to choose how to practice rather than cause unnecessary stress. There is also not personal data being transmitted externally and therefore, overall the system is built to be safe, accessible, and supportive for beginners at different skill levels