Team Status Report for March 14

Project Risks and Mitigation

One of the biggest risks to the project is the reliability of the sheet music scanning process. Since LumiKey depends on converting sheet music images into musical data, poor image quality or recognition errors could lead to incorrect notes being detected. To manage this risk, the scan pipeline now includes stronger preprocessing steps to improve image quality before running the Optical Music Recognition (OMR) engine. The system also includes validation checks to detect problems with the generated MusicXML files before the data moves to the next stage of the pipeline. If recognition accuracy becomes an issue, the team can restrict inputs to clearer scans or PDFs while continuing to improve the preprocessing system.

Another risk involves timing synchronization between the software playback system and the hardware. If the system does not process note events in the correct order or if communication delays occur when sending data to the ESP32, the LED guidance may not match the intended timing of the music. To address this, the playback algorithm was improved to handle event sequencing and timing more consistently. Further testing will be done once the software is fully connected to the hardware.

A third risk is handling different types of input files, such as corrupted images or invalid PDFs. To reduce this risk, additional automated tests were added to check the scan pipeline and ensure that errors are detected early rather than causing failures later in the system.

The fourth risk is ghost detection from the FSR keys. To mitigate this risk, we will separate the FSRs into a cleaner wiring scheme to reduce electrical noise and interference. This will help ensure that the ESP32 only detects legitimate key presses and avoids false signals caused by ghost inputs.

Design Changes

Some improvements were made to the system design to make the software pipeline more flexible and easier to use. One important change was adding support for both image and PDF sheet music inputs. The system can now convert PDF pages into images so they can go through the same preprocessing and recognition pipeline.

The scan pipeline was also reorganized into smaller modules to make the code easier to maintain and test. In addition, the OMR integration was improved so the system can automatically detect the Audiveris engine and handle different output formats more reliably.

On the playback side, updates were made to the event processing system so musical events are handled in a clearer and more predictable order during playback. This helps improve the timing behavior of the system and prepares it for integration with the visualizer and hardware components.

Schedule Update

The project remains on schedule. The scanning pipeline, OMR integration, and playback system are now functioning parts of the software. The team is moving from building individual components toward integrating the full system.

Demonstrated Progress

This week the team made significant progress on the software system. The scan pipeline can now process sheet music images or PDFs, run the OMR engine, and produce MusicXML output that can be used by the rest of the system.

Improvements were also made to the playback system to ensure musical events are processed more consistently. In addition, development began on a sheet music visualizer that will allow the system to display musical progress during playback. This visual component will help users connect the sheet music with the LED guidance on the keyboard.

Overall, the project is continuing to move from early development toward a fully integrated system that connects sheet music input, musical event processing, and guided playback.

The Gantt chart (Gantt) outlines the team’s current timeline.

Simon Lee’s Status Report for March 14

This week I focused on expanding and stabilizing the Scan Layer so it works as a reliable part of the full LumiKey software pipeline. Building on last week’s work integrating the preprocessing system with the Audiveris OMR engine, the goal this week was to improve preprocessing, strengthen OMR execution, and make the system easier for teammates to run and test.

Expanded Scan Preprocessing Pipeline

The preprocessing module was expanded into a full normalization pipeline capable of handling both images and PDFs. When a user uploads a PDF, the pages are first rasterized using the pypdfium2 library so they can go through the same preprocessing steps as images.

The preprocessing pipeline now includes illumination normalization to reduce lighting artifacts, perspective correction for photos taken at an angle, skew estimation with deskew correction, adaptive thresholding tuned using staff spacing, and morphological filtering to remove small artifacts. These steps help produce cleaner images that are easier for the OMR engine to interpret. I also reorganized the code into smaller helper functions so that image processing steps are easier to maintain and test.

Improved OMR Integration

The OMR module was updated to run the Audiveris engine directly through the command line using Python’s subprocess interface. The system now automatically detects the Audiveris binary using environment variables, system paths, or common installation locations on macOS.

Additional handling was added for locating the MusicXML output generated by Audiveris, including support for both .musicxml and compressed .mxl files. When compressed output is produced, the system automatically extracts the MusicXML so it can continue through the pipeline.

MusicXML Validation Improvements

The MusicXML cleaning stage was improved to make the pipeline more reliable. The cleaner now performs basic validation checks such as newline normalization, empty-file checks, and XML parsing verification. If the MusicXML file is malformed or empty, the system produces clear error messages instead of failing silently. This helps ensure the Scan Layer always outputs valid data before it reaches the next stage of the pipeline.

Testing and Pipeline Validation

A major focus this week was improving testing coverage for the scan pipeline. I added several test files to validate preprocessing, OMR integration, MusicXML cleaning, and the full scan pipeline. These tests check that the system behaves correctly when processing different types of inputs and when errors occur.

The current test suite contains 27 passing tests. I also added a real sheet music example (twinkletwinkle.pdf) as a test asset so the full pipeline can be validated using realistic input.

Project Setup Improvements

I also made improvements to the project setup process so other team members can run the system more easily. I added a bootstrap script that automatically creates a Python virtual environment, installs dependencies, and detects or installs Audiveris on macOS. The UI launch script now calls this setup step automatically before starting the application. I also updated the README with clearer setup instructions so teammates can run the project with a simple command.

Schedule

The project remains on schedule. The Scan Layer can now process both image and PDF inputs, run the Audiveris OMR engine, and produce validated MusicXML outputs. The system is also supported by automated tests, which will help maintain stability as new features are added.

Deliverables for Next Week

Next week I will focus on validating the full pipeline now that the Scan Layer is integrated with the existing Conversion Layer. I plan to build a small benchmark set of about 15–25 sheet music files, including clean photos, skewed photos, shadowed images, and PDFs. This dataset will be used to evaluate how reliably the end-to-end pipeline works.

I will also add an evaluation script that runs the pipeline on the benchmark set and reports results such as success or failure, parse errors, and basic note matching where expected outputs are available. In addition, I plan to improve error reporting so the application clearly identifies which stage of the pipeline fails if a scan does not succeed.

Finally, I will begin preparing for hardware integration by documenting the event format used to send note data to the ESP32 and adding a test to ensure the event serialization remains consistent for hardware communication.

Chris Oh’s Status Report for March 14

This week I focused on improving the playback system and starting development of a basic sheet music visualizer. The main work involved refining the playback algorithm to better handle musical event timing and building an initial framework that will allow the system to visually represent sheet music during playback.

Playback System Updates
• Refined the playback algorithm to improve event sequencing and timing consistency.
• Reorganized the event processing logic to ensure that note events are triggered in a predictable order during playback.
• Improved handling of note-on and note-off events to make playback behavior more stable and easier to extend.
• Adjusted the playback pipeline so that event timing is managed more cleanly between the software processing layer and the output layer.
• These refinements help ensure that playback aligns more closely with the intended musical structure.

Sheet Music Visualizer Framework
• Began implementing a basic framework for a sheet music visualization component.
• Established an initial structure for representing musical events visually during playback.
• Implemented early components that track playback position and note timing.
• Designed the framework so it can later support highlighting notes or measures while the music is playing.
• The goal of this component is to help users visually connect the sheet music with the LED guidance on the keyboard.

Schedule

I am currently on schedule. The playback algorithm has been refined and the initial visualizer framework has been established. The project continues to move from early architecture work into functional implementation of core system components.

Deliverables for Next Week
• Integrate the playback algorithm with the sheet music visualizer framework.
• Improve timing synchronization between playback events and visual display updates.
• Expand the visualizer to support note highlighting and playback progress tracking.
• Begin testing the system with multiple pieces of sheet music to verify playback behavior.

Team Status Report for Feb 28

  • Status Report 4 Requirements: Part A was written by Chris, B was written by Simon, and C was written by Gene.

Project Risks and Mitigation

One of the biggest risks for the LumiKey project is inconsistent Optical Music Recognition (OMR) results when processing phone photos of sheet music. Phone images can have shadows, glare, skew, or uneven lighting, which can make it difficult for the OMR engine to correctly detect notes. To manage this risk, the Scan Layer preprocessing pipeline performs several normalization steps such as lighting correction, adaptive thresholding, and skew detection. The system also includes a quality check stage that detects problems like blur or glare before running OMR. If the scan quality is too poor, the system can flag the input rather than producing incorrect results. As a contingency plan, if phone photos remain unreliable, the system can restrict input to higher-quality images or PDF sheet music.

Another risk is latency between the software and the ESP32 hardware. If the system takes too long to respond to a key press, the LED feedback may feel delayed or out of sync with the music. To reduce this risk, the latency budget was divided across sensing, processing, Bluetooth transmission, and LED actuation. The system also uses buffered communication so that the ESP32 always has upcoming notes ready to display. If communication delays become an issue, a fallback solution is to preload larger sections of the music sequence onto the microcontroller.

A third risk involves reliable key press detection using force-sensitive resistors (FSRs). Because FSR sensors measure pressure, they can sometimes produce noisy signals depending on how the user presses the key. To manage this, the firmware uses threshold filtering and debounce logic to reduce false triggers. If sensor readings remain inconsistent, thresholds can be adjusted or the sensor placement can be refined without redesigning the hardware.

Simon Lee’s Status Report for Feb 28

This week I focused on integrating the Scan Layer preprocessing pipeline with the Optical Music Recognition (OMR) stage and building tools to validate that the preprocessing output produces reliable symbolic music data. My goal was to move the system from an isolated preprocessing pipeline to a functional end-to-end scan workflow that converts sheet music images into structured MusicXML data.

Integrated Audiveris OMR Execution

The main milestone this week was connecting the preprocessing pipeline directly to the Audiveris OMR engine. Previously the system stopped after producing normalized images, but it can now automatically pass the cleaned image output into Audiveris and generate MusicXML files. I implemented this integration using Python subprocess calls so the pipeline can invoke Audiveris programmatically and capture the resulting MusicXML output.

Once this was working, I tested the system on multiple sheet music samples to verify that the preprocessing improvements from last week were producing images that Audiveris could reliably interpret. I also validated the generated MusicXML by importing the results into MuseScore, which made it easier to visually compare the recognized notes against the original sheet music. This proof-of-concept confirmed that the preprocessing stage is producing OMR-compatible images and that the overall scan pipeline is technically feasible.

Implemented MusicXML Validation and Cleaning

After OMR execution, I began implementing a validation and cleanup stage for the generated MusicXML. OMR output can occasionally include inconsistent metadata, missing attributes, or formatting irregularities that can break downstream parsing. To address this, I used the music21 library to parse the generated MusicXML files and enforce a consistent structure.

This stage verifies that measures are correctly defined, removes unnecessary metadata, and ensures that note and duration information can be safely extracted by the Conversion Layer. The goal is to guarantee that the output from the Scan Layer is not only readable but also structurally reliable for the rest of the LumiKey pipeline.

Built Visualization Tools for Scan Layer Processing

To better understand how preprocessing affects OMR performance, I developed visualization scripts that highlight the structural features extracted from sheet music images. These tools overlay detected staff lines, barlines, and candidate noteheads on top of the processed image, making it easier to confirm that the system is correctly identifying musical structure before OMR is executed.

These visualizations also helped refine several preprocessing parameters, such as thresholding behavior and morphological filters used to isolate staff lines. In addition to helping with debugging, the visualization outputs will also be useful for illustrating the Scan Layer architecture in our design report.

Status Report 4 Requirements: Part B

Cultural factors were considered in the design of LumiKey because music education is strongly shaped by cultural traditions, learning practices, and accessibility within different communities. Many cultures place significant value on learning traditional instruments such as the piano, yet access to formal instruction and specialized digital instruments can vary widely depending on economic and regional circumstances. By designing LumiKey as a retrofit system that works with existing acoustic or digital pianos, the project respects the cultural importance of traditional instruments while lowering barriers to entry for learners. The system also avoids requiring proprietary hardware or software ecosystems, allowing it to be adapted for use in diverse educational settings, including home practice, community music programs, and classrooms across different cultural contexts.

Schedule

I am currently on schedule. The Scan Layer is now capable of running the full pipeline from image input through preprocessing and OMR execution. The remaining work in this stage primarily involves refining MusicXML cleanup and validating the pipeline on a larger dataset of sheet music images.

Deliverables for Next Week

Next week I plan to finalize the MusicXML validation logic and begin implementing the Conversion Layer that transforms symbolic music data into the time-indexed event representation used by the embedded system. I will also run the full scan pipeline on a larger dataset of sheet music images to evaluate pitch extraction accuracy and ensure the system can consistently meet our ≥90% correctness target. Additionally, I will work with the rest of the team to begin connecting the software pipeline to the ESP32 communication layer so recognized notes can be transmitted to the hardware for LED playback.

Team Status Report for Feb 21

The most significant risks at this stage come from the new features we recently added to our implementation plan: Beat Mode, the Lightweight UI, and Bluetooth transmission. These features are not replacements for existing components but additional functionality layered on top of our original scope. Because of this, they increase the overall system complexity and compress the development timeline. Integrating these additions while continuing progress on the core system introduces schedule pressure and a higher risk of integration challenges.

To manage these risks, we are being more deliberate about incremental integration and testing. Rather than building everything separately and merging at the end, we are validating functionality at each stage to ensure components remain stable as new features are introduced. This reduces the likelihood of large integration failures late in development.

Our primary contingency plan is to fall back to our previously defined MVP configuration if necessary. This ensures that even if advanced features require scaling back, the core system (sheet music processing, note detection, and feedback) remains functional and demonstrable. However, we believe that by increasing our development throughput and maintaining disciplined testing practices, we can integrate these additional features successfully within the current timeline.

The most significant hardware risks at this stage stem from signal integrity issues and potential component failures as we scale the system. Because our design relies on multiple force-sensitive resistors (FSRs) connected to a centralized ESP32, long wiring runs may introduce electrical noise and increased latency in the analog signals. Excessive noise could make it difficult to reliably distinguish between pressed and un pressed states, which would directly impact key detection accuracy and overall system responsiveness. As we expand from small-scale testing to a full 35-key implementation, these risks become more pronounced.

To manage these risks, we are approaching hardware integration incrementally. We will first prototype the sensing system on a breadboard and gradually increase wiring length to evaluate noise levels and latency under realistic conditions. Since the project remains within budget, we plan to order additional components to account for potential hardware failures during testing. If signal noise or latency becomes noticeable during live play, we will transition to a custom PCB design to shorten signal paths and improve electrical stability. Additionally, if FSR reliability becomes a concern, we are prepared to shift to a Hall-effect sensor and magnet-based detection approach, which would provide cleaner digital signals instead of analog resistance values.

Our primary hardware contingency plan is to preserve the LED feedback system even if key detection encounters setbacks. If analog sensing proves unreliable, we can integrate Bluetooth-based communication or pivot to MIDI input from a digital keyboard to ensure accurate note detection. This guarantees that the core functionality of visual feedback and guided learning remains demonstrable, even if the original sensing method requires modification. By planning alternative detection pathways in advance, we reduce the risk of complete system failure while maintaining a functional MVP.

There are no specific updates to the schedule as the software and hardware integration periods will include these new design changes. Work is progressing as planned, with no new visual milestones to report at this time. The Gantt chart (Gantt) outlines the team’s current timeline.

Simon Lee’s Status Report for Feb 21

This week I focused entirely on designing and implementing the Scan Layer preprocessing pipeline for LumiKey. My goal was to build a robust system that converts raw phone photos of sheet music into normalized, OMR-ready images suitable for Audiveris. I built, refined, tested, and structured a modular preprocessing system that is tuned for phone photo conditions as well as PDF uploads.

Rebuilt and Stabilized the Full Normalization Pipeline

Instead of keeping preprocessing as a collection of loosely connected image operations, I reorganized it into a clearer, more structured pipeline. I separated configuration, image processing steps, quality checks, and utility functions into different files so the system is easier to understand and modify. The main preprocessing function now returns a structured result that includes whether the scan passed, relevant metrics, and specific failure reasons, rather than just showing intermediate debug images. This makes the behavior much more transparent and easier to improve.

I also improved the page detection step by tightening how contours are filtered and adding fallback logic when the page can’t be confidently detected. After perspective correction, I normalize the image resolution and ensure the page is upright before continuing. These changes make the preprocessing output more consistent and reliable across different phone photos.

Replaced Basic Contrast Enhancement with Illumination

Last week I was using CLAHE to improve contrast, but this week I replaced that with a better lighting correction method that works more reliably for phone photos. Instead of just boosting contrast, I now estimate the background lighting and normalize the image to reduce shadows and uneven brightness. This made a noticeable difference when testing on real photos taken under indoor lighting.

I also improved the black-and-white conversion step. I’m now using adaptive thresholding, and if the result looks clearly wrong (for example, too much black or too much white), the system automatically falls back to a different thresholding method. This helps prevent cases where the sheet music becomes unreadable due to poor lighting.

Finally, I refined the deskew step so that the image is only rotated when the detected angle is reasonable. Previously, large or incorrect skew estimates could make the image worse. Now, the system avoids over-correcting and instead flags the image if it’s too distorted.

Implemented Formal Quality Gate and Testing Framework

The biggest improvement this week was adding a proper quality check stage. Before, I was mainly checking whether the image was blurry, whereas now the system checks multiple things: whether the image is too blurry, whether there’s too much glare, whether the black-and-white conversion looks reasonable, and whether the page is still slightly rotated after correction. If something looks wrong, the preprocessing step doesn’t just fail silently, but rather provides reasons for what went wrong. This makes it much easier to understand and improve weak spots, and it also lets us track whether we’re meeting the ≥90% usable scan goal.

I also added automated tests using pytest to make sure key parts of the pipeline behave consistently, such as blur detection and skew correction. On top of that, I built a simple script that runs preprocessing over an entire folder of phone photos, saves the cleaned outputs, and logs which ones failed and why. Instead of manually eyeballing results, I can now evaluate performance in a more systematic and measurable way.

Schedule

I am on schedule. The preprocess stage has been evolving into a structured, modular, and testable system as thought out by the team from a partially implemented prototype. Remaining work primarily involves larger-scale testing and integrating with the OMR stage.

Deliverables for Next Week

Next week, I plan to connect the preprocessing pipeline directly to Audiveris so the entire scan process can run automatically from image input to MusicXML output. I’ll also begin implementing the MusicXML validation and cleaning stage to make sure the OMR output is structurally sound before it moves into the conversion layer. In addition, I want to run preprocessing on a larger batch of real phone photos to measure our actual usable scan rate and fine-tune parameters as needed to confidently meet the ≥90% target.

Simon Lee’s Status Report for Feb 14

This week was mainly about stepping back and rethinking the scan subsystem so it’s realistic and achievable within our timeline. After looking more closely at what it would take to build a fully custom OMR pipeline, I decided to revise the architecture to something more modular and practical. Instead of implementing staff detection and symbol recognition from scratch, I restructured the scan layer into three parts: preprocessing, third-party OMR execution, and postprocessing. This change reduces risk and gives us more flexibility to focus on higher-impact features later.

Planning & Architecture Refinement

A large part of the week went into refining the scan pipeline design and updating our system diagrams to reflect that change. I clarified how image normalization feeds into the OMR engine, and how the raw MusicXML output will be validated and cleaned before being used downstream.

This week I clarified the following pipeline stages:

  • Image normalization and geometric correction
  • Quality-based input filtering
  • OMR engine execution
  • MusicXML parsing and structural validation

During this process, I also worked on completing the design presentation with my teammates in refining our user requirements, design requirements, solution approach, and our overall schedule and approach.

Initial Normalization Implementation

I began implementing the preprocessing stage using OpenCV in Python. So far, I’ve put together the basic structure of the normalization pipeline, including:

  • Detecting the page contour and applying perspective warping
  • Estimating and correcting skew using Hough line detection
  • Applying contrast enhancement (CLAHE)
  • Adding preliminary blur detection using Laplacian variance

The pipeline is partially implemented and producing intermediate debug outputs so I can see how each stage behaves. I’m still tuning parameters and testing on different sheet music images to make sure the behavior is consistent.

Schedule

I’m currently on schedule. The architecture change actually reduces complexity and makes the scan layer more manageable moving forward. There are no major blockers right now.

Deliverables for Next Week

Next week I plan to connect the normalization output to the OMR execution step and start integrating basic MusicXML parsing and validation. I’ll also continue refining the preprocessing parameters using additional test images.

Part A

On the topic of public health, safety, and overall welfare, the system we are building is designed to be safe, low-risk and supportive for users. Specifically on the safety side, the system operates using low-voltage microcontroller hardware and LED components which minimizes electrical hazard. The bluetooth communication between the hardware and software also follows standard protocols and is not exposed to high-power components. From a health and well-being perspective, the system is designed to create a positive learning experience where feedback is structured and is not overwhelming for the user. The guided playing modes are intended to give users more freedom to choose how to practice rather than cause unnecessary stress. There is also not personal data being transmitted externally and therefore, overall the system is built to be safe, accessible, and supportive for beginners at different skill levels

Gene Chang’s Status Report for Feb 7

This week focused on acquiring, testing, and integrating the core hardware components for the LumiKeys system, while continuing to define the technical direction of the hardware architecture. Specifically, I focused on testing NeoPixel LEDs, force-sensitive resistors (FSRs), and their integration with the ESP32 microcontroller, as well as researching UDP-based communication between the server and ESP32 for user-input verification.

Hardware Design and Component Work

  • Researched and evaluated hardware components required for key-press detection and visual feedback, including NeoPixel LED strips and force-sensitive resistors.
  • Developed simple test programs to read analog values from FSR strips and characterize resistance changes under applied force.
  • Defined threshold values to distinguish between pressed and unpressed states for reliable key detection.
  • Investigated communication methods between the ESP32 and the server, focusing on UDP as a lightweight protocol for transmitting note and verification data.
  • Planned the next stage of LED control using the Adafruit NeoPixel library and preparation for soldering and breadboard integration.

Initial Technical Prototyping

  • Prototyped force-sensing resistor input using the ESP32 ADC to measure resistance changes and convert them into usable analog values.
  • Validated that force input can be reliably detected and distinguished through thresholding.
  • Evaluated power requirements and identified the need for a stable power supply and voltage regulation for driving LEDs alongside the ESP32.

Schedule

I am currently on schedule, as core hardware components have been obtained, basic sensing functionality has been validated, and integration planning is underway.

Deliverables for Next Week

  • Integrate the NeoPixel LED strip with the ESP32 and implement basic LED control.
  • Set up a stable power system, including an appropriate supply and a 3.3 V step-down converter where needed.
  • Begin combining FSR input detection with LED feedback to demonstrate a basic end-to-end hardware loop.

Team Status Report for Feb 7

One of the most significant risks to the project is the complexity of integrating multiple software components together with the hardware layer. Because the system depends on several stages working seamlessly, any mismatch in interfaces or unexpected behavior during integration could delay progress. To manage this risk, we are intentionally designing the system to be highly modular, with clearly defined boundaries and data formats between components. Each module will be developed and validated independently through thorough test cases before being connected to the rest of the system. This approach allows us to verify correctness at the component level first, making integration more predictable and easier to debug.

Specifically a major risk for the project is the reliable nature of interpreting the sheet music under real-world conditions, in which given the factors of scan quality, lighting, and perspectives of the sheet music can affect accuracy of extracting necessary information and unfortunately if there are errors at this stage it would affect the final result of the LED. To manage this risk, the team will plan to design a software pipeline that will provide preprocessing, validation, and modular testing at each stage allowing inaccuracies to be noticeable and isolated before putting together other components.

An additional risk involves the responsiveness requirements that we set for the full system starting from the user input to LED output. The latency that will come from software processing, verification logic, and the software-hardware communication is something that is necessary to mitigate. To avoid these risks, we will measure the end-to-end latency across components, optimize paths, and avoid blocking operations between subsystems. If accuracy or latency measures are not met within the timeline, some backup plans include supporting alternative digital input formats or simplifying processing to maintain a functional user experience.

There is also a risk of hardware-related issues, such as FSR strip malfunctions or defects in the LED components, which are distinct from software bugs. To mitigate these risks, we plan to keep spare parts on hand and maintain fallback options. If FSR strips prove unreliable, we will explore alternative input methods such as IMU-based detection or MIDI input.

A minor change was made to the hardware sensing design during this stage of development. The original design assumed force-sensitive resistors (FSRs) would be used as the primary method for detecting key presses. After initial testing and evaluation, we decided to keep alternative sensing methods, such as IMU-based detection or MIDI input, as fallback options in case FSR reliability, durability, or sensitivity becomes a limitation in real use. This change was necessary to reduce technical risk early in development and ensure that the system can still meet functional requirements if the primary sensing method proves unreliable. The main costs associated with this change include purchasing additional components for testing and prototyping, as well as the extra time required to evaluate multiple sensing approaches. These costs will be mitigated by performing early validation to determine the most reliable sensing method and standardizing on a single solution once testing is complete.

There have been no updates to the schedule. Work is progressing as planned, with no new visual milestones to report at this time. The Gantt chart (Gantt) outlines the team’s current timeline.