Posted on

Team Status Report for March 14

Project Risks and Mitigation

One of the biggest risks to the project is the reliability of the sheet music scanning process. Since LumiKey depends on converting sheet music images into musical data, poor image quality or recognition errors could lead to incorrect notes being detected. To manage this risk, the scan pipeline now includes stronger preprocessing steps to improve image quality before running the Optical Music Recognition (OMR) engine. The system also includes validation checks to detect problems with the generated MusicXML files before the data moves to the next stage of the pipeline. If recognition accuracy becomes an issue, the team can restrict inputs to clearer scans or PDFs while continuing to improve the preprocessing system.

Another risk involves timing synchronization between the software playback system and the hardware. If the system does not process note events in the correct order or if communication delays occur when sending data to the ESP32, the LED guidance may not match the intended timing of the music. To address this, the playback algorithm was improved to handle event sequencing and timing more consistently. Further testing will be done once the software is fully connected to the hardware.

A third risk is handling different types of input files, such as corrupted images or invalid PDFs. To reduce this risk, additional automated tests were added to check the scan pipeline and ensure that errors are detected early rather than causing failures later in the system.

The fourth risk is ghost detection from the FSR keys. To mitigate this risk, we will separate the FSRs into a cleaner wiring scheme to reduce electrical noise and interference. This will help ensure that the ESP32 only detects legitimate key presses and avoids false signals caused by ghost inputs.

Design Changes

Some improvements were made to the system design to make the software pipeline more flexible and easier to use. One important change was adding support for both image and PDF sheet music inputs. The system can now convert PDF pages into images so they can go through the same preprocessing and recognition pipeline.

The scan pipeline was also reorganized into smaller modules to make the code easier to maintain and test. In addition, the OMR integration was improved so the system can automatically detect the Audiveris engine and handle different output formats more reliably.

On the playback side, updates were made to the event processing system so musical events are handled in a clearer and more predictable order during playback. This helps improve the timing behavior of the system and prepares it for integration with the visualizer and hardware components.

Schedule Update

The project remains on schedule. The scanning pipeline, OMR integration, and playback system are now functioning parts of the software. The team is moving from building individual components toward integrating the full system.

Demonstrated Progress

This week the team made significant progress on the software system. The scan pipeline can now process sheet music images or PDFs, run the OMR engine, and produce MusicXML output that can be used by the rest of the system.

Improvements were also made to the playback system to ensure musical events are processed more consistently. In addition, development began on a sheet music visualizer that will allow the system to display musical progress during playback. This visual component will help users connect the sheet music with the LED guidance on the keyboard.

Overall, the project is continuing to move from early development toward a fully integrated system that connects sheet music input, musical event processing, and guided playback.

The Gantt chart (Gantt) outlines the team’s current timeline.

Posted on

Simon Lee’s Status Report for March 14

This week I focused on expanding and stabilizing the Scan Layer so it works as a reliable part of the full LumiKey software pipeline. Building on last week’s work integrating the preprocessing system with the Audiveris OMR engine, the goal this week was to improve preprocessing, strengthen OMR execution, and make the system easier for teammates to run and test.

Expanded Scan Preprocessing Pipeline

The preprocessing module was expanded into a full normalization pipeline capable of handling both images and PDFs. When a user uploads a PDF, the pages are first rasterized using the pypdfium2 library so they can go through the same preprocessing steps as images.

The preprocessing pipeline now includes illumination normalization to reduce lighting artifacts, perspective correction for photos taken at an angle, skew estimation with deskew correction, adaptive thresholding tuned using staff spacing, and morphological filtering to remove small artifacts. These steps help produce cleaner images that are easier for the OMR engine to interpret. I also reorganized the code into smaller helper functions so that image processing steps are easier to maintain and test.

Improved OMR Integration

The OMR module was updated to run the Audiveris engine directly through the command line using Python’s subprocess interface. The system now automatically detects the Audiveris binary using environment variables, system paths, or common installation locations on macOS.

Additional handling was added for locating the MusicXML output generated by Audiveris, including support for both .musicxml and compressed .mxl files. When compressed output is produced, the system automatically extracts the MusicXML so it can continue through the pipeline.

MusicXML Validation Improvements

The MusicXML cleaning stage was improved to make the pipeline more reliable. The cleaner now performs basic validation checks such as newline normalization, empty-file checks, and XML parsing verification. If the MusicXML file is malformed or empty, the system produces clear error messages instead of failing silently. This helps ensure the Scan Layer always outputs valid data before it reaches the next stage of the pipeline.

Testing and Pipeline Validation

A major focus this week was improving testing coverage for the scan pipeline. I added several test files to validate preprocessing, OMR integration, MusicXML cleaning, and the full scan pipeline. These tests check that the system behaves correctly when processing different types of inputs and when errors occur.

The current test suite contains 27 passing tests. I also added a real sheet music example (twinkletwinkle.pdf) as a test asset so the full pipeline can be validated using realistic input.

Project Setup Improvements

I also made improvements to the project setup process so other team members can run the system more easily. I added a bootstrap script that automatically creates a Python virtual environment, installs dependencies, and detects or installs Audiveris on macOS. The UI launch script now calls this setup step automatically before starting the application. I also updated the README with clearer setup instructions so teammates can run the project with a simple command.

Schedule

The project remains on schedule. The Scan Layer can now process both image and PDF inputs, run the Audiveris OMR engine, and produce validated MusicXML outputs. The system is also supported by automated tests, which will help maintain stability as new features are added.

Deliverables for Next Week

Next week I will focus on validating the full pipeline now that the Scan Layer is integrated with the existing Conversion Layer. I plan to build a small benchmark set of about 15–25 sheet music files, including clean photos, skewed photos, shadowed images, and PDFs. This dataset will be used to evaluate how reliably the end-to-end pipeline works.

I will also add an evaluation script that runs the pipeline on the benchmark set and reports results such as success or failure, parse errors, and basic note matching where expected outputs are available. In addition, I plan to improve error reporting so the application clearly identifies which stage of the pipeline fails if a scan does not succeed.

Finally, I will begin preparing for hardware integration by documenting the event format used to send note data to the ESP32 and adding a test to ensure the event serialization remains consistent for hardware communication.

Posted on

Chris Oh’s Status Report for March 14

This week I focused on improving the playback system and starting development of a basic sheet music visualizer. The main work involved refining the playback algorithm to better handle musical event timing and building an initial framework that will allow the system to visually represent sheet music during playback.

Playback System Updates
• Refined the playback algorithm to improve event sequencing and timing consistency.
• Reorganized the event processing logic to ensure that note events are triggered in a predictable order during playback.
• Improved handling of note-on and note-off events to make playback behavior more stable and easier to extend.
• Adjusted the playback pipeline so that event timing is managed more cleanly between the software processing layer and the output layer.
• These refinements help ensure that playback aligns more closely with the intended musical structure.

Sheet Music Visualizer Framework
• Began implementing a basic framework for a sheet music visualization component.
• Established an initial structure for representing musical events visually during playback.
• Implemented early components that track playback position and note timing.
• Designed the framework so it can later support highlighting notes or measures while the music is playing.
• The goal of this component is to help users visually connect the sheet music with the LED guidance on the keyboard.

Schedule

I am currently on schedule. The playback algorithm has been refined and the initial visualizer framework has been established. The project continues to move from early architecture work into functional implementation of core system components.

Deliverables for Next Week
• Integrate the playback algorithm with the sheet music visualizer framework.
• Improve timing synchronization between playback events and visual display updates.
• Expand the visualizer to support note highlighting and playback progress tracking.
• Begin testing the system with multiple pieces of sheet music to verify playback behavior.

Posted on

Gene Chang’s Status Report for Feb 28

Implemented User Feedback Modes and Interaction Logic

This week I focused on designing and testing different visual feedback methods to guide users during key presses. The ESP32 hardware layer is now fully connected to the core software layer through Bluetooth communication, allowing the system to receive note data and control the LED indicators in real time.

Three interaction modes were implemented: Basic Mode, Beat Mode, and Play-Through Mode. These modes allow the system to guide the user through different learning experiences, ranging from simple key highlighting to full song playback visualization.

To improve usability, I implemented a timing tolerance system that evaluates whether a user presses a key within an acceptable time window relative to the metronome beat. This allows the system to determine whether a note is played correctly or incorrectly without requiring perfect timing, making the interaction more forgiving for beginners.

The visual feedback behavior was also refined. When a user presses the correct key, the corresponding LED turns green and gradually fades, indicating that the note was successfully played. If the user presses an incorrect key, the LED blinks red, signaling the mistake and preventing the note sequence from advancing. This feedback loop helps users quickly understand their errors while practicing.

Restructured Musical Timing Data

Another major improvement this week was restructuring the JSON format used to represent musical notes. Previously, notes were processed sequentially, but the new structure now includes explicit start time values for each note.

This change allows notes to activate based on their absolute timing positions, enabling the implementation of the Play-Through Mode, where LEDs illuminate the full sequence of keys according to the timing of the entire song. This significantly improves the system’s ability to represent more complex musical passages where multiple notes may occur simultaneously or overlap in time.

Implemented Metronome-Based Start Countdown

To help users prepare before the song begins, I implemented a three-beat countdown indicator using the LED system. Before playback starts, the LEDs blink three times in synchronization with the system’s metronome.

This visual countdown ensures that users can align themselves with the tempo before beginning to play, improving the overall learning experience and reducing confusion about when the song starts.

Status Report Requirement: Part C (Environmental Factors)

Environmental factors were considered in the design of LumiKeys with respect to material usage, energy consumption, and the lifespan of electronic components. The system is designed as a retrofit solution that can be installed on existing acoustic or digital pianos rather than requiring the production of a completely new instrument. By augmenting instruments that users already own, the project helps reduce electronic waste and minimizes the environmental impact associated with manufacturing new hardware.

Additionally, the hardware architecture was designed to use low-power components, such as the ESP32 microcontroller and addressable NeoPixel LEDs, which can operate efficiently with a single regulated power supply. The system also minimizes unnecessary processing by transmitting compact musical data through Bluetooth rather than requiring continuous high-bandwidth communication.

Another environmental consideration involves the durability and modularity of the hardware design. Components such as the FSR strips, multiplexer, and LED strip are connected in a modular manner so they can be repaired or replaced individually without discarding the entire system. This design approach extends the usable lifetime of the device and reduces electronic waste.

Overall, by focusing on retrofit compatibility, efficient power usage, and modular hardware components, the LumiKeys system aims to reduce its environmental footprint while providing an accessible tool for music learning.

Schedule

I am currently on schedule. Bluetooth communication between the core layer and the hardware layer is stable, and the LED feedback system is functioning correctly across all implemented modes. The restructuring of the JSON timing data also enables more complex playback functionality moving forward.

Deliverables for Next Week

Next week I plan to integrate the first octave FSR strips with the analog multiplexer so the ESP32 can begin detecting real key presses. I will set up a simplified prototype using one octave of LEDs and FSR sensors and test it using songs that only require a single octave.

During testing, I will measure the timing delay between LED activation and FSR detection to evaluate whether the system introduces any noticeable latency for the user. These tests will help verify that the sensing and LED feedback systems work reliably before scaling to the full keyboard.

Posted on

Team Status Report for Feb 28

  • Status Report 4 Requirements: Part A was written by Chris, B was written by Simon, and C was written by Gene.

Project Risks and Mitigation

One of the biggest risks for the LumiKey project is inconsistent Optical Music Recognition (OMR) results when processing phone photos of sheet music. Phone images can have shadows, glare, skew, or uneven lighting, which can make it difficult for the OMR engine to correctly detect notes. To manage this risk, the Scan Layer preprocessing pipeline performs several normalization steps such as lighting correction, adaptive thresholding, and skew detection. The system also includes a quality check stage that detects problems like blur or glare before running OMR. If the scan quality is too poor, the system can flag the input rather than producing incorrect results. As a contingency plan, if phone photos remain unreliable, the system can restrict input to higher-quality images or PDF sheet music.

Another risk is latency between the software and the ESP32 hardware. If the system takes too long to respond to a key press, the LED feedback may feel delayed or out of sync with the music. To reduce this risk, the latency budget was divided across sensing, processing, Bluetooth transmission, and LED actuation. The system also uses buffered communication so that the ESP32 always has upcoming notes ready to display. If communication delays become an issue, a fallback solution is to preload larger sections of the music sequence onto the microcontroller.

A third risk involves reliable key press detection using force-sensitive resistors (FSRs). Because FSR sensors measure pressure, they can sometimes produce noisy signals depending on how the user presses the key. To manage this, the firmware uses threshold filtering and debounce logic to reduce false triggers. If sensor readings remain inconsistent, thresholds can be adjusted or the sensor placement can be refined without redesigning the hardware.

Posted on

Simon Lee’s Status Report for Feb 28

This week I focused on integrating the Scan Layer preprocessing pipeline with the Optical Music Recognition (OMR) stage and building tools to validate that the preprocessing output produces reliable symbolic music data. My goal was to move the system from an isolated preprocessing pipeline to a functional end-to-end scan workflow that converts sheet music images into structured MusicXML data.

Integrated Audiveris OMR Execution

The main milestone this week was connecting the preprocessing pipeline directly to the Audiveris OMR engine. Previously the system stopped after producing normalized images, but it can now automatically pass the cleaned image output into Audiveris and generate MusicXML files. I implemented this integration using Python subprocess calls so the pipeline can invoke Audiveris programmatically and capture the resulting MusicXML output.

Once this was working, I tested the system on multiple sheet music samples to verify that the preprocessing improvements from last week were producing images that Audiveris could reliably interpret. I also validated the generated MusicXML by importing the results into MuseScore, which made it easier to visually compare the recognized notes against the original sheet music. This proof-of-concept confirmed that the preprocessing stage is producing OMR-compatible images and that the overall scan pipeline is technically feasible.

Implemented MusicXML Validation and Cleaning

After OMR execution, I began implementing a validation and cleanup stage for the generated MusicXML. OMR output can occasionally include inconsistent metadata, missing attributes, or formatting irregularities that can break downstream parsing. To address this, I used the music21 library to parse the generated MusicXML files and enforce a consistent structure.

This stage verifies that measures are correctly defined, removes unnecessary metadata, and ensures that note and duration information can be safely extracted by the Conversion Layer. The goal is to guarantee that the output from the Scan Layer is not only readable but also structurally reliable for the rest of the LumiKey pipeline.

Built Visualization Tools for Scan Layer Processing

To better understand how preprocessing affects OMR performance, I developed visualization scripts that highlight the structural features extracted from sheet music images. These tools overlay detected staff lines, barlines, and candidate noteheads on top of the processed image, making it easier to confirm that the system is correctly identifying musical structure before OMR is executed.

These visualizations also helped refine several preprocessing parameters, such as thresholding behavior and morphological filters used to isolate staff lines. In addition to helping with debugging, the visualization outputs will also be useful for illustrating the Scan Layer architecture in our design report.

Status Report 4 Requirements: Part B

Cultural factors were considered in the design of LumiKey because music education is strongly shaped by cultural traditions, learning practices, and accessibility within different communities. Many cultures place significant value on learning traditional instruments such as the piano, yet access to formal instruction and specialized digital instruments can vary widely depending on economic and regional circumstances. By designing LumiKey as a retrofit system that works with existing acoustic or digital pianos, the project respects the cultural importance of traditional instruments while lowering barriers to entry for learners. The system also avoids requiring proprietary hardware or software ecosystems, allowing it to be adapted for use in diverse educational settings, including home practice, community music programs, and classrooms across different cultural contexts.

Schedule

I am currently on schedule. The Scan Layer is now capable of running the full pipeline from image input through preprocessing and OMR execution. The remaining work in this stage primarily involves refining MusicXML cleanup and validating the pipeline on a larger dataset of sheet music images.

Deliverables for Next Week

Next week I plan to finalize the MusicXML validation logic and begin implementing the Conversion Layer that transforms symbolic music data into the time-indexed event representation used by the embedded system. I will also run the full scan pipeline on a larger dataset of sheet music images to evaluate pitch extraction accuracy and ensure the system can consistently meet our ≥90% correctness target. Additionally, I will work with the rest of the team to begin connecting the software pipeline to the ESP32 communication layer so recognized notes can be transmitted to the hardware for LED playback.

Posted on

Chris Oh’s Status Report for Feb 28

This week I focused on implementing and testing the conversion layer pipeline and adding an audio preview feature to support early validation of the system. The goal was to ensure that sheet music can successfully be converted into event data and previewed before hardware integration. In addition, I improved the pipeline to support direct MusicXML uploads so the system can be tested without relying on the OMR stage.

Conversion Layer Implementation

  • Completed implementation of the functionality to convert parsed MusicXML scores into structured note events.
  • Added a sanity check tool that verifies MusicXML parsing and event generation by loading a test score and printing basic statistics such as number of notes and generated events.
  • Improved robustness of the conversion module so it can run independently from the repository root, which simplifies development and testing.
  • Extended the pipeline to support direct MusicXML and MXL file uploads, allowing users to bypass the OMR stage when testing the system with known sheet music files.

Audio Playback and Testing

  • Implemented a lightweight audio synthesis system that generates simple sine-wave playback from MIDI note values. This allows the generated event data to be previewed without needing hardware.
  • Added support for chords and basic attack/release envelopes to reduce clicking artifacts during playback.
  • Implemented an event playback session system that supports play, pause, and stop controls while tracking elapsed playback time.
  • Integrated the playback system into the Streamlit interface as an Audio Preview feature. Users can now upload sheet music, generate events, and preview the resulting audio with adjustable BPM.
  • Added playback progress visualization and improved session state management so playback behaves correctly within Streamlit’s execution model.

Status Report 4 Requirements: Part A

From a global perspective, LumiKey addresses a common barrier in music education: the difficulty of learning to read sheet music and translate it into correct key presses on a piano. Many beginners around the world have access to inexpensive keyboards but lack access to formal instruction or structured learning tools. LumiKey aims to reduce this barrier by visually guiding users to the correct keys using LED indicators driven by sheet music input. Because the system relies on widely available hardware components and open-source software, it has the potential to be used by learners in many different environments, including schools, homes, and self-study settings where formal music instruction may not be available.

Schedule

I am currently on schedule. The conversion layer pipeline is now functional and can generate note events from MusicXML files. The addition of the audio preview feature also provides a useful way to validate the correctness of event generation before integrating with the hardware layer.

Deliverables for Next Week

  • Connect the event generation pipeline to the LED hardware interface.
  • Begin testing event timing synchronization between software playback and LED output.
  • Continue improving robustness of the conversion pipeline with additional sheet music examples.
  • Perform integration testing of the Streamlit interface with the full software pipeline.