Posted on

Gene Chang’s Status Report for March 21

This week primarily focused on continued development and debugging of the force-sensitive resistor (FSR) input system for the LumiKeys project. Progress was centered on stabilizing the detection of key presses across a smaller subset of the keyboard, specifically five keys, before scaling to the full system.

FSR Integration and Debugging

  • Worked on refining the code for reading and interpreting signals from five FSR strips connected through the analog multiplexer.
  • Focused on improving signal reliability and consistency, as initial readings showed noise and occasional false triggers.
  • Tested different threshold values and filtering approaches in software to better distinguish intentional key presses from background noise.
  • Continued debugging wiring and connection stability to ensure that each FSR input is correctly mapped and independently detected.

System Stability and Validation

  • Conducted repeated tests on the five-key setup to verify that key presses are consistently detected and transmitted.
  • Identified minor inconsistencies in readings, which are currently being investigated as potential issues related to wiring layout or power distribution.
  • Maintained the goal of achieving a stable and reliable baseline before expanding to additional keys.

Schedule

I am slightly behind the ideal timeline due to extended debugging of the FSR inputs, but this step is critical to ensure reliability before scaling to the full keyboard system.

Deliverables for Next Week

  • Finalize stable detection for the five FSR keys with minimal noise and false positives.
  • Implement improved filtering or smoothing techniques if necessary to enhance signal accuracy.
  • Begin scaling the working solution to additional keys once consistent performance is achieved.
Posted on

Team Status Report for March 21

Project Risks and Mitigation

A key ongoing risk on the software side is maintaining tight synchronization between playback events and visual feedback during sheet music visualization. If visual updates lag behind or drift from playback timing, users may receive misleading guidance, reducing the effectiveness of the system. This risk is especially relevant as the visualization layer becomes more tightly integrated with playback.

Fortunately, this risk is being actively and effectively managed in the current design. To mitigate this risk, the visualization system is driven directly by the existing playback event stream rather than introducing a separate timing mechanism. All computationally expensive processing remains in the preprocessing stage, and the visualization layer operates only on precomputed event data. This approach minimizes runtime overhead and reduces the likelihood of timing jitter during playback.

Another ongoing risk is variability in OMR accuracy across different input conditions, such as skewed images, shadows, or lower-quality photos. Inconsistent MusicXML output can propagate downstream and affect both event generation and visualization accuracy. This risk is being managed by expanding the preprocessing pipeline and introducing a benchmark dataset to evaluate performance across varied inputs. Additionally, validation and cleaning of MusicXML ensures that only structurally consistent outputs are passed to the Conversion Layer.

A related risk is pipeline fragility when handling edge cases, such as corrupted files, missing OMR outputs, or malformed MusicXML. These failures could disrupt the user experience if not handled properly. To mitigate this, the system now includes explicit error reporting at each stage of the pipeline, along with targeted tests for invalid inputs. This ensures failures are predictable, easier to debug, and do not silently propagate through the system.

As the system moves toward hardware integration, there is also a risk that mismatches between the software-generated event format and the ESP32 expectations could introduce timing or parsing issues. This is being addressed by clearly defining the event payload structure and validating serialization before integration, ensuring that communication between software and hardware remains consistent and reliable.

Design Changes

This week, the team implemented the sheet music visualization component and integrated it with the playback pipeline. The visualization layer now renders musical events in real time and highlights active notes based on playback position. This change improves system observability and usability by allowing users to visually verify that playback behavior matches the intended musical structure.

The visualization was designed to interface cleanly with the existing event-processing architecture, avoiding changes to the core timing logic. While this adds additional UI complexity, the design preserves modular boundaries by keeping visualization logic separate from event generation and hardware communication.

Schedule Update

There have been no changes to the project schedule. Development remains on track, and progress continues to align with the milestones outlined in the Gantt chart (Gantt).

Posted on

Simon Lee’s Status Report for March 21

This week I focused on validating and stabilizing the full end-to-end LumiKey software pipeline. Building on last week’s work where the Scan Layer was integrated with the OMR engine and connected to the existing Conversion Layer, the goal this week was to ensure that the entire flow – from image or PDF input to note event generation – works reliably and is ready for the interim demo.

End-to-End Pipeline Validation

A major focus this week was testing the complete pipeline: image/PDF input → preprocessing → OMR → MusicXML cleaning → conversion → note events. Since the Conversion Layer was already implemented by another team member, my work focused on verifying that the Scan Layer output integrates correctly and consistently produces valid inputs for downstream processing.

To support this, I created a small benchmark dataset of sheet music inputs, including clean images, skewed photos, shadowed images, and PDFs. This allowed me to test how well the system performs across realistic conditions. I also began evaluating outputs based on whether the pipeline succeeds, whether parsing errors occur, and whether the generated events are reasonable in terms of structure and count.

Evaluation and Debugging Improvements

I added a simple evaluation workflow that runs the pipeline across the benchmark set and reports results such as success/failure and basic output consistency. This helped identify weak points in preprocessing and OMR where certain inputs fail or produce inconsistent results.

In addition, I improved debugging visibility by ensuring that failures are tied to specific stages of the pipeline. Instead of failing silently, the system now makes it clearer whether an issue occurs during preprocessing, OMR execution, MusicXML cleaning, or parsing. This makes it much easier to diagnose problems and iterate quickly.

Additional Testing and Edge Case Handling

I added several targeted tests to improve robustness of the Scan Layer and pipeline. These include handling invalid inputs such as corrupted images, malformed PDFs, missing OMR outputs, and invalid MusicXML files. These tests ensure that the system fails in a predictable and controlled way rather than crashing or producing unusable results.

The goal of this work is to make the pipeline stable enough for demo conditions, where unexpected inputs or edge cases may occur.

Preparation for Hardware Integration

With the software pipeline now functioning end-to-end, I began preparing for integration with the hardware system. This includes defining and documenting the event format generated by the pipeline, which will be used to communicate note timing and pitch information to the ESP32.

This step is important to ensure that the software output aligns with the expectations of the embedded system, particularly for timing accuracy and LED control. It also sets up the next phase of development, where software-generated note events will drive real hardware behavior.

Schedule

The project remains on schedule and aligned with the interim demo timeline. According to the planned system pipeline and milestones , the current focus on integration and validation is consistent with reaching an MVP before the demo. The full software pipeline is now operational, and remaining work is focused on stability and hardware integration.

Deliverables for Next Week (Interim Demo Preparation)

Next week I will focus on final preparation for the interim demo. This includes refining the pipeline to ensure consistent behavior on demo inputs, improving error handling for a smoother user experience, and working with the team to connect the software output to the ESP32 hardware.

The goal is to demonstrate a complete working system where a user uploads sheet music and receives corresponding LED guidance on the keyboard. I will also continue refining preprocessing and validation steps based on any issues observed during integration testing.

Posted on

Chris Oh’s Status Report for March 21

This week I focused on implementing the sheet music visualization component and integrating it with the existing playback pipeline. The main goal was to move the visualizer from a conceptual framework into a functional system that can display musical information in sync with playback.

Sheet Music Visualization Implementation

  • Implemented the core sheet music visualization pipeline, enabling musical events to be rendered visually during playback.
  • Added logic to map parsed musical events to visual elements, allowing notes to be displayed in their correct temporal order.
  • Integrated playback position tracking so the visualizer updates in real time as the music progresses.
  • Implemented basic note highlighting behavior to reflect currently active notes during playback.
  • Ensured the visualization system cleanly interfaces with the existing event processing and playback logic.

Playback Integration

  • Connected the sheet music visualizer to the playback system so visual updates are driven directly by playback timing.
  • Refined event timing handling to keep visual transitions aligned with note-on and note-off events.
    Verified that visualization remains stable across different tempos and musical passages.

Schedule

I am currently on schedule. The sheet music visualizer has moved from a prototype framework to a working implementation, and it is now integrated with the playback pipeline. The project continues to make steady progress toward a cohesive end-to-end system.

Deliverables for Next Week

  • Improve visual clarity and layout of the sheet music display.
  • Add support for additional musical elements (e.g., rests or measure boundaries).
  • Continue testing synchronization between playback, visualization, and hardware output.
  • Begin refining the visualizer UI based on usability considerations.
Posted on

Gene Chang’s Status Report for March 14

This week focused mainly on wiring additional FSR strips to the 16-channel analog multiplexer and ensuring that each key press can be individually detected and registered. Currently, I’m encountering an issue where ghost signals appear from other keys. I suspect this may be caused either by the wires being routed too closely together or by a power stability issue. I also reviewed the firmware to determine whether the problem could be mitigated through software.

A significant portion of the week was also spent soldering the LEDs and FSR strips together for the hardware assembly.

Posted on

Team Status Report for March 14

Project Risks and Mitigation

One of the biggest risks to the project is the reliability of the sheet music scanning process. Since LumiKey depends on converting sheet music images into musical data, poor image quality or recognition errors could lead to incorrect notes being detected. To manage this risk, the scan pipeline now includes stronger preprocessing steps to improve image quality before running the Optical Music Recognition (OMR) engine. The system also includes validation checks to detect problems with the generated MusicXML files before the data moves to the next stage of the pipeline. If recognition accuracy becomes an issue, the team can restrict inputs to clearer scans or PDFs while continuing to improve the preprocessing system.

Another risk involves timing synchronization between the software playback system and the hardware. If the system does not process note events in the correct order or if communication delays occur when sending data to the ESP32, the LED guidance may not match the intended timing of the music. To address this, the playback algorithm was improved to handle event sequencing and timing more consistently. Further testing will be done once the software is fully connected to the hardware.

A third risk is handling different types of input files, such as corrupted images or invalid PDFs. To reduce this risk, additional automated tests were added to check the scan pipeline and ensure that errors are detected early rather than causing failures later in the system.

The fourth risk is ghost detection from the FSR keys. To mitigate this risk, we will separate the FSRs into a cleaner wiring scheme to reduce electrical noise and interference. This will help ensure that the ESP32 only detects legitimate key presses and avoids false signals caused by ghost inputs.

Design Changes

Some improvements were made to the system design to make the software pipeline more flexible and easier to use. One important change was adding support for both image and PDF sheet music inputs. The system can now convert PDF pages into images so they can go through the same preprocessing and recognition pipeline.

The scan pipeline was also reorganized into smaller modules to make the code easier to maintain and test. In addition, the OMR integration was improved so the system can automatically detect the Audiveris engine and handle different output formats more reliably.

On the playback side, updates were made to the event processing system so musical events are handled in a clearer and more predictable order during playback. This helps improve the timing behavior of the system and prepares it for integration with the visualizer and hardware components.

Schedule Update

The project remains on schedule. The scanning pipeline, OMR integration, and playback system are now functioning parts of the software. The team is moving from building individual components toward integrating the full system.

Demonstrated Progress

This week the team made significant progress on the software system. The scan pipeline can now process sheet music images or PDFs, run the OMR engine, and produce MusicXML output that can be used by the rest of the system.

Improvements were also made to the playback system to ensure musical events are processed more consistently. In addition, development began on a sheet music visualizer that will allow the system to display musical progress during playback. This visual component will help users connect the sheet music with the LED guidance on the keyboard.

Overall, the project is continuing to move from early development toward a fully integrated system that connects sheet music input, musical event processing, and guided playback.

The Gantt chart (Gantt) outlines the team’s current timeline.

Posted on

Simon Lee’s Status Report for March 14

This week I focused on expanding and stabilizing the Scan Layer so it works as a reliable part of the full LumiKey software pipeline. Building on last week’s work integrating the preprocessing system with the Audiveris OMR engine, the goal this week was to improve preprocessing, strengthen OMR execution, and make the system easier for teammates to run and test.

Expanded Scan Preprocessing Pipeline

The preprocessing module was expanded into a full normalization pipeline capable of handling both images and PDFs. When a user uploads a PDF, the pages are first rasterized using the pypdfium2 library so they can go through the same preprocessing steps as images.

The preprocessing pipeline now includes illumination normalization to reduce lighting artifacts, perspective correction for photos taken at an angle, skew estimation with deskew correction, adaptive thresholding tuned using staff spacing, and morphological filtering to remove small artifacts. These steps help produce cleaner images that are easier for the OMR engine to interpret. I also reorganized the code into smaller helper functions so that image processing steps are easier to maintain and test.

Improved OMR Integration

The OMR module was updated to run the Audiveris engine directly through the command line using Python’s subprocess interface. The system now automatically detects the Audiveris binary using environment variables, system paths, or common installation locations on macOS.

Additional handling was added for locating the MusicXML output generated by Audiveris, including support for both .musicxml and compressed .mxl files. When compressed output is produced, the system automatically extracts the MusicXML so it can continue through the pipeline.

MusicXML Validation Improvements

The MusicXML cleaning stage was improved to make the pipeline more reliable. The cleaner now performs basic validation checks such as newline normalization, empty-file checks, and XML parsing verification. If the MusicXML file is malformed or empty, the system produces clear error messages instead of failing silently. This helps ensure the Scan Layer always outputs valid data before it reaches the next stage of the pipeline.

Testing and Pipeline Validation

A major focus this week was improving testing coverage for the scan pipeline. I added several test files to validate preprocessing, OMR integration, MusicXML cleaning, and the full scan pipeline. These tests check that the system behaves correctly when processing different types of inputs and when errors occur.

The current test suite contains 27 passing tests. I also added a real sheet music example (twinkletwinkle.pdf) as a test asset so the full pipeline can be validated using realistic input.

Project Setup Improvements

I also made improvements to the project setup process so other team members can run the system more easily. I added a bootstrap script that automatically creates a Python virtual environment, installs dependencies, and detects or installs Audiveris on macOS. The UI launch script now calls this setup step automatically before starting the application. I also updated the README with clearer setup instructions so teammates can run the project with a simple command.

Schedule

The project remains on schedule. The Scan Layer can now process both image and PDF inputs, run the Audiveris OMR engine, and produce validated MusicXML outputs. The system is also supported by automated tests, which will help maintain stability as new features are added.

Deliverables for Next Week

Next week I will focus on validating the full pipeline now that the Scan Layer is integrated with the existing Conversion Layer. I plan to build a small benchmark set of about 15–25 sheet music files, including clean photos, skewed photos, shadowed images, and PDFs. This dataset will be used to evaluate how reliably the end-to-end pipeline works.

I will also add an evaluation script that runs the pipeline on the benchmark set and reports results such as success or failure, parse errors, and basic note matching where expected outputs are available. In addition, I plan to improve error reporting so the application clearly identifies which stage of the pipeline fails if a scan does not succeed.

Finally, I will begin preparing for hardware integration by documenting the event format used to send note data to the ESP32 and adding a test to ensure the event serialization remains consistent for hardware communication.

Posted on

Chris Oh’s Status Report for March 14

This week I focused on improving the playback system and starting development of a basic sheet music visualizer. The main work involved refining the playback algorithm to better handle musical event timing and building an initial framework that will allow the system to visually represent sheet music during playback.

Playback System Updates
• Refined the playback algorithm to improve event sequencing and timing consistency.
• Reorganized the event processing logic to ensure that note events are triggered in a predictable order during playback.
• Improved handling of note-on and note-off events to make playback behavior more stable and easier to extend.
• Adjusted the playback pipeline so that event timing is managed more cleanly between the software processing layer and the output layer.
• These refinements help ensure that playback aligns more closely with the intended musical structure.

Sheet Music Visualizer Framework
• Began implementing a basic framework for a sheet music visualization component.
• Established an initial structure for representing musical events visually during playback.
• Implemented early components that track playback position and note timing.
• Designed the framework so it can later support highlighting notes or measures while the music is playing.
• The goal of this component is to help users visually connect the sheet music with the LED guidance on the keyboard.

Schedule

I am currently on schedule. The playback algorithm has been refined and the initial visualizer framework has been established. The project continues to move from early architecture work into functional implementation of core system components.

Deliverables for Next Week
• Integrate the playback algorithm with the sheet music visualizer framework.
• Improve timing synchronization between playback events and visual display updates.
• Expand the visualizer to support note highlighting and playback progress tracking.
• Begin testing the system with multiple pieces of sheet music to verify playback behavior.

Posted on

Gene Chang’s Status Report for Feb 28

Implemented User Feedback Modes and Interaction Logic

This week I focused on designing and testing different visual feedback methods to guide users during key presses. The ESP32 hardware layer is now fully connected to the core software layer through Bluetooth communication, allowing the system to receive note data and control the LED indicators in real time.

Three interaction modes were implemented: Basic Mode, Beat Mode, and Play-Through Mode. These modes allow the system to guide the user through different learning experiences, ranging from simple key highlighting to full song playback visualization.

To improve usability, I implemented a timing tolerance system that evaluates whether a user presses a key within an acceptable time window relative to the metronome beat. This allows the system to determine whether a note is played correctly or incorrectly without requiring perfect timing, making the interaction more forgiving for beginners.

The visual feedback behavior was also refined. When a user presses the correct key, the corresponding LED turns green and gradually fades, indicating that the note was successfully played. If the user presses an incorrect key, the LED blinks red, signaling the mistake and preventing the note sequence from advancing. This feedback loop helps users quickly understand their errors while practicing.

Restructured Musical Timing Data

Another major improvement this week was restructuring the JSON format used to represent musical notes. Previously, notes were processed sequentially, but the new structure now includes explicit start time values for each note.

This change allows notes to activate based on their absolute timing positions, enabling the implementation of the Play-Through Mode, where LEDs illuminate the full sequence of keys according to the timing of the entire song. This significantly improves the system’s ability to represent more complex musical passages where multiple notes may occur simultaneously or overlap in time.

Implemented Metronome-Based Start Countdown

To help users prepare before the song begins, I implemented a three-beat countdown indicator using the LED system. Before playback starts, the LEDs blink three times in synchronization with the system’s metronome.

This visual countdown ensures that users can align themselves with the tempo before beginning to play, improving the overall learning experience and reducing confusion about when the song starts.

Status Report Requirement: Part C (Environmental Factors)

Environmental factors were considered in the design of LumiKeys with respect to material usage, energy consumption, and the lifespan of electronic components. The system is designed as a retrofit solution that can be installed on existing acoustic or digital pianos rather than requiring the production of a completely new instrument. By augmenting instruments that users already own, the project helps reduce electronic waste and minimizes the environmental impact associated with manufacturing new hardware.

Additionally, the hardware architecture was designed to use low-power components, such as the ESP32 microcontroller and addressable NeoPixel LEDs, which can operate efficiently with a single regulated power supply. The system also minimizes unnecessary processing by transmitting compact musical data through Bluetooth rather than requiring continuous high-bandwidth communication.

Another environmental consideration involves the durability and modularity of the hardware design. Components such as the FSR strips, multiplexer, and LED strip are connected in a modular manner so they can be repaired or replaced individually without discarding the entire system. This design approach extends the usable lifetime of the device and reduces electronic waste.

Overall, by focusing on retrofit compatibility, efficient power usage, and modular hardware components, the LumiKeys system aims to reduce its environmental footprint while providing an accessible tool for music learning.

Schedule

I am currently on schedule. Bluetooth communication between the core layer and the hardware layer is stable, and the LED feedback system is functioning correctly across all implemented modes. The restructuring of the JSON timing data also enables more complex playback functionality moving forward.

Deliverables for Next Week

Next week I plan to integrate the first octave FSR strips with the analog multiplexer so the ESP32 can begin detecting real key presses. I will set up a simplified prototype using one octave of LEDs and FSR sensors and test it using songs that only require a single octave.

During testing, I will measure the timing delay between LED activation and FSR detection to evaluate whether the system introduces any noticeable latency for the user. These tests will help verify that the sensing and LED feedback systems work reliably before scaling to the full keyboard.

Posted on

Team Status Report for Feb 28

  • Status Report 4 Requirements: Part A was written by Chris, B was written by Simon, and C was written by Gene.

Project Risks and Mitigation

One of the biggest risks for the LumiKey project is inconsistent Optical Music Recognition (OMR) results when processing phone photos of sheet music. Phone images can have shadows, glare, skew, or uneven lighting, which can make it difficult for the OMR engine to correctly detect notes. To manage this risk, the Scan Layer preprocessing pipeline performs several normalization steps such as lighting correction, adaptive thresholding, and skew detection. The system also includes a quality check stage that detects problems like blur or glare before running OMR. If the scan quality is too poor, the system can flag the input rather than producing incorrect results. As a contingency plan, if phone photos remain unreliable, the system can restrict input to higher-quality images or PDF sheet music.

Another risk is latency between the software and the ESP32 hardware. If the system takes too long to respond to a key press, the LED feedback may feel delayed or out of sync with the music. To reduce this risk, the latency budget was divided across sensing, processing, Bluetooth transmission, and LED actuation. The system also uses buffered communication so that the ESP32 always has upcoming notes ready to display. If communication delays become an issue, a fallback solution is to preload larger sections of the music sequence onto the microcontroller.

A third risk involves reliable key press detection using force-sensitive resistors (FSRs). Because FSR sensors measure pressure, they can sometimes produce noisy signals depending on how the user presses the key. To manage this, the firmware uses threshold filtering and debounce logic to reduce false triggers. If sensor readings remain inconsistent, thresholds can be adjusted or the sensor placement can be refined without redesigning the hardware.