Posted on

Team Status Report for 4/12

This week, our team made progress on finalizing and debugging our subsystems as well as starting integration. Lucas added audio playback to the game loop and worked on integrating his components with Yuhe’s main menu. Yuhe worked on the beat map editor, adding waveform viewer and interactions to edit notes. Yuhe is also working on migrating the game to a Windows system in order to solve audio card reading issues when using the Linux virtual environment. Michelle continued testing and refining her rhythm analysis algorithm, moving to a new method that has yields higher accuracy, as shown below in a test on a test piano piece.

After integrating the subsystems we will do some integration tests to ensure all the components are communicating with each other correctly. There are several metrics we will need to focus on, including beat map accuracy, audio and falling tile synchronization, gameplay input latency, persistent storage validation, frame rate stability, and error handling. Both beat map alignment and input latency should be under 20ms to ensure a seamless game experience. The rhythm analysis should capture at at least 95% of the notes and have no false positives. Error handling should cover issues such as unexpected file formats, file sizes that are too large, and invalid file name inputs.

For validation of the use case requirements, we will do some iterative user testing and collect some qualitative feedback about the ease of use, difficulty of the game, accuracy of the rhythm synchronization, and overall experience. During user testing, users will upload their own choice of songs and play the game with the automated beat map and also try out the beat map editor as well. We will want to validate that the whole flow is intuitive and user-friendly.

Posted on

Lucas’ Status Report for 4/12

This week, I worked primarily on integration and debugging. I added an audio component to the game loop itself, allowing the music and game to play simultaneously. I was primarily focused on integrating mine and Yuhe’s parts of the game, adding some way to transition from the menus into the game itself by selecting the song the user would like to play. On a song request, the game must read which song was selected and then fetch the associated audio file and json representation of the game, which I still need to find a way to efficiently store.

Next week I’ll finish integration and audio synchronization, allowing for a seamless transition from menu to game and ensuring that the game itself is exactly what it needs to be for the user request.

As far as testing and verification, since it’s a bit more difficult to quantitatively test a game like this, I’ll start by having some user playtesting – I’ll then gather feedback and try to deploy it within the game. I’ll then try to measure performance, primarily FPS, and ensure that it meets our desired 30+ outlined in our initial design requirements. Further, I’ll likely need to just play the game with a number of different audio files, ensuring that the notes and music are synced, and making sure that there are no errors in parsing the JSON file.

Posted on

Yuhe’s Status Report for 4/12

Over the past two weeks, I have made significant progress in both the development and stabilization of the Beat Map Editor, one of the core components of our rhythm game. My primary efforts were focused on implementing interactive editor features, resolving persistent audio playback issues encountered in the Ubuntu virtual environment, and initiating the migration of the project to a native Windows environment to ensure audio stability. This report details the technical progress, debugging process, cross-platform adaptations, and testing strategies aimed at meeting the engineering design specifications and use-case requirements of our project.

Beat Map Editor Development

The Beat Map Editor serves as the interactive interface for creating and modifying rhythm game charts. I designed and implemented this module with modularity and real-time user interaction in mind. The editor is built around a central BeatMapEditor class that integrates a collection of UI components such as buttons (UIButton), text input boxes (UITextBox), and a waveform visualization module (WaveformViewer).

The WaveformViewer class is responsible for rendering a real-time visual representation of the audio waveform using sf::Vertex primitives. It allows users to scroll and zoom through audio data within a fixed time window (defaulted to 10 seconds), providing an intuitive interface for note placement. I implemented the waveform extraction by sampling peaks from the left audio channel stored in an sf::SoundBuffer.

To facilitate note creation, I implemented two synchronized input fields: a timestamp box and a lane index box. When both fields are populated, the editor stores note data into a std::unordered_set of timestamps and serializes them into a beatmap vector. This mechanism ensures constant-time lookups to prevent duplicate entries and simplifies beatmap export functionality.

I also integrated a minimal but effective UI manager class (UIManager) to streamline event routing and rendering. This modular structure lays the foundation for extensibility and paves the way for implementing future features such as drag-and-drop note placement or real-time preview playback.

Audio Subsystem Debugging (Ubuntu)

During implementation and testing, I encountered recurring failures in the audio subsystem when running the editor in an Ubuntu 22.04 virtual environment. These issues included the inability to initialize or play audio using SFML’s sf::Sound, sporadic freezing during playback, and cryptic runtime errors from the underlying OpenAL backend such as:

AL lib: (EE) alc_open_device: No such audio device
AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device

These errors appeared to stem from the mismatch between the VM’s virtualized audio hardware and OpenAL Soft’s hardware requirements. Even with correct installation and runtime dependencies, the virtual sound card often failed to interface properly with SFML’s audio engine, especially after seek operations or audio buffer reloads.

To isolate the issue, I simplified the CMake configuration to exclude SFML’s audio components initially, later reintroducing them with explicit linkage:

find_package(SFML 2.5 COMPONENTS graphics window system audio REQUIRED)

Despite these adjustments, the playback remained inconsistent. I conducted multiple tests with .wav, .ogg, and .mp3 files and monitored buffer behavior. Logging showed that buffer assignment and sound object lifetimes were valid, but device-level playback would intermittently fail, especially during real-time seeking.

Windows Migration for Audio Stability

Given the limitations of virtualized audio under Ubuntu, I initiated a platform migration to a native Windows environment. On Windows, SFML’s audio subsystem integrates directly with the physical sound card through Windows WASAPI, bypassing the openAL soft backend configuration altogether.

To support this transition, I rebuilt the project’s CMake toolchain using Clang++ and reconfigured dependencies. The following Windows-based setup was finalized:

  • SFML 2.5 (precompiled binaries)

  • spdlog (via vcpkg)

  • tinyfiledialogs (compiled as a static library)

  • Python3 and pybind11 (for future audio analysis integrations)

On Windows, all audio files were loaded and played back reliably. The waveform viewer remained in sync, and no audio crashes or freezes were observed during extended test runs. This confirmed that migrating to Windows would provide the stable execution environment needed for reliable audio interactions during gameplay and editing.

Testing Strategy and Metrics (Verification of My Subsystem)

To ensure the editor meets both engineering design specifications and functional use-case goals, I have implemented and planned a suite of tests that span user interaction, timing accuracy, scoring behavior, and system stability.

Current Testing Status

I manually verified UI responsiveness by logging event triggers and action completions, confirming a latency under 50ms for input actions such as play/pause toggles, seek operations, and waveform interactions. The beat map editor’s file saving and loading functionality was profiled using a stopwatch and consistently completed under 5 seconds for standard-length songs (up to 7 minutes).

Waveform rendering and scrolling behavior was tested with songs across different sample rates and durations. The system maintained accurate synchronization between the progress bar and the waveform display. This also verified correct use of SFML’s sample indexing and buffer durations.

While most manual editor features passed their respective design thresholds, a subset of functionality—such as input latency and beat map accuracy—will require automated benchmarking to validate against exact numerical thresholds.

Planned Testing and Metrics (Verification of the Entire Game)

In the few weeks before the final demo, we will expand testing as follows:

  1. Beat Map Accuracy
    Using annotated reference beat maps and a Python evaluation script, I will calculate average temporal deviation between auto-generated and human-placed notes. A root mean squared error below 20ms will be required to meet alignment standards for BPM ranges between 50–220.

  2. Gameplay Input Latency
    I will record key press events and game state transitions using an SDL2-based input logger or high-frame-rate video capture. These measurements will quantify the time from physical input to in-game response, with a target latency below 20ms.

  3. Persistent Storage Validation
    Save and reload operations for beat maps will be tested using a hash-diff script that validates structural integrity and data completeness. Changes to the note set must persist across game sessions with no corruption.

  4. Frame Rate Stability
    Using SFML’s internal sf::Clock, I will measure delta time across multiple frames during stress test scenarios (waveform scrolling + playback + note placement). The editor must maintain at least 30 frames per second without frame drops.

  5. Error Handling
    Simulated tests with malformed audio files and corrupted beat map data will validate error-handling routines. The editor must exit gracefully or display user-friendly error dialogs instead of crashing.

Posted on

Michelle’s Status Report for 4/12

This week, I continued testing and refining the rhythm analysis algorithm. I tested out a second version of the algorithm that more heavily weights the standard deviation of the onset strengths into determining whether to count a peak as a note or not. This version is much more accurate across various tempos, as shown in the figure below. These are the results of testing a self-made single clef piano composition. The first version would have more false positives at a very slow tempo and false negatives at a very fast tempo, the missed notes typically being any 32nd notes or some 16th notes. The second version when tested on the same piece performs much better, only missing a few 32nd notes at very fast tempos.

The verification methodology involves creating compositions in MuseScore and generating audio files to test the processing algorithm on. This way, I have an objective truth of the tempo and rhythm and can easily manipulate variables such as the instrument, dynamics, time signature, etc. and see how these affect the accuracy. Additionally, I also test the algorithm using real songs, which often have more noise and more blended sounding notes. Using a Python program, I can run the analysis on a song I uploaded, and playback the song while showing an animation that blinks on the extracted timestamps and record any missed or added notes. To verify that my subsystem meets the design requirements, the algorithm must capture at least 95% of the notes without adding any extra notes of single-instrument songs between 50 and 160 BPM.

Comparing results of V1 and V2 on a piano composition created in MuseScore

I also tested an algorithm that uses a local adaptive threshold instead of a global threshold. This version uses a sliding window so it compares onset strengths more locally, which can allow the algorithm to be more adaptive over the course of a piece especially when there are changes in dynamics. The tradeoff with this is that it can be more susceptible to noise.

I am on track with the project schedule. I think the current version is sufficient for the MVP of this subsystem, so further work will just be more extensive testing and stretch goals for more complex music. I have begun creating more compositions with even more complex rhythms, including time signature changes, which I plan to test this V2 on next week. I also will test the algorithm on pieces with drastic dynamic changes. I plan to play around with the minimum note length more as well. Since V2 is experiencing less false positives, I may be able to decrease this from the current 100ms to accommodate more complex pieces. Additionally, I want to test out a version that uses median absolute deviation instead of standard deviation to see if this outperforms V2. This method will be less sensitive to extreme peaks.

Posted on

Team Status Report for 3/29

This week, we made progress toward integrating the audio processing system with the game engine. We implemented a custom JSON parser in C++ to load beatmap data generated by our signal processing pipeline, enabling songs to be played as fully interactive game levels. We also added a result splash screen at the end of gameplay, which currently displays basic performance metrics and will later include more detailed graphs and stats.

In the editor, we refined key systems for waveform visualization and synchronization. We improved timeToPixel() and pixelToTime() mappings to ensure the playhead and note grid align accurately with audio playback. We also advanced snapping logic to quantize note timestamps based on BPM and introduced an alternative input method for placing notes at specific times and lanes, in addition to drag-and-drop.

On the signal processing side, we expanded rhythm extraction testing to voice and bowed instruments and addressed tempo estimation inconsistencies by using a fixed minimum note length of 0.1s. We also updated the JSON output to include lane mapping information for smoother integration.

Next week, we plan to connect the main menu to gameplay, polish the UI, and fully integrate all system components.

Posted on

Lucas’ Status Report for 3/29

This week I focused on finishing integration of my game with the signal processing by adding a functioning JSON parser that will allow for the output from the music analysis to be played as a game. I also added a result splash screen that displays at the end of each game, with a few more detailed statistics and graphs that I plan to add to later.

Next week I’d like to finish integration to allow for a gateway between the main menus and the game itself, ensuring that each song saved by the user can be clicked on and played. I’d also like to clean up some UI and make the game more visually appealing.

Posted on

Michelle’s Status Report for 3/29

This week, I continued finetuning the audio processing algorithm. I continued testing with piano and guitar and also started testing voice and bowed instruments. These are harder to extract the rhythm from since the articulation can be a lot more legato. If we used pitch information, it may be possible to distinguish note onsets in slurs, for example, but this is most likely out of scope for our project.

Also, there was a flaw in calculating the minimum note length based on the estimated tempo because sometimes a song that most people would consider 60 BPM, librosa would estimate 120 BPM, which is technically equivalent, but then the calculated minimum note length would be much smaller and result in a lot of “double notes”, or note detections directly after one another that resulted from one more sustained note. For the game experience, I believe it is better to have more false negatives than false positives. I think having a fixed minimum note length will be a better generalization. A threshold of 0.1 seconds seems to work well.

Additionally, In preparation to integrate the music processing with the game, I added some more information to the JSON output that bridges the two parts. Based on the number of notes in for a given timestamp, the lane numbers are randomly chosen from which the tiles will fall from.

Example JSON output

My progress is on schedule. Next week, I plan to finalize my work on processing the rhythm of single-instrument tracks and meet with my teammates to integrate all of our subsystems together.

Posted on

Yuhe’s Status Report for 3/29

This week, I focused on implementing and debugging the waveform viewer and its synchronization with the audio playback system. I refined the timeToPixel(float timestamp) and pixelToTime(int x) functions to ensure accurate coordinate transformation between time and screen space, allowing the playhead and notes to sync with real-time audio. I also began implementing the snapping grid logic by quantizing timestamps to the nearest beat subdivision (e.g., 1/4, 1/8) based on BPM and audio offset. For note placement, I encountered difficulties with SFML’s event system for implementing smooth drag-and-drop behavior. As an alternative, I developed an input-based method that allows users to specify the number of notes, their lane indices, and timestamps via a dialog box, which then injects the notes into the grid. These changes aim to improve precision and editor usability. However,  it is still very challenging to ensure waveform-playhead alignment under various zoom levels.

Posted on

Team Status Report 3/22

The team has all been progressing individually on their respective parts of the game over the past week. Michelle has been continuing work on the audio processing side, deciding to focus on perfecting the processing of monophonic piano pieces while Lucas and Yuhe are continuing work on the game itself. I’ve continued work on the core game loop, adding back many of the key features while implementing a JSON parser to turn processed audio into a game, and Yuhe has begun working on the in game beat map editor while also adding some neat visual tools to the game.

We haven’t had any major changes to our game’s design, other than the fact that the audio processing side will focus more on monophonic pieces than longer and more dense stuff. This will probably be our biggest challenge going forward in the coming weeks; as long as we are able to integrate the game itself and the beat map editor/menus, we should be able to devote more time and resources to figuring out more complex audio processing.

Posted on

Yuhe’s Status Report for 3/22

This week I worked on designing and implementing key features of the Beatmap Editor for our game. In the first half of the week, I worked on the layout system, structuring the editor interface to accommodate a waveform display area, time axis, note grid overlay, and playback controls. I integrated JSON-based waveform data preprocessed externally to visualize amplitude over time using SFML’s VertexArray class. In the latter half, I worked on the note grid system that allows users to place notes along multiple lanes. Each lane corresponds to a specific input key, and note placement is quantized based on timestamp intervals to ensure rhythm accuracy. A snapping mechanism aligns notes to the closest beat subdivision, improving usability.

I think the major challenge is synchronizing the visual playhead with the music playback and managing coordinate transforms between waveform time and screen space. My goals for next week include implementing audio playback controls and note data serialization.