Our project was scheduled to reach MVP this week. We’ve met this benchmark in some ways, but have fallen short in other ways.
An MVP is suppose to be the minimum viable version of our game. This means that game is supposed to be in a playable state, and that we could theoretically give the game to a friend and they could play it start to finish. Our game isn’t quite at this stage yet. Instead, this week we were able to successfully complete a tech demo of the entire pipeline of technologies: a signal can get sent from the drum, be received by the computer, and then displayed on the screen.
Here is a screenshot of the game:
Our project currently meets the MVP benchmark in the sense that “all of the technologies have been tested and they can interact with each other”. However, our project falls short because it is not yet an engaging video game experience. There is still a lot of aesthetic changes that need to be added to the game before it feels like a fun experience.
We ran into an unexpected challenges with integration this week. The game does not compile on Windows, which means that Shreya and Stephen cannot run the game on their computers yet. We need to divert some of our energy this week to fixing this before proceeding with other features of the game. As a result, the gantt chart has been updated. Here is a screenshot of our latest Gantt chart:
Our MVP deadline was pushed back by a week, which means that we are currently 1 week behind schedule. However, we incorporated 1 week of “slack time” in our schedule, so we are still on track to meet the end of semester deadline.
For this week, I was working on testing and verifying the accuracy of the beat map. I mentioned before that I didn’t particularly trust the accuracy of the beat map. While looking at the beat tracking accuracy on the game itself, there was a slight offset and then some issues with the accuracy of the beat tracker. I think this is primarily due to the energy calculation I had instituted.
Originally, I was working with the equation that the scalar multiplier was equal to c = (-.0025714*energy) + 1.5142857
This worked fine on the primarily instrumental audio file. However, with the popular pop songs, I was unable to use this algorithm because the shift in intensities is a lot more varied when looking at the beats. Through testing, I need to find a new variable function for c.
One issue we ran into was that it’s difficult to visualize the beatmaps without the game code itself. The code unfortunately does not work on Windows, so we are having some trouble testing it. That is also reflected on the new gantt chart in our team status report.
Unfortunately, this does put us a few days behind schedule and I may have to implement a simpler melody tracker to compensate, but that will be determined. I created a side testing platform with the Python standard audio processing data, but the visualizer is still more accurate. I can continue work on the scalar c function without it for now.
We are slightly behind schedule, but we should be able to complete the project in time.
As mentioned in my previous status report, this week’s tasks were primarily focused on designing the finalized version of the drum module which incorporated what I learned through the prototype, designing a PCB to house the circuitry within that final module, and drafting the BOM for the components I’d need.
Here are the results of that progress:
The New Drum Module Design:
The EAGLE PCB board:
And the BOM for the EAGLE board and for the FSRs I’ll need:
The main change between these designs and the old prototype module is that I’ve removed the center platform in the drum module. This platform led to some undesired force dissipation along the crossbars, and as such I’m going to instead have the FSR supported through a platform that comes directly through the center of the PCB from the module’s base. Additionally, I’ve trimmed some of the wasted material from the prototype design, which when combined with the removed crossbar should hopefully reduce the printing costs.
The PCB also has LEDs in its 4 corners, which was a post-MVP target we had and which I feel should be readily achievable. I’ve already performed tests of the drum module using some spare LEDs I ad on hand, so I’m confident the ESP32 should be capable of adding this light feedback to the system. The needed holes for these LEDs to stick through have also been added to the drum module design.
Regarding the BOM, all of the components have been ordered. The PCBs are currently expected to arrive this Monday and I picked up the other components (excluding the FSRs) yesterday. I currently plan to test these components on a breadboard this weekend, and then solder them to the PCBs once they arrive this week. Assuming everything goes well, I’ll likely also print the new drum module by the end of this week to test the entire system together.
Regarding how this progress fits into our timeline, I think I’m on good pace with our initial estimations. As I mentioned last week, I feel the place I’m currently at aligns well with where I’d like to be for MVP even though I technically didn’t have the drum count we targeted for MVP (due to the additional 3D printing costs). If everything goes exactly as planned, I’ll likely have some flexible time in the coming weeks, so I’ll hopefully be able to help with improving the C++ code in areas like the game’s main menu.
We’ve hit a few roadblocks this week. My goals for this week were to figure out how to execute Shreya’s Python code from the cpp code, and debug some beatmap timing issues. I didn’t get to either of these. Instead, I worked on debugging some drum input issues and cross platform compilation issues so that we could reach MVP.
Drum Input Issues
Our MVP was scheduled to be completed by this week, but I unexpectedly encountered an issue with the drum input earlier this week. I reprioritized by tasks this week to ensure that we met the MVP. Here is the issue: the drum could connect to the game, but the inputs were being interpreted incorrectly. When the player holds the drum down, the game would think that the player is hitting the drum repeatedly. I debugged and resolved the issue.
Cross Platform Compilation Issues
The game currently does not compile on Windows. I currently am the main contributor to our gameplay code, and I use MacOS. However, we would like Stephen and Shreya to add features to the game as well. We tried to compile the game this week and we encountered many compilation issues on Windows, and I met with them to debug this. I think that I am going to need a solid 3 hours of debugging before I can figure out clearly why the game is not working on their machines.
More Files
I added some starter code for MainMenuMode.cpp, SongSelectionMode.cpp, and ScoreScreenMode.cpp.
Much progress towards reaching MVP has been made since my last status update on 2/26. For starters, I was able to 3D print the prototype drum pad module before spring break began so that I could create the working prototype while away from campus. The process of removing the internal scaffolding was slightly more involved than I originally anticipated, as without having used a 3D printer before I wasn’t aware of just how much scaffolding would be needed for my design. However, I still managed to remove the scaffolding without damaging any features of the module, so it still works as I intended.
After this, I arranged the FSR and other circuitry inside the module and began testing the responsiveness, with somewhat mixed results.
Although the ESP32 was recognizing strong hits to the direct center of the drum pad, hits off the center and to the sides were inconsistent in their recognition, and hits along the crossbars that supported the center platform had basically no recognition. The below diagram details my estimates for the relative accuracies.
The main issue that seemed to be causing this unreliability was the fact that the center platform (where the FSR is located) was at the same height level as the crossbars supporting the platform. This lead to any forces that are on top of the crossbars or close to the crossbars being dissipated through the crossbars rather than through the center platform, which meant the FSR never got a chance to recognize them. To remedy this temporarily, I placed a spare piece of rubber between the FSR and the center platform, which raises the FSR compared to the crossbars. Additionally, I used the adhesive edges of the rubber to fix the drum pad to the module’s rim, holding it in place as it is hit. This creates a pivot directly on the FSR, which redirects much more force through the center platform and greatly increased my recognition accuracy during my trial testing. A diagram illustrating this and the setup for the drum currently can be seen below.
After getting reaffirming results with this new makeshift setup, I began work on the final drum pad module design. This prototype module will be sufficient for MVP, but a new and slightly modified design would better accommodate for this new setup. Additionally, one issue I encountered with this prototype is that the small breadboards I had planned on using were somewhat hard to fit underneath the FSR’s platform, so I would like to design a PCB to go along with that new drum module design.
Designing the PCB and new drum module, as well as creating the BOM/order for them will be my primary tasks this following week.
Regarding how my progress fits into our timeline, I think the progress I’ve made so far has aligned well with our goals. We originally aimed for an MVP target of 3/20, and this progress meets most of our goals. Regarding the functionality, this drum setup has everything we need, as its capable of providing a signal quickly and reliably to the game’s program. The one area my progress doesn’t meet our MVP goals is that I only have one drum as opposed to our MVP target of 2. Creating a second drum wouldn’t be difficult, but I chose not to do so because the current drum design is only a prototype, and I know for certain that I will be changing it later. As such, printing a second prototype module would unnecessarily eat into our budget, as it would likely just get discarded once I began printing the final modules. However, other than that one consideration, I feel like this demo module satisfies our MVP targets and sets me up well to move into the post-MVP stage of the design.
This week, we worked on finishing up the individual parts of our assignments that needed to be completed before we can reach MVP. For George, this meant getting the game to read the hits from the drum apparatus and the JSON from the beat mapper. For Stephen, this meant getting the drum module prototype into a functional state and sending its hit information to the game code in a way that was readable. For Shreya, this meant finishing up the beat tracker and working on sending the beat information to the game code to be integrated.
Ideally, we should be able to hit MVP in the coming week. These components are mostly finished and this weekend, we should be finishing up the integration as well.
We have several challenges/concerns over the next week:
Scheduling – if integration is not finished this weekend, we may push our MVP to Friday of this week instead of Monday. This is not a large issue because MVP would suggest most of our game is done. The gantt chart does not need to be updated yet, but it may need to be next week if we do not meet our deadline.
Accuracy of the beat tracker – because Librosa (the Python standard) is not necessarily as reliable as we originally believed, the accuracy of the beat tracker is still unknown. Shreya’s blog post has more details on the specific problem. However, if it is not as accurate as we need, there will need to be more specific time in testing and tweaking to ensure we get the needed results.
I’ve made lots of progress on this project since my previous update on 2/26.
Here’s a list of the major things I’ve added:
Sound playback
Font rendering
More accurate calculation of score
Reading beatmap data from JSON file
Communication with drum peripheral
To keep everything organized, I’ve separated most of these changes into different files. Some changed files include Sound.cpp, DrumPeripheral.cpp, and Beatmap.cpp. Here is a diagram that shows how the gameplay code is structured:
Playing sounds
I used PortAudio for realtime audio playback, and I used a free library called libaudiodecoder that reads WAV and MP3 files. To play audio with PortAudio, we must write a special callback function that gets executed by PortAudio every time it needs more data to send send to the computer’s sound device. This function is called the “stream callback” and it is run in a separate thread from the main gameplay code. See Sound.cpp for more details.
Font rendering
If we were using a game engine, then this step would be trivial. However, because we are not using an engine, we need to write an interface for rendering fonts to an OpenGL window. Thankfully, I took Computer Game Programming last semester, and rendering fonts was one of our assignments, so I was already familiar with how to do this. I was able to re-use most of the code from that assignment when adding TextRenderer.cpp and TextRenderProgram.cpp.
One issue we currently have is that the fonts are not being shaped correctly. The positions of the letters look a little bit off. I think this is an issue with Harfbuzz, which is a library used for shaping text, but I haven’t spent enough time debugging this to know for sure. I’m going to put this issue aside for now, because it is not critical to the performance of our game, but after we reach MVP I would like to revisit it.
Communication with drum peripheral
I found a simple library called ‘serialib’ that does cross platform serial communication. It’s only 1 header file and 1 source file. The drums microcontroller and the player’s laptop are communicating using the RS-232 Communication Protocol. The drums microcontroller will send a 1-byte message to the player’s computer when one of the drums has a high output voltage. If no drums are being pressed, then no message is sent. See DrumPeripheral.hpp for more implementation details.
Here is a video of what the game currently looks like. In this example, the game plays a test audio file and a test beatmap. The beatmap notes are not synchronized to the music in this example.
Nextsteps
Next week we expect to reach MVP. In our gantt chart, we have our MVP scheduled to be completed by Monday. I think that I will need a couple extra days to reach MVP because I got delayed due to the linking issues earlier in the semester, and the font + sound code took a little longer than expected. My main tasks for next week will be:
Figuring out how to run Python scripts from C++ so that the game can process song data at startup
Verifying that the beatmap notes are synchronized to the music
After this next week, the game should be in a playable state. I’m expecting that this will be the final week where I am working on building the game framework. After this week, I can work on adding features that make the game look and feel fun to experience.
This week and at the tail end of Spring Break, I continued work on the beat tracker. I was able to output a set of time stamps from the beat tracker from the given audio file. I also created the json that holds the information.
Currently, the json assumes that everything will be on the same button. However, this is planned to change post-MVP.
On first glance, the tracker seems to have an alright accuracy. Compared to Librosa’s beat tracker, it is usually with a .2s timer. However, after further analysis of the Librosa beat tracker, I have noticed that their tracker stops about thirty seconds before the end of the audio file. I am unsure if we can use their beat tracker as an accurate count of beats.
The plan for tomorrow and the beginning of next week is to visualize the beat tracking to get a better sense of its accuracy to the song. It probably does need to be tuned and there are a few parameters I can change if it needs to be tuned.
So far, we seem to be mostly on track for having the project done. Fine tuning the beat tracker will need to be done next week with the visualization and then research on melody tracking will likely have to happen.
I feel a few days behind schedule because I don’t have final verification that the beat tracker works, but it should not be too difficult to catch back up
