Posted on

Team Status Report for 2/26

Last weekend, all three of our members were working on the finishing touches for our design presentation, which we presented this week. Following that, each of us picked back up the individual tasks we had been working on the week prior.

The details of our individual progress can be found in our individual reports, however we did have some brief meetings over the week to discuss  concerns that would effect the group as a whole.

One of these concerns is the impact of Spring Break on our timeline. Although we accounted for the loss of time Spring Break would inevitably have, we didn’t fully account for non-time related factors such as location. For example, Stephen’s work is physically based, and as such certain tools such as a 3D printer won’t be available to him over the break period. The other major location concern we had was that we wouldn’t be able to meet in person during this break period (as we won’t be on campus). Although this wouldn’t effect our ability to share code segments, this would effect tasks like plugging Stephen’s Drum module into George’s computer for testing the communication between the two.

Luckily, our planned MVP date is approximately a week after spring break, so we can perform our integration during that period. However we needed to plan around the other issues. As such, our current plan is that Stephen should print at least one drum module prior to the break so he can have it for testing over the break and that George will help Stephen install the software needed to run George’s code so that the code can be sent wirelessly during the break if needed.

Other than those concerns and plans, everything is proceeding as planned as each group member is beginning to form the basic framework of their project portion. In Shreya’s case, she has almost completed the instrumental portion of her code, in George’s case, he has developed a basic game GUI and has basic sound functionality, and in Stephen’s case, he almost has his information pipeline from the hardware to software fully functional. As long as we keep on pace and these basic frameworks are completed in the following weeks, we should be able to make our MVP by 3/20.

Posted on

Shreya’s Update 2/26

Last week, I was having issues with determining the beat amplitude (general). I am continuing to use the amplitude threshold-based onset beat detection algorithm. One idea for this algorithm is to determine a subset of the audio file where beat detection is definitely occurring (done using the BPM). On the subset, the algorithm appears to be working alright. Unfortunately, I am still working on setting up the testing sequence, but on first glance, the time stamps from the subset of the audio file created by my script appear to match what I expected it to. One thing is I do believe it is overdetecting some beats, which is due to the nature of the sampling rate of the wav file. I need to figure out how to sort through these detections to get it into a user friendly pattern.

The next steps are to again verify the subset’s beat detection and then attempt to extend it throughout the entire instrumental file. Because this is the instrumental file, further testing will need to occur to understand the limits of the algorithm. We were asked a question during the presentation over what exactly the limits of the algorithm are, so testing for whether it can handle short clips, or long clips, etc will have to begin next week if the algorithm can handle the entire wav file.

Currently, I believe I am still on schedule because the instrumental piece is close to being done. I think we should be okay to reach our goals set on the gantt chart.

Posted on

Stephen’s Status Report for 2/26

The beginning of this week was primarily spent preparing for the design presentation, as we needed to finalize our slides as a group and I personally needed to practice for the presentation itself.

However, I also began testing my various piezoelectric sensors this week so I could narrow down and hopefully select a final component that will go into the final design. Initially, I had three piezoelectric options.

  1. The Flexiforce Piezoelectric Sensor we purchased from SparkFun (Bottom)
  2. Two FSR 400 Series sensors from Interlink Electronics (referred to as IE going forward) that I had left over from a previous project (Middle)
  3. Some cheap handmade Piezoelectric sensors I purchased off Amazon (Top)

After just some quick initial force tests, it was clear that the FlexiForce sensor was the least easy sensor to use. When tapping the sensor with a level of force similar to that of a hit with a drum stick, the resistance hardly changed, and even when applying as much force as I could with my fingers, the change was still only half that of the other two sensors. On the other hand, both the IE FSR and discount Amazon sensors worked much better, with quick and large responses to only light applications of force. However, the Amazon sensors performed poorly in the test I performed before turning on the camera, which gives me the concern they may act inconsistently if used in the final product. As such, I’ve decided to focus all my testing on the IE FSR going forward, as that one seems to be the most promising sensor.

First clip: FlexiForce

Second Clip: Amazon sensors

Third Clip: IE FSR

After selecting my final sensor, I began familiarizing myself with Fusion 360, so that I can 3D print an initial draft of the drum pad housing. Once I have this complete, I’ll be able to combine it with the IE FSR and the Neoprene rubber sheets to create a prototype drum pad, which should greatly aid in future testing by better replicating the expected endcase environment. Here is the module I’ve developed so far using Fusion 360.

The sensor will be fixed to the raised central platform and all the needed supporting circuitry will be located in the cavity underneath it. Additionally, the gap in the base on one side of the module will be used for connecting this module to another module that will house the ESP32 development board. Lastly, we should be able to attach a 3D printed sheet to the bottom of this module after placing the circuitry inside, which should protect the sensitive circuitry inside.

After designing this module, I began working with the ESP32 development board, as this one is slightly different than the ones I’ve used in the past and as such I expected that I’d have some issues getting it to work as my last board did. As I expected, I have been encountering some difficulties but I expect to be able to run simple code by the end of this weekend.

My plans going forward for the rest of the week are to work more with the ESP32 so that going into spring break I’ll be ready to jump right into coding the ESP32’s ADC features. Once I have the ADC features implemented and I have connected it with my sensors I’ll then begin working on sending the ADC information over USB in a form recognizable by George’s program, which should set me up well for reaching MVP by our goal date next month.

Posted on

George’s Status Report for 2/26

This week I worked on adding audio and making improvements to the game UI. Adding audio to the game ended up being much more challenging than I had expected.

Last week I tried to add the libraries irrKlang, SMTL, and miniaudio, but they weren’t working for me. I had trouble linking them them into the project from within my CMakeLists.txt. In my status update for last week, I mentioned that I wanted to try to add the audio library SoLoud to the code base. I tried linking that as well, but it wasn’t working out. Eventually, I came across a sound library called PortAudio, and I was able to successfully add it to the game. I got a small sound demo working. The sound demo sounded terrible, because it played a simple saw wave, but it worked nonetheless.

In addition to adding PortAudio for playing sounds, I also added libaudiodecoder for reading wav and mp3 files.

Also, I updated the game GUI to show notes raining down towards the bottom of the screen.

For next week, I will be synchronizing the audio playback to the note scrolling, and I will also be working on decoding the beatmap JSON files. I would also like to work on detecting whether the player hit the beat at the correct time. I was a little bit behind schedule last week, but I feel like I’ve caught up now. We have from now until March 20 to reach our MVP, which is more than enough time to create a playable demo of the game.

 

Posted on

Team Status Report for 2/19

During the lab work sessions this week we mostly focused on the design review presentation for next week. We worked as a group on the slide deck and we sent the slides to our TA for feedback on Friday. All 3 of us worked independently on our respective aspects of the game – Stephen worked on the hardware, Shreya on signal processing, and George on graphics programming.

One concern we currently have is regarding the piezoelectric sensors, as an issue with their detection accuracy could pose a minor setback. As mitigations to this concern, we ordered several types of piezoelectric sensors in case one fails to meet our expectations and we also plan to develop a small prototype as soon as possible in case another approach needs to be taken. Stephen ordered these parts earlier this week and they arrived on 2/17, so we plan on doing small tech demos with these parts after the design presentation to make sure that they work as intended.

Another risk that we discussed this week was related detecting beats in the input music. We are going to use a Python module called Librosa for detecting the beats per minute (BPM) of the audio, but we are a little unsure to what extent this BPM measurement will help us identify the beats of the song. In this case,  a “beat” is more than just the pulse from the BPM – it also refers to the groove and rhythm of the song. We want to sample the song at regular intervals relative to the BPM of the song, and use these samples to identify important beats. These beats can be used to generate the beatmap, which is the series of notes that rain down towards the player while their song is playing. This week our team discussed strategies for beat tracking and whether we should reduce our target metric of 80% for beat accuracy to make this goal more achievable; but we haven’t reached a conclusion on this yet.

Overall, we are on schedule as a team to have an MVP by the end of March, and there are no updates to out Gantt chart in the past week.

Posted on

George’s Status Report for 2/19

This week I worked on rendering textures into the game. I am using a graphics programming textbook called “Learn OpenGL – Graphics Programming” by Joey de Vries, and I followed along with the chapter on textures this week. I’m using a library called stb that can read image files and place their data into a char array. The game currently loads a test image into the window:

Additionally, I worked on importing sound libraries into our game. However, I was running into some issues with this. I tried downloading and importing Simple and Fast Multimedia Library (SFML) and IrrKlang. It seems like both of these libraries are popular for sound in C++ video games. I am using CMake to compile our game, and for some reason CMake couldn’t link IrrKlang’s dynamic library on MacOS. I spent a few hours debugging this but I wasn’t able to get anything compiled into the game. I found another audio library called SoLoud, and I want to try using this instead.

For next week I want to focus on getting a small playable demo where notes fly down the screen towards the player. I originally had wanted to get font rendering working this week, but I think that for now I will focus on getting a playable demo with sounds. I want to start testing Shreya’s beat mapping algorithm in the game as soon as possible, and having a playable demo is necessary for this. I am a couple days behind schedule – I wanted to have more of the GUI completed before today.  Our current schedule supports a little wiggle room this week because our next task from 2/21-2/28 is just reviewing our final design and making little tweaks; these tweaks won’t take much time (i.e. near 9 hours), so I’ll have extra time to work on the GUI.

 

Posted on

Shreya’s Update for 2/19

At the beginning of this week, I was beginning to experiment with the exporting of time stamps for an instrumental version. In order to determine the desired amplitudes for the beats, I visually determined (from the time-domain representation attached below), the correct peaks of the beats and then searched through the wav file for the timings related to the desired peaks at the time difference per beat determined by the BPM.

Because I hardcoded the desired amplitudes for beats (I used absolute peaks) last week, I was trying to determine an algorithm for getting the amplitude of beats in a particular song. That amplitude would be used to search for the time stamps in a different file. However, this has been slower going. Because I can’t rely on the beats being the highest amplitude and instead, it needs to be primarily based on the BPM. For this week, I have been getting more false positives than actual positives.
My deliverable for next week hopefully will be an accurate amplitude determination for beats (automatic, not hardcoded) integrated into a different file that exports time stamps. So far, I am on schedule for the project as a whole. I have several weeks to finish up an accurate beat tracker.

Posted on

Stephen’s Status Report for 2/19

Last week, I submitted purchase requests for the items in my initial testing BOM so that they could be ordered on Tuesday 2/15 and hopefully arrive on 2/17.  During the stretch from 2/13 – 2/17, I worked on developing a testing plan such that when the components arrived I would already know what tests I needed to perform to assess the quality of these initial parts.

These initial tests can be found here: test_plans_01

I retrieved the components in the afternoon of 2/17 and everything matched my expectations other than the Neoprene rubber sheets. Specifically, the Neoprene seems less rigid than I anticipated, which isn’t a catastrophic or even a particularly large issue, but this does mean that the drum pads should be made thinner than originally planned to avoid too much loss of force as the pads compress. To compensate, I now plan on placing a rigid material (such as plastic) under the neoprene to create extra drum thickness without losing much more force.

On 2/18, I spent most of the day editing the design presentation slides that pertained to the hardware sections of the presentation so that we could receive feedback over the weekend. Unfortunately, this took longer than expected and so I didn’t have time to perform the tests yesterday (2/18). My current plan is to perform these tests later today (2/19), and if I finish early I may be able to edit this report before midnight.

For the next week, I’d like to finish all outstanding tests I have planned, develop and perform any new tests I deem necessary based on the results of those tests, and begin designing a 3D model of the drum pad prototype using Fusion 360.

Posted on

Stephen’s Status Report for 2/12

The weekend of 2/5 and 2/6 was primarily spent revising the proposal presentation based on feedback we had received during the week before that. Afterwards from 2/7 through 2/12,  I researched component options and drafted a Bill of Materials (BOM) for our first round of purchases.

(Sorry the image is blurry. I’ve attached the Excel sheet at the end of this post for better clarity)

The five items in this BOM are a piezoresistive force sensor, op-amps (for use with the sensor), a set of cheaper piezoresistive sensors (as a back-up precaution in case the higher quality sensor is damaged), neoprene rubber sheets, and an ESP32 development board. Altogether, these components will form the initial drum pad prototype we’ll need to assess the difficulty of converting forces into electrical signals. If everything goes well with these initial tests, it will even become possible to begin programming the ESP32 to receive these signals and communicate them to a PC, which would allow us to test the software side of this project more easily.

One important detail to note here is the fact that the rubber sheets are made of neoprene. From my research, neoprene is a quality elastic substance which should produce the bounce we’ll need for our drum pads, unlike other synthetic rubbers like polynorbornene. I would’ve liked to order some natural rubber as well to compare the two options, but natural rubber is far more expensive than neoprene and we wouldn’t be able to afford natural rubber for all our drum pads while remaining in our $600 budget.

Regarding ordering/shipping dates, this research unfortunately took longer than expected and so I wasn’t able to submit this order by Thursday (2/10) and as such it will go out this next Tuesday instead (2/15). This is suboptimal scheduling-wise, so to mitigate this loss of time I opted for the quicker shipping options of each product not available through Amazon and I currently plan on developing my testing plans for these components before their arrival so that I’ll be ready to jump directly into testing once they arrive (expected arrival is 2/17). By the next update (2/19) I hope to have these components within my possession and have collected data on the force sensitivity of the sensor.

Here is the BOM in excel form: BOM_18500

Posted on

Team Status Report for 2/12

The first part of this week was dedicated to the proposal presentation. Together, we determined the user requirements and the design requirements, as well as general ideas for implementation. Our presentation was given on Monday. We have also attached the slide deck to the website for perusal.  Overall, as long as we can maintain some closeness to schedule, we feel that this is a viable project.

Here’s a sketch of our current project design:

Currently, we are on schedule to complete our design by the deadline with one week of slack time. In the next week, we are looking to begin to determine the final packages and equipment we will be using for this project for all three portions of the project. There are several risks associated with this project that we are looking to mitigate at the moment:

  1. Determining which packages we need to use for beat tracking. With our latency constraints, we are worried about the usage of base packages like numpy to iterate through the song, which was explained more in the personal sections.
  2. Slow delivery of the hardware components. Given that they are currently expected to arrive on 2/17 (with 2-day shipping), there will only be 2 days before the next status update to assess the feasibility of our hardware plans and decide if we need to make any changes going forward. As such, we’ve ordered several different piezoresistive sensors as a precaution against this, as if our currently targeted sensor is not satisfactory for our needs, we’ll have a backup set of piezoresistors to allow us to continue testing while we spec out a preferable alternative (assuming the backups are also not satisfactory). 

No changes have occurred to the design project as of yet. The schedule will remain consistent with the next weeks being dedicated to determining if our individual solutions are viable as well as determining the final packages we want to use in our design project.

Here is the current version of our Gantt chart: