Category: Ivy’s Status Reports

This week I worked on final integrations and testing our web app. Since Christy figured out how to get librosa on the cloud, I’ve reverted back to using that for syncing. One of the changes I had to make now was to have the syncing iterate through all the files attributed to the group , as we’ve change our project to let all group members upload their recorded audio first, so that all tracks could be displayed on the webpage for users to listen to first before deciding which ones to keep and sync.

Other than that, what’s left is allowing users to download their finished recordings off the webpage. I’d like to have an option of merging all tracks down to one audio file, but right now it seems easiest to download all tracks as separate audio files. Finally, I’m putting together a tutorial about how to use our website. After conferring with some test users, it seems the recording can be a little complicated, so I think it would be nice to have an instruction manual of sorts. In the next few days, I will be working on our poster. I’d like to have it our final presentation materials done before Monday as I have a file on that day :’)

This week I worked some more on integrating our individual parts in preparation for the final presentation and demo video. In addition to that, I worked on our final presentation, updating our system diagram to reflect our changes.

The last thing we have left to do is cloud deployment. On Friday, Christy and I met on campus to iron out some last minute details and get our project deployed on AWS. One of the issues she ran into while trying to deploy our project was that the python Librosa library I used for syncing the audio tracks will not work with deployment. From what I’ve found on the web, it does not seem like there is a lot of resources for solving this issue. Instead, I will rewrite that portion of my code using a different library, essentia. Essentia’s documentation describes explicitly how to compile on the web, so this should get rid of the error.

We are meeting again on Sunday so that we may get cloud deployment done by Monday. Afterwards we will test the latency by connecting with users at different locations

This week, I worked on trying to upload the recorded audio in a file format recognized by python. Chrome does not support recording in a .wav format (as a matter of fact, the only format that seems to be supported across all browers is webm), so we have to do this ourselves. Attempts involved trying to write the audio data into a wav file in the back end (which just resulted in noise), and trying to convert the recorded audio blob into a .wav file before passing it to the server.

After some research, I found this tutorial, which shows how write our own WAV header before uploading the audio to the server. Since webm does work with PCM encoding, appending our recorded audio to a WAV header seems to be the right way to go. However, after trying it, I’m still getting errors trying to read the file in the back end. I think the problem is we need to specify our sample rate and bit depth before recording and am currently looking into how to set that up.

Though I have the syncing post-upload done, not being able to get the audio in a readable format makes the entire functionality moot. I am behind right now, but I hope to get all this figured out before Monday, so we can update our gantt chart and get ready for the demo.

I am almost finished up with the audio upload to server right now. This past couple weeks, I realized my implementation discussed in my previous status report was impractical, as I was creating and playing the click track on the server rather than on the actual webpage. To fix this, I had to rewrite my code in .js, using WebAudio API to create the metronome clicks. Unfortunately, I was unable to replicate the clock I had created in Python in Javascript, and instead resorted to recursive TimeOut calls for the intervals between the clicks. But this implementation will create inevitable delay, which would causes successive ticks to drift further and further away from the ‘correct’ timing. To fix this, I would decrease the intervals for every other tick, to make up for time if the previous tick arrived a few ms late. I don’t like this solution too much as it only fixes the delay after it happens, rather than addressing it head on. But, for the range of tempo we’re aiming for, it seems the problem isn’t too exacerbated. If we have more time at the end, I will look to see if there is another, more accurate solution.

I think our groups biggest concern now that Jackson’s figured out how to implement monitoring is the UI. I don’t really have much experience with HTML outside of basic social media layouts and our proposed plan for it is much more involved than just a static webpage with some buttons.

This week I finished implementing the click track. I settled on creating the click track in python. To do this I used the playsound module and the sleep command to play a short .wav file after a delay based on the beats per measure.

This raised some issues however, as the sleep command is not very accurate.  While testing,  I found that the beats consistently had up to 150 ms of delay. To improve upon this, I created a separate clock which I could initialize to run based on the imputed tempo.

The meter/beats per measure dependent click track ui was much harder than I thought. I only knew some basic HTML going into this so it took a while to figure out how to fetch variables from other elements on the webpage. Even now I’m not so sure it’ll fit with the rest of the UI; since I’m unsure of the dimensions of our actual site, I made it out of <div>s. I’m a little behind right now, as I have yet to merge my code with the current version on the github, but I will get it done by or after our lab meeting on Monday (should I end up with questions), and thus will begin working on the track synchronization by then.

Our biggest concern is the networking aspect of our project. We are not too knowledgeable about networking, and as the concern was raised to us during the Design Review presentation, we aren’t too sure if our proposed socket solution will even meet our requirements.

 

 

 

This week, I worked on the design review presentation with the rest of the team. I created this tentative design of the UI for our web app’s main page, where users will be recording and editing their music together.

In the beginning of the week, Jackson and I tested out SoundJack, and found we could communicate with one another with the latency of 60ms through it. This was much better than either of us were expecting, so using this method (adjusting the packet size to increase speed and amount of packets to be sent/received to increase audio quality) as a basis for our user-to-user connection seems to be a good idea. But instead of manual adjustments, which can become really complicated with more than two people, I will be creating an automatic function that takes into account of all the users’ connectivity, and set the buffer parameters based on that.

We have settled a major concern of our project, as we will be reducing the real-time latency so that users will be able to hear each other and synchronizing their recording afterwards. We have updated our gantt chart to reflect this.

My first task will be to create the click track generator. To begin, I created a CSS form which will send the beats per measure, beat value, and tempo variables to the server when the user sets them and clicks on the ‘test play’ button. A function will then generate a looped audio sound with this information and play it back to the user. As for the latter, I’m still not too sure whether the sound should be created with Python DSP Library or the Web Audio API. Further research is needed, but I imagine both implementations will not be too different, so I should be able to get the click track generator  functioning by 3/9, the planned due date for this deliverable.

This week I presented our project proposal as well as did further research into the synchronization issue. This remains my biggest concern: being able to synch real time vs only synching after the tracks are recorded will greatly affect how our project is constructed. I want to know the advantages and technological limits of implementing both of them asap, so we can decide on which one to focus on moving forward.

In saying this, I’ve found partial solution in the web app, SoundJack. The application can control the speed and number of samples that are sent over to other users which allow users to have some control over the latency and greatly stabilize their connection. It calculates the displays the latency to the user, so they may make the appropriate adjustments to decrease it. Users then can set multiple channels to mics and chose what audio to send to each other via buses.

One coincidental advantage of this is that, because we will be taking care to reduce latency during recording, the finished tracks will not need much adjustments to be completely on-beat. Still, where this solution falls short is that the latency will either have to be compounded with multiple users in order for real time to keep up with digital time, or other users will here an ‘echo’ of themselves playing. Additionally, the interface of all the programs (SoundJack, Audiomovers) I’ve looked into is pretty complicated and hard to understand. One common complaint I’ve seen in comments from YouTube guides is that it makes sound recording more engineering focused than music-making focused.  Perhaps our algorithm could do these speed and sample adjustments automatically, to take the burden off of the user.

Furthermore, in these video guides, the users use some sort of hardware device so that they are not reliant on wifi connection, like what our project assumes they will be doing in. So far, I’ve only read documentation and watched video guides of this. Since it is a free software, I want to experiment with this in our lab session on Monday and Wednesday.

I completed the Django tutorial and have started on metronome portion of the project. I have a good idea what I want this part of our project to look like, however I have less of an idea of what exactly the metronome’s output should be. One thing I know for sure is that, in order to mesh with our synchronization output, there needs to be a defined characteristic in the wave where the beat begins. I also think that, because some people may prefer to have a distinguishing beat at the beginning of measures, we need to take that into account when synchronizing.

During the week, we worked on the Project Proposal Presentation as a team, as well as set up the wordpress and gantt chart to layout our workflow and track our process.

Furthermore, I wrote up an outline for the oral presentation of our proposal here. One of the major concerns we have going into this project is figuring out a method to synchronize individual tracks. In our initial research, we’ve come across some papers (Carnegie Mellon Laptop Orchestra, Dannenberg 2007) and programs (rewire, audiomovers) that aim to do something similar. The former gives us an idea of how to sync performances to a click track, but we hope to sync performances as they are being played live as well. I will look further into the commercially available options this weekend.

We will be using Django to build our website. Since I have not used that before, I’ve been following a small tutorial that’ll hopefully get me familiarized with its functionality and interface.

Our design hasn’t changed much from our abstract, but we’ve added some more methods of testing our final products viability, including testing for security, website traffic, and qualitative feedback from its users.