Posted on

Jackson’s Status Report for 3/6

This week, I spent a lot of time looking into websockets and how they can be integrated with Django. I updated the server on our git repo to work with the “channels” library, a Python websockets interface for use with Django. This required changing the files to behave as an asynchronous server gateway interface (ASGI), rather than the default web server gateway interface (WSGI). The advantage this provides is that the musicians using our application can receive audio data from the server without having to send requests out at the same time. As a result, the latency can be lowered quite a bit.

Additionally, I worked pretty hard on our design review presentation (to be uploaded 3/7), which included a lot more research on the technologies we plan to use. In addition to research on websockets, I looked specifically at existing technology that does what we plan to do. One example is an application called SoundJack, which is essentially an app for voice calling with minimal latency. While it doesn’t deal with recording or editing at all, Ivy and I were able to talk to each other on SoundJack with latency around 60ms, far lower than we thought was possible. It does this by sending tiny packets (default is 512 samples) at a time using a peer-to-peer architecture.

We are still on schedule to finish in time. Per the updated Gantt chart, my main deliverable this week is a functional audio recording and playback interface on the web.

Posted on

Team Status Report for 2/27

Since last week, we have gained familiarity with audio processing on the web, and we know how we’re going to implement all in-browser operations. The biggest remaining challenge is streaming the audio data to and from the server. The most significant risk is that we will not be able to do this in real-time, and thus the recording musicians will not be able to hear each other. As we said last week, this is something we would love to have, but if we can’t figure it out, there are other solutions (e.g. uploading the audio after the whole track is recorded, or asynchronously but in larger chunks than would be workable for real-time). We need to decide on this early on, as it will effect many other facets of our project going in.

We haven’t made any big changes in our implementation, but we’re able to be more specific now about how things are going to get done. For example, the click track can be created using an audio buffer similar to the white noise synthesizer in Jackson’s status report. This way, the clicks can be played to recording musicians the entire time they are recording. Better yet, no server storage is needed for this. A buffer containing a click sound can be looped at a certain tempo (i.e. every quarter note), so only a small buffer is needed.

Since there aren’t any big changes to report, our initial schedule is still valid. That said, it is possible we will get the in-browser recording and click track aspects to work much sooner than anticipated, in which case we’ll have far more time to work on the challenges mentioned above (uploading and downloading audio data chunks in real-time). Research into websockets is still needed, and remains a critical part of our desired implementation.

Posted on

Jackson’s Status Report for 2/27

This week, I familiarized myself with the Web Audio API. This is crucial to our project since it deals with in-browser audio processing, and none of us have experience with that sort of thing. Though I’ve done a lot of audio DSP, I’ve never used the web APIs meant to do that. What I’ve learned is that it’s actually quite simple, and fairly similar to Nyquist, though a lot less elegant. Audio elements such as sound sources, oscillators, filters, gain controls, etc. are all built into your browser already, and all you have to do is route them all properly.

Following a number of different tutorials, I made a few different small in-browser audio applications. I ran them on a Django server I created with a single view pointing to a url, which loads an html file. This html file loads some javascript, which is where I did most of my work. This post contains some code snippets of interest that make up a small portion of code that I wrote this week.

Firstly, I created a small synthesizer, which generates white noise on a button press with a simple for loop:

The channelData is connected to an audio buffer called “buffer”, which is routed to a gain node before being sent to your computer’s audio output. Generating a click track (which will be part of our final product) can be done in a similar way.

The white noise can be filtered in real time by inserting a biquad filter in between the buffer and the gain node. The syntax for this is very simple as well:

With some Web Audio API basics down, I moved on to the main thing our application needs to do in-browser, which is audio recording and playback. I created a small recorder, which saves CD quality audio in chunks (in real time), and after a set amount of time, it stops recording, creates a blob containing the recorded audio, and plays the audio back to the user. This recording interface was my deliverable this week, so I would say we are still on schedule.

For next week, I will integrate this recorder with the Django server we’re using for the final product, and hopefully get the recorded audio sent to the server. More than likely, I will write a new one from scratch, with the information I’ve learned this week.

Posted on

Team Status Report for 2/20

One of the most significant risks in our project is our reliance on WebSockets to deliver packets of audio, and our expectation that Django will work kindly with the packets. We know from experience that Django has built-in measures to handle file uploads in very particular ways (for security purposes), and sending audio data packets with WebSockets to a Django web app server is not something there is very much information about on the internet. To add to this, none of us have prior experience using WebSockets. In order to mitigate this risk, we plan to do some experimentation with WebSockets before we fully specify the design for our project, to determine if this approach is feasible. As a contingency plan, if sockets cannot be used, we know for sure that the audio can be recorded in-browser, and sent afterwards to the server as a complete file in a multipart form-data upload. However there are many drawbacks to this approach. This would require a large file upload at once (CD quality wav audio files get large very quickly), but perhaps more importantly, there is no possibility for audio to be streamed “live” to other musicians or spectators.

The only major structural changes made to our design this week have been narrowing the scope of the project from our initial abstract as we created our project proposal slides. We have set more specific guidelines for the project, which can be found on slide 2 of the proposal slides. Our schedule has not changed, since the proposal slides contain the first draft of our timeline (slides 11 and 12).

Aside from that, we have each done our share of research for our specific areas of the project: Jackson in audio processing and file uploads, Ivy in synchronization and Django, and Christy in UI and audio visualization.

Posted on

Jackson’s Status Report for 2/20

This week, I worked on the Project Proposal slides, I set up the WordPress site with the required pages and formatting, and I worked on familiarizing myself with some technologies we plan to use for our project.

Specifically, I found a JavaScript audio processing interface called Web Audio API, which we plan to use to handle all in-browser audio operations. This includes recording, basic DSP, and some UI components as well like displaying waveforms and spectrograms. I’ve followed a few tutorials on the Web Audio API, since I’m fairly familiar with audio DSP, but not as much with web programming.

In addition to the Web Audio API, I’ve also started experimentation with Python audio processing libraries from this tutorial which will help with any necessary audio manipulation on the server side. Since the main challenges involved in our project are timing-related, server side audio processing will likely not be as important as in-browser manipulation, but some basic processing will of course be necessary.

Right now we are on schedule, though the project is still in the design stage. We need to familiarize ourselves with the specific technologies (like web audio programming in my case) before we can reasonably plan out exactly how the project will be done, and I have made good progress with that.

In the next week, I hope to have some kind of working audio recording app in-browser using the Web Audio API, which can be converted into a Django app for our final project. We also will have received feedback on our project proposal, so we will likely have a more concrete idea of exactly what our project needs to do, and we’ll make any necessary adjustments.