This week our backend underwent a refactoring due to limitations surrounding certain elements of the frontend that we wanted to use, and also due to latency concerns. Besides that, I spent time this week finishing up the filter select menu (backend and frontend) as well as testing the code for latency.
Jason’s Status Report 8
This week, I finished calibrating all our motion-controlled effects. Then, I worked with Jeffrey to integrate our frontend with real-time effects updating from gyroscope values. Next week, I will work with Jeff further to refine our frontend and with Michael to implement the instrument select with controller values.
Michael’s Status Report 8
This week, I helped Jason in implementing real-time gyroscope controlled sound effects and also continued my work with integrating the code. During integration, I started with the backend part of the controller portion of our MVC in Python, but due to how fast the code was expanding/getting more complicated, we decided to refactor the code into a clean design pattern. The main flow of our projects operation would be between two main modes: play and edit mode. In play mode, which is the default mode, the VR inputs are processed and used for sound output and effects toggling. In edit mode, VR controller inputs are used in navigating a series of menus used for selecting instruments, changing filters, adjusting filter parameters etc., and Jeffrey is working on the frontend such that the user can directly interface with the website using their remotes. At the moment, I am continuing work on the menu for selecting and changing the filter sets.
Michael’s Status Report 7
This week our team decided to pivot away from using peak detection for note separation and to utilize different sensor data mapped to certain effects instead. As a result, I’ve spent this week working on setting up the gyroscope data to be processed by the tone library. Besides that, I have been working with Jason on implementing effects that have tunable parameters that we can map to motion data.
Jason’s Status Report 7
This week, I focused on improving existing functionality with Tone.js. I successfully incorporated samples into the playing, so now there is a much greater variety of sounds Caprice is able to produce (piano, strings, etc. as long as a sample pack is provided). Additionally, we also began to brainstorm ways to control certain effects with the motion controller, and a very promising one so far is the 3D panner, which simulates the sound output in a 3D setting (ie. move to the right/left, closer/away, change orientation, etc.). This effect is very interactive and couples well with the gyroscopic data that we can get from the VR controller.
Jason’s Status Report 6
This week, I worked with Jeffrey to integrate the smartphone app with the Flask backend. After integrating the submodules, we now can play sounds on the Javascript frontend, with notes controlled by the smartphone. However, the latency initially was very slow and jittery, averaging around 150ms but very inconsistent. We discovered that the source of the latency comes from the socket connection between the smartphone and the Flask server, and tried several different socketing methods to try and resolve this issue.
The best solution we’ve found so far was to use an interval-based socketing mechanism, sending 30 updates/sec =~ 33ms between each update. This reduced the latency significantly, but the jitter is still an issue, and we are still unsure where the root of the problem lies (might be from internet load from other applications/people in my apartment?).
Besides that, I also added functionality in Tone.js to activate/deactivate effects and filters. Next week, we will work on integrating octave shifts and pitch shifts using the touchpad.
Michael’s Status Report 6
This week I continued work with integrating my VR controller code with the sound output code and have been continuing with tuning the gesture detection. In regards to integration, I am now able to trigger sounds using just the VR controller–as of now, sounds are hardcoded and will be more usable once integrated with the smartphone. I found that latency between controller to audio was not as good as we wanted, but after trials with Jeffrey and Jason’s setups, discovered that the latency was due to my laptop itself. They measured the latency on their systems to be around 40 ms, which was within our requirements.
Jason’s Status Report 5
This week, I worked on integrating the javascript Tone.js submodule with the Flask server. Data is transmitted by sending a json string through a socket, and the json contains note information, when to activate note separation, effects toggle controls/parameters, and touchpad swipe data.
The Python backend can now communicate with Tone.js, play notes, set up effects, and configure effects settings in real time.
Next week, I will work with Jeffrey to integrate the smartphone app with Flask, hopefully with time to conduct our first end-to-end latency tests.
Jeffrey’s Status Report 5
For this week, I focused on solely the phone side of our system. We had proved that it was possible to send signals from the phone to the Flask server through sockets. Now, I built off of that by creating a UI to represent 8 buttons the left-hand fingers would press. Each of these buttons corresponds to a command from 1-8 that is received by the Flask server. I also implemented a UI for changing the key that the current user is playing in. This status is also kept in sync with what the server knows. Along with UI, I also experimented with the accelerometer on the iPhone using the Expo accelerometer package. The API is easily accessible and can be customized by things such as update speed. Will plan to apply peak detection algo Michael worked on to detect bends in the left hand. The goal would be to match these bends to pitch bends generated from garage band.
Team Status Report
Updated Gantt Chart
https://docs.google.com/spreadsheets/d/1f5ktulxfieisyMuqV76F8fRJTVERRSck8v7_xr6ND80/edit?usp=sharing
Control Process
Selecting Notes
The desired note can be selected from the smartphone controller by holding a finger down on the note grid. The eight notes represented on the grid are solfege representations of whatsoever key and tonality (major/natural minor) selected from the smartphone menu. For example, a selection of the key of C Major would involve a grid with the following notes: C, D, E, F, G, A, B, C.
Octave Shift
To shift the range of notes in the smartphone note grid up or down an octave, swipe the thumb right or left on the touchpad of the VR controller (in the right hand). Swiping right once would denote a single octave shift up, and swiping left would denote an octave shift down.
Chromatic Shift
To select a note that may not be in the selected key and tonality, the user can utilize the chromatic shift function. This is done by holding your right thumb on the top or bottom of the VR controller touchpad (without clicking down). Holding up would denote a half step shift up, and holding down would denote a half step shift down. For example, playing an E-flat in the key of C Major would involve selecting the “E” note in the left hand and holding the thumb down on the right hand touchpad. The same note can also be achieved by selecting “D” and holding the thumb up on the touchpad.
Triggering a Note
To trigger the start of a selected note, pull and hold down the trigger on the VR controller. The selected note will be played for as long as the trigger is held down, and any additional notes toggled on the left hand will be triggered as the note is selected in the left hand. If no notes are selected while the trigger is pulled, no sounds will be outputted.
Note Separation
If the user wishes to divide a selected note into smaller time divisions, there are two options:
- Release and toggle the trigger repeatedly
- Use motion to denote time divisions
The system recognizes a change in controller direction as a time division. For example, to subdivide held note(s) into two subdivisions, one would initialize the note with a trigger press and initiate the subdivision with a change in controller motion. The same outcome can be accomplished with just repeatedly pulling the trigger.
Polyphony
Polyphony can be simply achieved by holding down multiple notes on the smartphone grid while the trigger is pressed.
Toggling Effects
Four different effects can be toggled by clicking on any of the four cardinal directions on the VR controller touchpad.
Updated Risk Management
In response to the current virtual classroom situation and progress with our project, the risk has changed a bit.
For the phone grip, it doesn’t seem very feasible to build it in the same manner/design we had originally intended. We had planned to build the grip with a combination of laser cutters and maker space materials. Instead, we have decided to go for a simpler approach for attaching the phone to the users left hand. Instead, we want to use a velcro strap that would go around the user’s hand. We would then have the other end of the velcro attached to the back of the phone.
Another area of risk we found was the limitations of the current Javascript package we are using to generate sound. While there are many features in the library such as accurate pitch selection and instrument selection, there were some features we didn’t see. One of these features we wanted to use was the ability to pitch bend. A workaround we have brainstormed is to use a Python bending that does support pitch bending library. We could run this in parallel on our Flask server with the front end Javascript to achieve features we want from both libraries.