Demo Video
Jeffrey’s Status Report 9
Flushed out the frontend much better as well cleaned up the backend to handle more menus in a more scalable, general way. Added two other menus (key selection and filter selection). For key selection, there are 8 keys and 8 modes the user can pick from in edit mode. I also debugged some of the problems we were having with edit/play mode interactions. Users can now swipe up/down/left/right in edit mode to pick what they want to edit. We looked into some of the latency problems and addressed a few but there is still a base latency built into sockets but have reduced it down to about 15ms on each direction of sending data. We also realized that the frequency of data transmitted out of the controller may influence the general latency but that is something we can’t control.
Jeffrey’s Status Report 8
Created a working phone application with the existing backend. Worked out exactly how we would send data from backend to frontend to keep everything in sync. Decided on the 4 menus we would allow the user the edit when they are in edit mode. Was able to develop several menus including instrument changing and parameter changing for certain effects. To maintain the same state on the front end back end for these editing menus, we decided to have the changes occur in the backend, and on any update to from the controller, that would have to be reflected in the frontend. To reflect this, a socket was set up for each menu and any changes (Ex. current selected instrument) were made by editing the CSS classes/ids responsively.
Jeffrey’s Status Report 7
This week, I helped to finish combining our separate work into several functions on the python server. Got touch/press detection to work on the touchpad so that the user could go up and down half a pitch by pressing up or down on the touchpad. We decided to use this over swipes since these were more responsive than swiping. I also got octave changing to work where the user would touch left or right on the touchpad to go up/down half an octave. We also looked into the problem of latency and found that it was still an issue but isolated the problem to solely the internet that the phone is connected to. I also looked into ways to improve the UI/UX for the application on the mobile and laptop side.
Jeffrey’s Status Report 6
This week, I completed the integration of the phone with the current backend workflow for handling sound generation. With 8 buttons, each button sends a certain command to the Flask server using sockets Along with sending certain notes, I also brainstormed ways of allowing the user to change octaves. We determined the best way to do this could be to use the right-hand swipe motions that have already been working. Then to send this data to the Flask server and then to the phone. The keys would then change by shifting up or down based on which direction the swipe was made. For example, if the user wanted to go half an octave up, the bottom 4 notes would shift up an octave while the top 4 would stay the same. This would act as a sliding window for the notes to go up and down half octaves. We also discovered there was quite a bit of latency in the sending of sockets from the phone to the laptop. We found the average to be around 100ms after many trials and are currently working towards solutions to lower this latency down. We also found the latency to be around 40ms on average between the remote and Flask server.
Jeffrey’s Status Report 1
For this past week, I attempted to understand how we would connect to the Gear VR remote controller. We had several options to consider that each had specific tradeoffs. The controller could either be connected to the laptop and have sensor data be streamed there or we could have the controller stream data directly to the phone and eliminate the middle man (laptop). Initially from our research, we found someone that was able to successfully connect to the controller to a laptop. However, this was through the Web Bluetooth API and at the time seemed restricting since we would have to make our application on the web. For this reason, Michael decided to tackle the task of trying to connect the VR remote to a python program so that we would have access to this data from within python.
I focused on a different approach by trying to figure out how to connect the remote to the phone directly without having a laptop involved. In this case, the remote would act as the Peripheral and the phone would act as the Central instead of the laptop. We considered this approach because it would be a much more user-friendly experience to only have 2 components to use our application instead of having 3.
I initially started by looking into ways to develop for the phone and found React Native to be the most compatible and easy way to develop for either iPhone or Android. I started by looking into the best platforms to develop with React Native and found Expo.io. However, after trying to install the Bluetooth4 packages, I realized they were not compatible so instead decided to build a React Native application from scratch. After setting up the project, I tried two different Bluetooth packages: react-native-ble-manager and react-native-ble-plx. For react-native-ble-plx, I was able to successfully connect to the Gear VR but was unable to send notifications. I then tried the other package and was able to successfully connect and receive sensor data but was unable to establish a stable connection.