Author: tianyue2

• What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk right now is the integration process, and the most important link is the radio transmission between gloves and the receiving module. If we met problem with the radio transmission, we will go to search for online tutorial and follow the instruction to solve the problem. We might also ask our TA for help. We don’t think we have any backup plan if it doesn’t work. As a contingency plan, we will have to resort to a wired connection to send angle information from the gloves to the receiving module.

• Were any changes made to the existing design of the system? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There are no design changes to the system. 

• Provide an updated schedule if changes have occurred.

So far there is no major update on our current schedule. Although our DigiKey order hasn’t arrived yet, the major components have arrived (radio and gyroscope IC). Hence, our design process is still on schedule.

• Component you got working.

You can now control the the volume of the synthesizer with the pitch angle of the gyroscope chip. Here is a demo video.

• What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk right now is the process of incorporating the pre-trained models into our system. Although we have accounted for the possibility that these models don’t fit our requirements and came up with backup models, it is still possible that the integration method that we planned might not work as intended. This brings up another potential issue: the future integration process. Since all of our team members worked on the project during spring break separately, there is not as much communication as if we were to sit inside the classroom. This could potentially cause the integration process to be more complicated than expected. Currently, we are planning to meet more often in our development process in order to better understand each other’s work, and more research is being done at the moment to reduce the confusion in this process.

• Were any changes made to the existing design of the system? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There are no design changes to the system yet. We did change a metric, the weight of the system is now 4.8 lbs instead of 6 because this number is more reasonable and should better fit user requirements.

• Provide an updated schedule if changes have occurred.

So far there is no major update on our current schedule, several tasks are slightly extended due to spring break, but will not impact the timeline of future items.

• Component you got working.

The gesture-command UI now works with some dummy gestures, it allows matching and save + load the data when python is launched. The synthesizer also supports gliding between notes and customized timbre (demo video).

• Addressing faculty feedback on design presentation

Most of the questions are addressed in the design report, and the rest will be addressed below.

“Use of glove and what it will control and how? How does the battery on one hand connect to the other?”

Each glove mounts an individual circuit that contains 2 LiPO batteries, an Arduino Nano, a radio transmitter, and a gyroscope chip. Since each glove is individually powered, there is no physical connection between them. The left hand’s pitch angle will be mapped to volume in the synthesizer, where the right hand’s roll angle will be mapped to pitch bend.

“Why no Windows?”

After changing some python libraries, our software now runs both on Windows and macOS.

“Details about the synthesizer”

Below is a more detailed diagram of the synthesizer.

1. The frequency f is based on the note selected by the user’s hand position on screen and the amount of pitch bend.
2. Oscillate the waveform given by the oscillator at the frequency f.
3. Samples the oscillating waveform into buffer and multiply it with the volume parameter determined by user’s left hand.
4. Write the buffer to audio stream.

• What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

After resolving the risks with our software last week, we believe that our most significant risks lie within our hardware. Because the quality of ICs (especially the tilt sensor and the radio chip) is not guaranteed, we will start component testing immediately after our chips arrive. If there is a defect, we will order another one (maybe from another seller) within one day.

 

• Were any changes made to the existing design of the system? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We did change the battery for our gloves from alkaline batteries to rechargeable lipo batteries. Lipo batteries are lighter and have a power density, which will in return make our system lighter and last longer. It will also cut down our development costs as we don’t have to purchase batteries all the time. To address the possible (although unlikely) overheating of the battery, we will monitor the power consumption and temperature of the lipo batteries as we develop the system. We will also add an insulator layer to our glove if necessary.

 

• Provide an updated schedule if changes have occurred.

There was no change to our schedule.

 

• Component that we got to work.

Oscar continued working on the synthesizer and now it supports real time pitch and volume control and timbre selection. Here is a short demo video.

 

• Adjustment to team work to fill in gaps related to new design challenge

Although Oscar was in charge of developing the synthesizer, he encountered a major sound card issue that couldn’t be resolved when developing on Ubuntu. He then switched to Windows and wrote everything from scratch using new libraries. In order to make sure the resulting product is still macOS compatible, Karen and Yuqi (both owning a MacBook) both jumped in and conducted a preliminary setup experiment. After changing the code a bit and installing the correct version of libraries, our synthesizer is now both Windows and macOS compatible.

 

 

Team Reports for 2/18

• What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

We still believe the most significant risk is the hand gesture + movement detection system. As a result, we decided to use a demux structure so that our program will either perform gesture recognition or hand tracking. The video feed will be fed into two time-multiplexed models, one for gesture recognition and one for hand tracking. Splitting the functionality allows us to access more pretrained models and hand tracking program, which greatly reduces the development time. An abstract block diagram is shown below:

• Were any changes made to the existing design of the system? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Instead of training our own gesture recognition model we decide to use a pre-existing one because of time constraints. Since we are not at the step yet this design change did not create any extra cost for us. We also included backup options for most of the parts that we plan to write on our own in case something goes wrong. We believe that risk mitigation is necessary and it is always better to have a plan b on everything.

 

• Provide an updated schedule if changes have occurred.

There was no change to our schedule. After factoring spring break and other outside events, the schedule is still reasonable for everyone.

 

• Component that we got to work.

Oscar wrote a basic, cursor-controlled synthesizer that supports real-time playing of 2 notes. The frequency value of those notes were generated lazily using iterator functions and OOP.

 

• The “keep it simple” principle

The major decision we had last week was to split the gesture control part and hand tracking part into two time-multiplexed programs. That way, we can easily develop the two parts concurrently and make best use of any pretrained/existing programs. There are two modes of our instrument, a sound selection mode (where the user can use gestures to select a particular timbre), and a play mode (where the user plays the instrument with hand positions). We will also include a software demux that allows users to freely switch between these two modes.

The most significant risk is the hand gesture/movement detection system. Since we plan to use our own data set and train our network, it could be more time-consuming than expected. As a contingency plan, we found two pre-trained gesture recognition models online: Real-time-GesRec and hand-gesture-recognition-mediapipe. However, these models do support more gestures, so we will only use a subset that maximizes the detection accuracy.

So far there are no changes made to the existing general design of the system. However, we did introduce a plan B to some of the subparts in the system. For example,  we are considering the possibility of using a pre-trained model in case we are not able to collect enough data in a short period of time and train our own. This a necessary precaution to ensure that a problem in one part of the process would not cause the overall hand gesture recognition component of the system to not function properly.

No updates to the current schedule have been created. Currently, we are still in the process of designing our implementation and finalizing design details.

Although we are still in the research and design stage, we did come across a DIY Python synthesizer tutorial that actually mimics the theremin quite well.

Our project includes environmental and economic considerations. We added an autonomous lighting system so that our instrument can function in a very dim environment. We also want our end product to be significantly cheaper than theremin on the market, which limits our budget to be under $400 dollars.