Category: Team Status Reports

What are the most significant risks that could jeopardize the success of the
project? How are these risks being managed? What contingency plans are ready?

HW: I am a little scared that with all of the testing, we have been doing that our solenoids will burn out (from “too much” use). However, we have some extra solenoids and have been using MIDI in place of the solenoids for testing at home.

SW (CV): I think the biggest risk at this point is integration between mine and Lances part, but it should be fine. We still have a working MVP either way.

SW (Music): Serial pipeline failure. It’s robust and can accurately handle errors, but during recent testing with a robotics capstone project it has had some strange errors where it randomly disconnects (from a Raspberry Pi). A disconnect during our demo wouldn’t be the end of the world, but it would be annoying, especially since both programs would need to be restarted. Integration isn’t much of a concern in my opinion. I just need to know what values to expect on the Python side, and I just need pins on the Arduino side.

Were any changes made to the existing design of the system (requirements,
block diagram, system spec, etc)? Why was this change necessary, what costs
does the change incur, and how will these costs be mitigated going forward?

HW: N/A — but if the solenoids were to burn out, then we will be using MIDI keys as a replacement.

SW (CV): Working on symbol detection, as mentioned before, but thinking of combining it with color detection as described in my post.

SW (Music): Not really a huge design change, but I’m no longer representing music in 32nd notes. Instead I’ve jumped down to 8th notes so that we can play at higher tempos and jump through more chord progressions quicker. This will lead to a smoother user experience in generative mode as well.

Provide an updated schedule if changes have occurred.

HW: N/A

SW: N/A

SW (Music): N/A

This is also the place to put some photos of your progress or to brag about a
component you got working

HW: N/A

 

(Extra question for this week) List all unit tests and overall system test carried out for experimentation of the system. List any findings and design changes made from your analysis of test results and other data obtained from the experimentation.

HW: when building the circuit, I tested individual solenoids to ensure that they were connected correctly and tested them with the rest of the system to ensure that they were connected/integrated correctly with the entire system. Some of the unit tests I conducted were: just turning on one solenoid at a time, randomizing the arduino keys and playing at most 4 at a time. For “old” solenoids, I noticed that when more than one key was playing concurrently, there was a bit of lagging time, which I roughly timed it as < 100ms — this latency issue was purely from the age of the solenoids. As for the general latency between the arduino code and the solenoids, 40ms latency was observed — which was not significant in our system. I am hoping to measure the latency between the user playing the key using our object-detection system to the actuators playing the keyboard this week — however, in our informal testing, the latency was approximately 150ms (this was measure as soon as we saw in the print statement that a certain key was detected — however, there is definitely some latency in reading/detecting the object itself).

SW(CV): One test I needed to run was the time from when the user puts the color into the right box and when the code recognizes it. This is a test that I expect to change when I am finished with the final implementation of combining symbol detection + color detection, but I doubt it will change very much. I measured about 600 ms on average, which is not bad, but I do think it will change by demo time. The other test I ran was the rate of detection. This can be negatively affected by other objects that have a red hue that the camera picks up. I measured it to be accurate about 92% of the time. However, I do expect to get a better value when I fully integrate the symbol detection in.

SW (Music): I’ve done a lot of unit testing for both music and serial code. For music, my main concern was the generative mode. There’s only one thing to test, and that’s the generation output. I use normal distributions to pick notes and rhythms, so what I did was just generate a bunch of melodies and checked to see if they made sense with the progressions that they were over. This was mostly opinion based, so I just vibe checked all of them. I also checked to make sure that the rhythms were reasonable. I wasn’t really satisfied with those results, but since I’m switched to a slower model, I have a lot of chances to tweak generation probabilities. I did do some brief tests on generating rests, but honestly it just wasn’t really worth it. Notes naturally die out anyways, so there’s not much point in actively “playing” the rests (even though good musicians should do that!) since our project is more geared towards simple things. For serial/Arduino, a huge majority of my tests were done during the heat of debugging, so I don’t really remember them exactly. However, it was things like “stepping through each and every byte I’m supposed to send using the Serial Monitor” and “sending a single byte and tracking every location it goes to.” This evolved into tests with Python interfacing like “sending a sequence of data and having the Arduino report success back” which inevitably evolved to “sending the same string of data and then reading in as much data as humanly possible from the Arduino so that I can figure out where my bytes are going.” Then once the pipeline worked (or was at least half-functional) I started sending melodies off to the Arduino, then chords. For the majority of the semester, I haven’t had any solenoids to test with, so at first I started off with just having the Arduino report what notes it was playing. Then, after Professor Sullivan recommended trying out MIDI output as well, I remembered a MIDIUSB library I briefly used when trying to build an electric vibraphone a few years ago. I shook the dust off of that and made it so that I could have the Arduino interface with FL Studio, and I started testing notes and melodies that way. FL Studio doesn’t really ensure that circuit integration will go well though, so I also set up a simple LED circuit that I could use to test melodies on. It’s currently 7 LEDs, and I was going to extend it to 14, but since we only have about 10 solenoids, I’ll most likely just stick with the 7 and pass extra notes into FL studio. The tests here were just in sending notes, chords, melodies, everything. I also did some simple timing tests by flashing LEDs at various tempos, etc. As long as the Arduino is in time, aged solenoids will sound rubato at best and drunk at worst, which still isn’t terrible at lower tempos. I plan on continuing with these tests throughout integration and up until the demo.

What are the most significant risks that could jeopardize the success of the
project? How are these risks being managed? What contingency plans are ready?

HW: one of the unexpected risk that we encountered this week was us needing to buy more solenoids. Another thing that I noticed was some latency issues with the actuators. I noticed that some of the old solenoids were actuated with a bit of latency and I think it’s just due to us using it “too much.” I marked it in case this becomes a bigger issue. As long as I just connect everything correctly to the first octave, we should have some extras to replace the “lagging” actuators with new ones.

 

SW/Fabrication: One risk is that the adhesive we ordered could not work as well as hoped on acrylic and the mounting system could fall apart like last time. Our contingency plan can just be superglue, because I think it will probably work well enough. Worst case, I have some super strong T-Rex tape.. it will just not look very pretty. Software wise, same as last week. If the symbol detecting isn’t good enough, I will fall back on the color detection.

Were any changes made to the existing design of the system (requirements,
block diagram, system spec, etc)? Why was this change necessary, what costs
does the change incur, and how will these costs be mitigated going forward?

HW: n/a — we did have to buy more solenoids due to my carelessness.

SW: no

“ Provide an updated schedule if changes have occurred.

HW: n/a

SW: n/a

“ This is also the place to put some photos of your progress or to brag about a
component you got working.

HW: https://youtube.com/shorts/Cnb2aESx0gw

 

What are the most significant risks that could jeopardize the success of the
project? How are these risks being managed? What contingency plans are ready?

HW: I am a little scared that we might have to order more solenoids in case of burnout? We currently have 7 working solenoids and ordered 7 more because we had leftover solenoids in the toolbox but now they are gone. It’s not the worst risk since we can always order new ones. However, I am also a little worried about the power supply availability. If we can’t get the 30V/3A power supply, we can’t run this system.

SW: I am still worried about object detection, but like before, my backup is using the color detection that is already working.

MSW: I’m a little worried about melody generation. Generating random things isn’t that difficult, but I think that even generating mediocre melodies will be difficult. I don’t think that it will be possible to generate great melodies, at least consistently. Great melodies are difficult to write by hand, so creating an algorithm to do so would be an achievement by itself (and something I could probably sell for a large sum of money).

Were any changes made to the existing design of the system (requirements,
block diagram, system spec, etc)? Why was this change necessary, what costs
does the change incur, and how will these costs be mitigated going forward?

HW: nope

SW/Fabrication: I did redesign the mounting system because the 3mm did not hold the weight of the solenoids very well. The cost is going to be around $30, but hopefully this is the last redesign. SW – no.

MSW: Nothing major. Just small data structure changes to accommodate new data/changes.

Provide an updated schedule if changes have occurred.

HW: n/a

MSW: No change.

This is also the place to put some photos of your progress or to brag about a
component you got working.

What are the most significant risks that could jeopardize the success of the
project? How are these risks being managed? What contingency plans are ready?

HW: I am a little worried that we are not able to solder the components to a protoboard. However, as long as we keep our wires organized, I think we can keep building our circuit on a breadboard. However, this is not a very significant risk in terms of running the system.

Music SW: Melody and chord generation have me somewhat worried. I don’t want to generate unusable garbage, but I don’t want to make everything predictable either. Our generative mode is one of the two main modes we want to implement, and if it’s not good, that’s going to be a problem.

Software (note that my stuff was overwritten earlier so I am resubmitting now): The object detection is the biggest risk still, as I have tried a lot of methods but nothing is working well still. The contingency plan is to use the color detection that is already working.

Were any changes made to the existing design of the system (requirements,
block diagram, system spec, etc)? Why was this change necessary, what costs
does the change incur, and how will these costs be mitigated going forward?

HW: we have decided not to solder.

Music SW: None since last week. I’ve been changing up struct internals to see what works best, but it doesn’t affect the overall system.

SW: Not since last week.

Provide an updated schedule if changes have occurred.

 

This is also the place to put some photos of your progress or to brag about a
component you got working

HW: N/A — will show our working circuit on Wednesday!

Music SW: Nothing here. I did sync the Arduino LED to an interesting rhythm but it wasn’t much to write home about.

SW/FAB (Katherine): added photo of the mounting system I designed in the slack.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

HW: I am a little concerned about my lack of ability to implement this hw system without burning components and possibly running out of money. Also, for some reason, when I soldered the components to the proto-board, it wasn’t working as well. Unclear why that was happening though. For now, we are using the breadboard.

SW: I am concerned about creating the dataset needed to detect my symbols. However, my contingency plan is using the color detection that is already working if I can’t get that to work.

Music SW: I’m vaguely worried that in changing a lot of things around both for integration and improvements that I may break the code, leading to large delays without rollbacks. I have been backing up my code, but I’m not satisfied with its current performance.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

HW: I originally wanted to solder my components but for the upcoming demo, we will just use the breadboard to do it. There is no change in cost.

SW: I changed how we detect movement, because using color was hard if there were any colors in the background and colors on the perosn’s clothes. Now I am trying to use symbols. No additional costs.

Music SW: I changed a lot of the internal working of the code, mostly for ease of representation and modularity, but overall it has not changed any large portions of the project.

Provide an updated schedule if changes have occurred.

HW: N/A

SW: I can’t upload photos in WordPress for some reason… It is just giving an error. But we had my schedule end now anyways so I am just adding a week for symbol detection and creating the mounting system, and the following week for more advanced feedback using the symbol detection.

Music SW: My schedule has an added week for code refinement, but overall it has not changed much.

What are the most significant risks that could jeopardize the success of the
project? How are these risks being managed? What contingency plans are ready?

Hardware: I am a little concerned about designing a circuit and soldering it onto a protoboard. It is more permanent than doing it on a breadboard so I will ask Prof. Tamal, Sullivan, or Budnik for advice if I do come across any issues. There’s really no contingency plan because I NEED to do this.

Music Software: Not much has changed. I’m still planning on tightening up any timing measurements for deciding where the beats fall. The reasoning has previously been elaborated on, but to quickly restate: if the timing is off, the user won’t really feel as though they’re playing. It’s one thing to be off-time because you aren’t matching a beat, and it’s an entirely different thing for your instrument to be off-time from you.

Software (CV): I still believe that color detection could be one of the greater risks at this point, as I started to fine-tune the color detection and it proved to be more difficult than expected. I am still not too worried about this, but the contingency plan is just putting up a plain background or having a set color to use.

“ Were any changes made to the existing design of the system (requirements,
block diagram, system spec, etc)? Why was this change necessary, what costs
does the change incur, and how will these costs be mitigated going forward?

Hardware: n/a

Software: No major changes. Bayesian Updating simplified in belief that a more complex system is unnecessary. However, there are still structures in place that allow for a full implementation.

“ Provide an updated schedule if changes have occurred.

Hardware: everything is going well! on track!

Music Software: On track.

Software (CV): On track.

“ This is also the place to put some photos of your progress or to brag about a
the component you got working.

Hardware: got my testing circuit working!

Software: Nothing too interesting to look at on the music side, though you can see some console output in Lance’s post.

Software (CV): more details in Katherine’s post, but the generative mode is recognizing patterns using the grid:

Team_D2_Dettmer_Yarlott_Cho_design_report

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Hardware: the biggest risk that I can think of is buying too expensive of an actuator that I don’t have much left in the budget for other members to use. As of now, I am testing with a less than 10 bucks actuator but if I have to go with an actuator from DigiKey (which is 26 bucks a piece), it will be difficult to plan for contingencies (at least contingencies that would require us to spend money on items that we currently don’t have). If I have to buy 26 bucks a piece actuator, I will have to stick to building one octave.

Software/OpenCV: the biggest risk is probably the accuracy of the color detection and how sensitive it is to other colors. I don’t think I have a lot of risks right now because I have a pretty good basic implementation working that wouldn’t jeopardize the success of the project. To manage this risk, I am researching how to fine-tune the color detection and possibly use depth to ensure the right colors are being picked up. The contingency plan would be to just put up a black curtain behind the user so there is no issue with outside color.

 

Music Sequencing + Generation: the largest concerns at the moment are timing issues and chord prediction issues. If we can’t consistently send notes at regular rates, the feeling of playing an instrument turns into messing with a buggy mess. On a real instrument, being out of time is the fault of the player rather than the instrument.  Although we won’t give our users the option to be out of time, we should ourselves be in time so as to not inconvenience them. As for chord prediction, if our algorithm doesn’t work, we’ll end up with chords that just don’t match what’s being played.  Of course, melody is 100% left to user preference, so they may find it more interesting to play non-chord tones. However, it’s our job to ensure the chord matches the notes as best as possible. I’m also considering methods of how to predict what notes will be played next. On one hand, notes in a key have the tendency to want to resolve towards other notes, but on the other hands, players tend to play notes sequentially. If looking ahead doesn’t work, we can always a random chord selection to play for one phrase while analyzing how the users play, and then change the progression for the second phrase, and so on and so forth. There’s definitely a lot to consider.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Hardware + Software: no, there haven’t been any changes made to the existing design of the system since I am still testing which solenoid to use. No for software as well.

Provide an updated schedule if changes have occurred.

Hardware: N/A

Software: I (Lance) am considering moving my schedule forward some so that I have time to relax over spring break. Also, sound generation has been renamed to music generation to better fit the direction our design went in. Music generation directly relates to the chord prediction algorithm mentioned above.

This is also the place to put some photos of your progress or to brag about a component you got working.

This is the sample / test circuit to test a solenoid:

 

The computer is now playing appropriate piano notes for the user moving into the note which is very exciting. I tried to show it in this picture of the terminal (it is best seen in a video)

\

Enumerate how you have adjusted your teamwork assignments to fill in gaps related to either new design challenges or team shortfalls (e.g. a team member missing a deadline).

Thankfully we have not had a member who did not meet a deadline during the project. However, Sun A had to post Lance’s status report for the first week because he did not have access to WordPress. Thankfully, this was quickly addressed by Prof. Tamal and Lance and now he is able to post his individual report and contribute to the team’s report on WordPress. Other than that, we have not yet had an incident to make adjustments to fill any gaps.

In the future, because Sun A does not have much experience in fabrication, Katherine will step in to help with the actual physical mounting structure for the actuators. And, given that Lance has a lot of experience in Robotics (as it is his additional major), he will be helping out with any difficulties that might rise while building the actuator system.