Posted on

Lance’s Status Report for Feb 25th

Over the past week, I finalized the basic sequencing algorithm and and pitch selection through the Arduino.

Let’s start with the easy one first. Pitch selection, since we only have a small number of keys in use, is just pushing a signal to an Arduino’s digital out pin. However, in the case where we have a far greater number of keys (i.e. future extensions of the project), we would just need to implement a simple switch matrix, allowing us to select from 90 keys with just 9 pins. Pictured below is a basic schematic from the above link:

Tempo is measured in beats per minute, so it’s fairly easy to translate this into a time value for quarter notes and other subdivisions. At 60bpm, you have one quarter note per second (in 4/4 time), at 120bpm, it’s one quarter note every half second, and so on and so forth. For a quick explanation, you can just convert beats per minute to beats per second by dividing by 60, then just flip the expression to find seconds per beat. If you want 16th notes, divide the expression by 4. Dotted expressions are slightly more but still manageable, as they add one half the value of the note, but following dots add half the value of the added value. A single dotted half note adds a quarter note, double dotted adds an eighth note onto that, and a triple dotted note adds another 16th note onto that.  Rests are easy to sequence as well, since it’s just a time period of playing nothing (sending no signal). I will need to be slightly careful about rounding errors, but none have popped up so far, so I’m leaving that on the back burner as I work on more pertinent issues. There isn’t much to show here, especially since this was just console output that was similar to one of my teammate’s, but the hope is that next week there will be a very interesting video to post!

The Bayesian Updating algorithm is also being written, but it’s for our secondary mode, so its priority is somewhat lower compared to other tasks. However, some data structures have been set up and testing it should go quickly once it’s finished.

As of now, I’m currently on track and am considering accelerating my schedule with spring break approaching.

Posted on

Katherine’s Status Report for Feb 25

This week, I worked on fine tuning the key detection and actually producing sound to fine tune it to respond naturally to ‘key presses.’ This week was presentations, so I wasn’t able to get as much done as normal but I spent a while trying to get some python libraries that produce sound to work (ran into some unexpected issues with versions and installing). I finally got one to work, so now when someone puts their hand in a note, it waits an appropriate amount of time (added a counter for this), and then presses the note and plays the actual sound on my computer (super exciting!). It looks very cool in video, but to show this I put the terminal output below, showing how the notes are generating after I move in the box.

So far, I am on schedule.

Next week, I am hoping to map gestures and get generative mode working for certain patterns.

Posted on

Sun A’s Status Report for Feb 25th

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I was the one who presented for the design review presentation so most of my week was consumed in making the presentation and preparring for it. I also received my orders from Adafruit and DigiKey so I was building the sample this week but I did not get to test it… I will do the testing on Monday when the receiving is open (the keyboard is with the receiving because it’s hard to store a full sized keyboard).

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

So far, I am not behind the schedule.

 

What deliverables do you hope to complete in the next week?

I hope to finish testing next week and figure out if I should get a new keyboard (less weight on the key) or if I should get a new type of solenoid. This will be the biggest task for me next week!

 

 

Posted on

Team Status Report for Feb 25th

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Hardware: the biggest risk that I can think of is buying too expensive of an actuator that I don’t have much left in the budget for other members to use. As of now, I am testing with a less than 10 bucks actuator but if I have to go with an actuator from DigiKey (which is 26 bucks a piece), it will be difficult to plan for contingencies (at least contingencies that would require us to spend money on items that we currently don’t have). If I have to buy 26 bucks a piece actuator, I will have to stick to building one octave.

Software/OpenCV: the biggest risk is probably the accuracy of the color detection and how sensitive it is to other colors. I don’t think I have a lot of risks right now because I have a pretty good basic implementation working that wouldn’t jeopardize the success of the project. To manage this risk, I am researching how to fine-tune the color detection and possibly use depth to ensure the right colors are being picked up. The contingency plan would be to just put up a black curtain behind the user so there is no issue with outside color.

 

Music Sequencing + Generation: the largest concerns at the moment are timing issues and chord prediction issues. If we can’t consistently send notes at regular rates, the feeling of playing an instrument turns into messing with a buggy mess. On a real instrument, being out of time is the fault of the player rather than the instrument.  Although we won’t give our users the option to be out of time, we should ourselves be in time so as to not inconvenience them. As for chord prediction, if our algorithm doesn’t work, we’ll end up with chords that just don’t match what’s being played.  Of course, melody is 100% left to user preference, so they may find it more interesting to play non-chord tones. However, it’s our job to ensure the chord matches the notes as best as possible. I’m also considering methods of how to predict what notes will be played next. On one hand, notes in a key have the tendency to want to resolve towards other notes, but on the other hands, players tend to play notes sequentially. If looking ahead doesn’t work, we can always a random chord selection to play for one phrase while analyzing how the users play, and then change the progression for the second phrase, and so on and so forth. There’s definitely a lot to consider.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Hardware + Software: no, there haven’t been any changes made to the existing design of the system since I am still testing which solenoid to use. No for software as well.

Provide an updated schedule if changes have occurred.

Hardware: N/A

Software: I (Lance) am considering moving my schedule forward some so that I have time to relax over spring break. Also, sound generation has been renamed to music generation to better fit the direction our design went in. Music generation directly relates to the chord prediction algorithm mentioned above.

This is also the place to put some photos of your progress or to brag about a component you got working.

This is the sample / test circuit to test a solenoid:

 

The computer is now playing appropriate piano notes for the user moving into the note which is very exciting. I tried to show it in this picture of the terminal (it is best seen in a video)

\

Enumerate how you have adjusted your teamwork assignments to fill in gaps related to either new design challenges or team shortfalls (e.g. a team member missing a deadline).

Thankfully we have not had a member who did not meet a deadline during the project. However, Sun A had to post Lance’s status report for the first week because he did not have access to WordPress. Thankfully, this was quickly addressed by Prof. Tamal and Lance and now he is able to post his individual report and contribute to the team’s report on WordPress. Other than that, we have not yet had an incident to make adjustments to fill any gaps.

In the future, because Sun A does not have much experience in fabrication, Katherine will step in to help with the actual physical mounting structure for the actuators. And, given that Lance has a lot of experience in Robotics (as it is his additional major), he will be helping out with any difficulties that might rise while building the actuator system.

Posted on

Team Status Report for Feb 18th

We decided to slightly simplify our control method in favor of usability. Instead of controlling single notes with one hand and chords with the other, we have opted to instead control note pitch with one hand and various other parameters (volume, subdivision, rests, etc.) with the other. This should give the user far greater control over what they play. To compensate, we will need to choose what chords to play while accompanying the user. This issue is discussed in more detail in Lance’s post. There is a slight risk of accidentally selecting chords that do not mesh with what the user wishes to play, but in all honesty, because the key is always going to be C major (other another key in a different scale mode with all white keys), this can be handwaved with the justification of interesting harmonic intervals. We don’t plan on placing any chord notes above the melody note, so there should also be no issues there.

We also created the grids needed for the CV recognition in both note playing and generative mode, and the actual generation of notes in the simple note playing mode. The generative mode was defined to use a grid and patterns that we store, and added a block that the user can place their hand in to switch the mode. (more information in Katherine’s post)

As for the hardware side of the project, I finished ordering the parts to create a proto-type version of the actuator system. I only ordered one solenoid from Adafruit to see if I need to resort to ordering a $20+ solenoid for the project because the small solenoid from Adafruit only costs us $8 per piece (and we need 14 solenoids in total). I drew up the circuit diagram for the proto-type and I also created a block diagram for the presentation and got it approved by the team. I will be presenting next week so I have been practicing for that as well!

No update was needed for the schedule because we are all on track 🙂

 

Posted on

Katherine’s Status Report for 02/18

This week, I worked on generating the on-screen grid for the playing mode and the generative mode, generating notes in playing mode, as well as finalizing how each of these will work. For the note mode, I decided to implement a delay to determine whether someone is actually pressing a key or chord — so a counter starts as soon as one of the notes / chords sees the color enter and if a certain amount of time passes, the note is generated. There is a “button” at the bottom, shown in purple in the attached photo, that will switch to generative mode if either hand is inside the box for 2 seconds. Generative mode is shown as a grid, and we are going to have patterns stored defined by the order of grid boxes passed through. By constantly recording which grid boxes the person moves their hand through, we can determine if any patterns have been matched and use that to generate a sequence of keys. Next, I am going to try to hook up sound to test how the playing is actually going to sound and work more on the generative mode. I am currently a bit off schedule since we didn’t account for initial work in the schedule, but the schedule needs to be reworked after making a lot of changes to howwe are going to approach the project. There was a lot of open time in my section before, so it will be easy to just shift everything down a week and add some more into the schedule.

Shown in the images is note mode and generative mode. I have my finger over the camera so that the graphic can be seen a lot easier.

Posted on

Lance’s Status Report for Feb 18th

Over the past week I’ve been reevaluating how I want to handle music generation.

Previously, I had planned on having multiple different solutions to different problems, but have come to the conclusion that it would be best to focus on one functional approach.

I’ve approached the problem with the idea of the user playing single notes. This slightly simplifies the problem while also leaving room for algorithmic improvisation. With the user playing single notes, we can use software to analyze what notes they’ve played, and create a chord progression from that.

Chord progressions are cyclical (generally), so we can assume that it should repeat every 4 to 8 bars with little variation. This can be seen below:

Here you can see that- although not in the key of C- chord progressions can be simple, even for popular songs.

The way a chord progression will be assumed is through Bayesian Updating. The cyclic nature of progressions allows us to examine bars individually. Through this, we can measure the frequency of notes played and create a probability measure for what chord is being played during the measure.
As a simple example, say we play the note D five times, F six times, A 3 times, and B twice. If we were to make an assumption about what chord was being played  over(triads only), we could assume that either D minor or B minor were being played. A was played more than B, so we know that there’s a higher chance that the chord is D minor (the triad being D, F, A).

This does involve some assumptions, and could require us to examine surrounding measures to ensure aforementioned assumptions make sense. However, with this approach, the algorithm is free to comp the user with chords and rhythms that can make the piece more interesting (we can also use a similar approach to decide rhythm!).

I’ve been testing algorithms by hand, and overall they seem to work well enough. I’ve also managed to send regular signals through an Arduino, now I just need to make a specific parser for translating to different tempos. I would say that overall, I am on track with our current schedule. During this upcoming week, I plan on writing the Bayesian Update algorithm and ensuring it works.

Posted on

Sun A’s Status Report for Feb 18th

For the first half of the week, I spent most of my time looking at different solenoid options before placing an order. There are different types of solenoids — push type, pull type, and push-pull type. For the external actuator system, push system might be the most intuitive option; however, push solenoids are more expensive than pull solenoids in general. Then, I looked at Adafruit and found that their solenoids are actually push-pull type and offers a reasonable price for their small solenoids. So, for this week, I ordered ONE small solenoid from Adafruit to see if their solenoid is, indeed, push-pull type and its push is enough to play a key on the keyboard we got from the receiving.

I also designed a super rough circuit to just test the push-pull solenoid, which I am attaching below:

For the testing, I would only need a regular power supply that’s already in the A level lab, but for the future building process, I would need to request a “beefier” power supply, which I will reach out to Prof. Budnik to ask about the model that I used in his control systems class.

Posted on

Lance’s Status Report for Feb. 11th

Over the past week I’ve been weighing options for melody generation. There are a plethora of options because of the nature of music as an artform.

If we choose to let the user directly control the notes they play, then the form is simple. Given some input (grid coordinates for now), we can tell the hardware to press a specific note on a keyboard. The question then becomes, how do we make it interesting? If you’re stuck always playing exactly the note you’re at, it’s not going to sound like music. Well, it will, but it could always be more interesting. Then, let’s say that the other axis of your grid determines how fast the notes are playing, their subdivision. This is fine- and it definitely adds more spice- but there are still some key elements missing. What if we want to “play” a musical rest? What if we want to jump over some number of notes, say, an octave? There are definitely options for this, for example, in the case that we have colored gloves, we could use the absence of color to indicate a rest, which can then be used to make note jumps. This solution should be incredibly intuitive, but difficult to play skillfully at first, which could be discouraging.

If we want to procedurally generate melodies based on two notes at some point, we have a few options. The simplest is to say, “What is the subdivision? If you give me that, I’ll create a string of notes that run up (or down) from the first to the second note, in the amount of time you give me. If there are too many intervals between the two notes to be played, I’ll skip over some of the boring ones. If there aren’t enough, I’ll make some longer.” This works, and we can even add random variations. For example, we could randomly choose to generate a “harmonic enclosure,” which is when you land on a note after playing the notes above and below it in a sequence. We can add plenty of other flairs to our melodies like this. This solution is very versatile, but may be difficult to create a smooth user experience for.

The last solution is something I need to further consider, but it involves generating a melody based on “mood” determined from gestures. We could possibly do this by parameterizing certain musical intervals with various emotions, and then selecting notes based on that. This gives the user a lot of leeway, but also greatly increases the amount of effort needed to create something musically coherent and satisfying to play/listen to. Sure, moving from the 7th to the 1st interval is satisfying, as is a ii-V-I progression, but everything can be used in so many different ways that it’s hard to say what the best outcome would be. This solution, if implemented properly, would be extremely fun to use, but definitely sacrifices some of the user’s autonomy.

As it stands, I am on track to start with pitch selection algorithms and continue with musical progression generation.

Posted on

Team Status Report for Feb 11th

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

  • Hardware (Sun A): my biggest concern is designing an actuator system that will be safe to use and functional. For instance, because I am using 14 solenoids — not using all 14 at a time though — the amount of current required for the system will be close to 5A. I think that the 18100 power supply may not be suitable for the system. This also means that I can’t use a breadboard to build a system — which means that I would have to solder onto a PCB board to create this actuator system. A little intimidating… But, I am hoping that my experience as a 220 TA and in 474 will be helpful 🙂 However, if this design does not work (fingers crossed), I will have to call for SOS (and hopefully prof. Budnik is available to help). But, if that doesn’t work out either, we’ll just have to build a smaller system with less number of solenoids.
  • Software (Katherine): my concern is gesture recognition. Being able to recognize certain gestures using computer vision may be difficult, but I am confident that we could atleast recognize where a hand with a colored glove is in a screen, so that would be a perfectly good contingency plan.
  • Music Software (Lance): Although I am slightly worried about accidentally designing the music generation algorithm to be out of scope, I am simultaneously excited to begin working on it and creating music. Personally, I’d love to implement all of the ideas discussed in my own post (and more!), but I will focus on gradually building the algorithm from the ground up.

“ Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

  • Hardware: we secured a keyboard from the ECE receiving, which means we have 50 dollars more flexibility to buy additional MOSFETs in case I burn some of them. But, other than that, no changes were made so far.
  • CV: Not as of now.
  • Music Component: N/A

“ Provide an updated schedule if changes have occurred — N/A

“ This is also the place to put some photos of your progress or to brag about a component you got working.

Recognizing color!

 

 

Our project includes considerations for welfare and societal implications because we hope that our product could allow people who do not have the ability to play music to play music.