Author: lcy500

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I’m doing a bunch of integration work at the moment. This involves interfacing with the CV code, which is mostly done since it’s just a matter of passing values and reading them. Once I actually know what the final form of the values will be, I can immediately pass them to my own code, in which case they’ll be passed to the Arduino. As for my Arduino code, it’s generally complete, though there are some serial bugs that I’d like to grind out still. One issue I had was with note deaths. In player mode, if a note was played and then immediately replaced with another note, it would be left on forever because of the nature of the player struct. Only one melody note can exist at a given time, and replacing it without culling the note leads to the note remaining on. I had thought this was being properly handled, but for some reason this was because of my ordering. At a high level, I read in data and then update the player struct and manage note times, etc. What I did when reading new notes was immediately setting the note age to maximum before creating a new note. My data reading is non-blocking, but for some reason the player update function wouldn’t work. So, I added it in the first conditional after receiving data saying that I’m reading a new note. This still didn’t work? Then, with only an inkling of intuition, I moved it from the bottom of the conditional code to the top, and then it worked. Notes were properly culled. Thinking back on it, the second code addition should’ve made it so that no notes could play at all, but this didn’t happen. If I were to look more into the issue, I’m sure I could figure out why exactly this was happening, but it doesn’t matter enough since the code works and I have more important matters to focus on. I’m also in the middle of changing the music generation code to create 8th notes instead of 32nd notes. This was really easy on the Python side, but the Arduino is taking a bit of time. This is definitely a load off of my back in terms of data transfer speed, though. For integration with the solenoids, I’ve confirmed with a simple LED array that notes are pushed to the correct pins at the correct times. I’ve also tested delay (without solenoid activation) and when using the Arduino Due’s native USB port, it’s incredibly low. I can’t say that it’s less than 10ms, but it’s low to the point where I can’t notice it with 16th notes being played at 90bpm in player mode. Player mode doesn’t have any sort of sequencer, so if notes are in time, that means that they’re being sent over at consistently quick speeds as well. There was some issue regarding a solenoid activating three times in quick succession every time the Arduino was either flashed, booted, or connected to via serial (which “sort of” restarts the Arduino). Apparently, on boot, the Arduino briefly sets all of its pins to INPUT, meaning that the connected pins are receiving floating voltages. This can clearly be seen when testing with LEDs, as they’re all dimly lit whenever any of the previously mentioned cases occurs. Although the solenoids don’t all attempt to activate at once, I think that must be related to every pin being set to input. When they’re set to their proper state at the end of startup, only pin 13 (the highest pin connected) would activate its solenoid. I am completely unaware of the circuitry on the Due, but maybe it’s related to some flow from surrounding pins being set to OUTPUT. I’m almost certain that if I hooked a measurement device up to the pins and ground I would see a maximum voltage that’s above the turn on voltage for the MOSFET. I’ve mentioned multiple times that a pull-down resistor between the Arduino and the MOSFET would probably solve the issue, so now I’m just waiting for that to be added to the circuit. Then, I can at the very least confirm or rule out whether or not these floating voltages are the cause. In the first place, if the pins are set to OUTPUT, and then immediately set to LOW during the Arduino’s setup() function, wouldn’t one expect them to not activate anything? Either way, once this issue is solved, complete integration is just a matter of hooking up wires to their respective pins. Oh, I also made the switch from the PWM pins to just standard digital pins. I didn’t bother confirming why but the PWM pins inconsistently powered the LEDs, causing them to flicker. The regular digital pins work fine, so that’s what I’m going with.

Also, just in case we lose any more solenoids, I added code that allows us to play MIDI notes only for specific pins. So, if the solenoid at pin 22 dies, I can reflash the Arduino with code that marks the pin with a -1, which can signal for the Arduino to play notes over MIDI instead.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Due to me changing up the minimum subdivision in my music code as per the recommendation during my presentation last week, I’m slightly behind. Plus, my robotics capstone demo is this Monday, so my priorities are a little shifted at the moment considering that system is far larger than this one, and there’s still a lot of final work to be done. However, I plan on completing everything for this well in advance of demo day, and there’s nothing in here that I haven’t done before, I just need to get it done.

What deliverables do you hope to complete in the next week?

I need to finish my planned code changes and that’s about it. Obviously there’ll be debugging involved with that, but I’m predicting that things will go smoothly. Integration should be easy enough, as the serial pipeline is already completed, meaning the most difficult part has been finished for integration. Now, it’s just a matter of passing values between Python code or sending levels to pins.

What did you personally accomplish this week on the project? Give files or

photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

 

Apparently there were some “issues” with previous status reports, so here’s an overview of my entire set of software that I’ve written, including things I’ve done these past two weeks. Strap in!

I played around with some new chord progressions and added them to the list of possible sets to choose from. These go into the generative framework and will be used to help guide generated melody notes along their path. Progressions are represented as the interval of their root note in the C major scale, so a C major chord is 1, an F major chord is 4, a G major chord is 5, and so on and so forth. The progressions are stored in a 2D array, being n*4 in size, n being the total number of possible progressions. These progressions were picked by hand for two reasons. I want them to resolve back to the 1 somewhat nicely, and I want to avoid strange repetitions or weird patterns that might sound unpleasant to users.

For reference, I made a slight change to generative mode, which hasn’t been mentioned in a short while thanks to dealing with embedded code. As stated, chord progressions are now picked from a list and the program itself isn’t allowed to generate them. Then, melody notes are generated on top of it. There are two parameters that decide what is generated. One decides the tonal mood while the other decides the rhythmic intensity. It should be noted that tonal mood isn’t necessarily saying that the notes should make you feel one way or the other, it’s just saying that the initial vibe of chord progressions and melody are either somewhat bright or somewhat dark (though dark can be hard to achieve in C major without non-diatonic notes). The tonal moods of chords are decided arbitrarily by myself. Rhythmic intensity defines how quickly the piece should be played. The program steps through each possible count, this ranges from 32nd notes to whole notes depending on the intensity, and chooses what count to play on. This is done mostly randomly, since the main randomness comes from what the input parameter is. However, it is also possible to decide on a rolling basis by taking the parameter to be the mean of a probability distribution and randomly selecting a rhythm at each step. If a parameter is centered at 0.5 in the range [0, 1], then it should mean that while half and quarter notes are going to be more common, there’s still a chance for more interesting rhythms to be created. Similarly, if the input parameter is 0.1, then there’s going to be a large number of 16th and 32nd notes (or vice versa, it’s very easy to flip). This process would be repeated for every step along each 32nd note, and after a note subdivision is chosen, then the program will move to the next available count until none remain. The current algorithm does this well. The actual note will then be decided afterwards. Fast rhythms will tend towards smaller changes while slower rhythms will tend to move more freely, though large jumps can happen.

The serial side of things just keeps growing in size, all in the name of optimization. I’ve said before that I’ve changed structs around, and that’s because it’s exactly what I’ve been doing. Frankly speaking, I don’t think that much more needs to be said about that, but it appears that verbosity is the norm rather than the exception. Luckily, I excel in being verbose the exact way I do in being brief. So, here’s an overview of just a few of the various struct changes that I’ve made in the past few weeks. “Oh, I need to know that a note is done playing so that it can be replaced? Got it.” Then I add a field to the phrase struct that’s an array of 256 booleans that determines whether or not a note has been played. At startup, this is marked as true so that any new incoming notes can be added. The struct that stores this is used for generative mode and stores the entire phrase. Then I realized, I can’t block anything, so I need a global variable that tells me what variable to play and when. So, there’s now a global index into this array. Of course, this still isn’t enough to know if it’s safe to change existing notes. Due to the nature of generative mode, new notes may be added before old ones get the chance to be played. However, we’re also working with musical phrases. There’s a few ways to do this. To preserve the current phrase being played, we can only change subphrases that aren’t active. This is just a struct field of booleans. If it’s safe, change it, if not, don’t. Another way to do it is to absolutely change everything on the fly, measure by measure. This involves a counter that marks what measure (of 8) is being played. Then, we pass that measure to a separate struct that goes into a non-blocking player function. We’re then free to change everything else about the music, from its phrase to its melody. Every measure, we take what we currently have, stop updating if really necessary, then pass that measure onto the player struct which takes everything and runs with it. This is explicitly separate from the player mode structure and instead runs off of a global timer that tracks the exact note location in measures and beats down to 32nd notes. It makes use of Arduino’s millis() function and updates every single time loop() is called. The length of 32nd notes is calculated either at the start of the file, or when the tempo is updated (which is a message that can be sent and will be expanded upon in a moment). The function tracks and says, “Ok, where am I?” and calculates its position in a measure based on the start time of that measure. Naturally, this means that there will be times when some notes aren’t selected exactly on their downbeat, but this is still functional enough, as 32nd notes at 120 bpm last for about 15.6 milliseconds. I don’t think the solenoids can handle that, so the tempo will be much slower. The one risk is in reading data. If reading takes too long, it might bypass some notes, which isn’t ideal. So, how can I combat this? There’s a couple ways. The first is simple, I start blocking. Now, I know what I said earlier about blocking and non-blocking, but hear me out. The note isn’t going to change every 32nd note most of the time, so there will be downtime where the pin can hold itself at HIGH for some longer period of time. I would simply account for this by adding a new structure that marks beats as available for updates. This is called at the start of loop(), and based on its result, new data may or may not be read. This seems to be the most simple, since it only involves adding one new array. In fact, I’m pretty sure it is the most simple. A small concern of mine is whether or not the serial buffer will clear itself by the time I get to it since the serial buffer’s size is only 64 bytes. If I send in new chord information, that’s 6 bytes sent in as fast as I can. If my generative phrases are sent in batches, I’ll be sending 128 notes plus a chord progression at around the same pace. That’s a whopping 519 bytes. So, if I start blocking, I might lose data. I could try to send confirmation back, but that just slows the pipeline down and makes everything worse. So, what to do? This actually relates to the second way I mentioned for reading in data without missing notes. One solution is to fill out a bunch of notes before they’re played, then start playing. This gives me a good bit of leeway to fill in new notes and fill notes in behind as well, if I manage to catch up (“What happens if I do catch up?” was one question I had but answered above with the boolean for notes that have been played). “What if you have so many 32nd notes in a row that you can’t read data?” If I get that many in a row, I’ll eat my shoe. The rhythms are based on a sample of a truncated normal distribution. Sure, it’s not impossible to get a lot of 32nd notes in a row, especially with the rhythm parameter maxed out, but the odds of getting so many in a row that I can’t even take a fraction of a second to read in data should be astronomically low. There is one more solution to the problem that I’m not particularly fond of but am willing to implement if necessary, and that’s limiting the speed of data transmission. This opens up a whole new can of worms. I would need to find a good balance between sending enough data that the Arduino can play something, and then sending it at a slow enough rate that it can read new data. The current solution I’m going for is just waiting for a while before starting. The mood is somewhat arbitrary, so it’s output only needs to generally follow the user’s movements. This is going well so far. I also implemented a MIDI framework that allows the arduino to directly interface with a DAW in case we lose any solenoids during our demo. This works great with player mode and just needs a tad bit of tweaking to with with generative mode.

 

Is your progress on schedule or behind? If you are behind, what actions will be

taken to catch up to the project schedule?

 

I’m on track. I just need to finish up generative mode’s serial framework and I’ll be good to go with both full testing and debugging.

 

 

What deliverables do you hope to complete in the next week?

 

The Arduino activates some solenoids on startup and connection and I need to diagnose and fix that. I also need to grind away any delays that might become a problem, either through optimization or changing the speed.

Posted late because the Slack announcement said “April 10th,” apologies!

 

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I mostly finished up the comms pipeline between the Arduino and Python scripts, which means that now we can fully work on assembling the system and writing more complex music generation code. Some issues have arisen, such as certain pins being activated when they shouldn’t be. As far as I can tell, it just shouldn’t be happening, but it is. I’ll need to look a little deeper into the code to figure out what the issue is. However, I don’t think that fixing the issue will take more than a few hours. I also had to fix some minor timing bugs regarding establishing the serial connection, but things are going well.

Is your progress on schedule or behind? If you are behind, what actions will be
taken to catch up to the project schedule?

I’m on track.

What deliverables do you hope to complete in the next week?

I’m going to completely finish the comms pipeline and then put my all into making the music generation code the best it can be.

Now that you are entering into the verification and validation phase of your project, provide a comprehensive update on what tests you have you run or are planning to run. In particular, how will you analyze the anticipated measured results to verify your contribution to the project meets the engineering design requirements or the use case requirements?

I’ve been doing unit testing on my comms code without any circuits involved, just back and forth communication between the two devices. This involves sending values from a Python script to the Arduino, and then having it send verification codes, received data, etc. For music generation, I can just print a list of notes and then play them on my own keyboard to see if the melodies sound passable (or at least not terrible).

I didn’t do anything :'(

 

Mandatory April Fools’ jokes out of the way, here’s the real update:

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Code has been rewritten and interfacing is set to be fully complete by morning. We’re running with a slightly simplified system for our demo, so I’m in the middle of making sure those aspects are perfect. I’m also writing a test program that showcases how our solenoid system can be used  if it were played perfectly. It’s just going to play a relatively simple melody with backing chords, but I think it’ll work as a good benchmark for timing and syncopation.

Is your progress on schedule or behind? If you are behind, what actions will be
taken to catch up to the project schedule?

All good on my end, or rather, I count Sunday as the end of my week, so everything will be on track by tomorrow evening.

What deliverables do you hope to complete in the next week?

After our demo, I’m going to complete melody and chord generation and get them at least somewhat functional and interesting. Integration should be completely finished, which means I can spend all day tweaking algorithms. Oh, boy!

I also want to see if there’s an easier way to detect the Arduino, because I’ve been having to manually find the COM port the entire semester. It’s not that big of a deal, but it does diminish the “plug and play” aspect of the system.

What did you personally accomplish this week on the project?

This week,  I finished up a bit of platform integration code, and after working with Katherine to discuss software integration I decided that it would be best to go back and rewrite large portions of my own code. This is both for ease of integration, and ensure that the code is easy to maintain, modify, and work on for someone else. I’ve gotten through parts of it, mainly the Python end of things. This involves some code that manages sending messages to Arduino and representing notes in the code. The note representation hasn’t changed much, but it has been refined. For testing purposes, it was mostly just a number, a length, and a checksum. Now, it’s a list that contains a length, a list of notes that are represented as their MIDI values, and finally a checksum. The current implementation has better support for chords, because the previous simply relied on the Arduino interpreting them fast enough to queue them into the same time step. The clocks of the computer and Arduino won’t be synced, so there’s no need to add a start value (rather, it would be difficult to, and it is currently unnecessary due to the speed of the current pipeline). For more info on the representation, notes will be sorted into measures, which are grouped into phrases. There will be one phrase that constantly updates which contains eight measures. The measures have no set length (in Python), but do have a maximum length of timeSignatureNumerator*timeSignatureDenominator*8. This means that in 4/4, you can have four sets of eight 32nd notes, which is generally the smallest subdivision anyone will have to deal with on a daily basis. Measures are ordered, as are phrases, but note lists are not. However, ensuring that there are no duplicate notes is important.

If I decide to make a version that does not enforce strict timing, I can just add a flag to the note list that determines whether or not they will be played immediately or sent to the phrase+measure list set.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I had initially planned on finishing up integration and testing around this time, so it’s debatable as to whether or not I’m on track, but I plan on finishing as much as I possibly can before the interim demo.

What deliverables do you hope to complete in the next week?

I plan on expanding upon what I already have and making it work with Katherine’s code, as well as letting the Arduino sequencer interact with the solenoids.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I finished up the interfacing code for having our computer talk to the Arduino. It sends data in chunks, where certain symbols (like ! or @) delineate portions of data. This way, we can ensure that the data we send is accurate by marking different blocks of data and including an ending checksum. It also allows for easy chord representation and reading (parse through bytes until you see a block footer, add every note in the block to a “play next” list). This code works on a small scale, but I’ll need to make sure it can work with large amounts of data as well.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

As it stands, I’m currently on track.

What deliverables do you hope to complete in the next week?

I want to tighten up my algorithms and ensure they produce at least somewhat nice sounding results. This involves tweaking Bayesian Updating and ensuring that the note queue functions properly at all times. I’m also going to add phrase support for the chord estimation, so that we can generate cycles instead of walking along one never-ending chord line. This also helps with deciding on weights for certain chord probabilities.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours):

Before Spring Break, I implemented a somewhat simplified version of Bayesian Updating. This will allow us to predict chords as the user plays. One thing to note is that the probabilities of every chord that doesn’t contain the played note will slightly decrease, but the values in the array are normalized between 0 and 1, so this isn’t apparently. This also means that the selected chord will always have a probability of “1.0” with this algorithm. Although it does simplify some aspects (like visually picking out the right chord), it can be slightly misleading, as we aren’t really “sure” that the most probable chord really is the chord being played over.

Some output from the algorithm can be seen below (array indices are [C, D, E, F, G, A, B]):

“ Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Thankfully, I am on track and can immediately begin working on integrating all of the musical systems together. This includes computer communication with the Arduino.

“ What deliverables do you hope to complete in the next week?

I plan to integrate the Bayesian Updating and general signal transmission code with the Arduino’s actuator code.

Over the past week, I finalized the basic sequencing algorithm and and pitch selection through the Arduino.

Let’s start with the easy one first. Pitch selection, since we only have a small number of keys in use, is just pushing a signal to an Arduino’s digital out pin. However, in the case where we have a far greater number of keys (i.e. future extensions of the project), we would just need to implement a simple switch matrix, allowing us to select from 90 keys with just 9 pins. Pictured below is a basic schematic from the above link:

Tempo is measured in beats per minute, so it’s fairly easy to translate this into a time value for quarter notes and other subdivisions. At 60bpm, you have one quarter note per second (in 4/4 time), at 120bpm, it’s one quarter note every half second, and so on and so forth. For a quick explanation, you can just convert beats per minute to beats per second by dividing by 60, then just flip the expression to find seconds per beat. If you want 16th notes, divide the expression by 4. Dotted expressions are slightly more but still manageable, as they add one half the value of the note, but following dots add half the value of the added value. A single dotted half note adds a quarter note, double dotted adds an eighth note onto that, and a triple dotted note adds another 16th note onto that.  Rests are easy to sequence as well, since it’s just a time period of playing nothing (sending no signal). I will need to be slightly careful about rounding errors, but none have popped up so far, so I’m leaving that on the back burner as I work on more pertinent issues. There isn’t much to show here, especially since this was just console output that was similar to one of my teammate’s, but the hope is that next week there will be a very interesting video to post!

The Bayesian Updating algorithm is also being written, but it’s for our secondary mode, so its priority is somewhat lower compared to other tasks. However, some data structures have been set up and testing it should go quickly once it’s finished.

As of now, I’m currently on track and am considering accelerating my schedule with spring break approaching.