Posted on

Nora’s Status Report for 3/18

This week, I worked on integrating the RPi GPIO outputs with the physical circuitry by attaching them to the gates of each MOSFET. A video of the chromatic scale that we showed in a previous status report being played on the solenoids can be found here

One thing we have not mentioned so far is that, from the rough piano layout seen on the paper, you can see there is an extra solenoid between the E and F spaces. This solenoid is placed in the area where there would be a black key in between the white keys. Another will be placed between the B and C spaces. This part of the design is so that we are able to move the accompanyBot up and down the scale. Since different starting keys of the octave will require different orders of black and white keys, we will just remap the solenoids to the correct keys. Thus we will be able to play more contiguous notes for certain melodies that do not start on a C. The downside of this design is that at any given time, two of the solenoids will be useless. However, the solenoids must be arranged like this if we do not want to have the user have to move the solenoids themselves each time they adjust the starting position.

To make the pin mapping more modularized, I have written functions that change the dictionary that maps the notes to pins based on the starting note passed into the function. This will allow the user to place the accompanyBot at different spots on the keyboard.

I also added additional XML files that can help in testing the tempo and the case where the piece tries to play more than five keys at a time

So far, I am on schedule according to the Gantt chart, although tomorrow I will need to add support for chords since they are not supported in the current implementation due to the fact that I did not realize they are a different data structure in the music21 library.

In the next week, I will work on making the process for playing a piece more streamlined since at the moment, I must manually change the files and rerun the code every time I want to switch the piece. I will also work with Rahul on accepting the user inputs to start and stop.

Posted on

Rahul’s Status Report for 3/18

The earlier portion of this week was spent working on the ethics assignment for this course and analyzing my project from an ethical lens. After that, I implemented more of the UI diagram from my previous status report into our GUI python application. Some details have been omitted while some will get added towards integration stages, but here is the current bootup view:

You might be able to tell that last week’s application view that I showed was running on MacOS while this week has a Windows style app window. Since the app will be designed for Windows, I have been making the effort to continue my initial development on Windows. Also, I left out the fast forward and rewind red arrows and replaced them with forward and back, to give the user a measure control. FF and RW just didn’t make sense for the accompanyBot. The only UI I have not developed was the measure display and the slider. This should be simple enough especially in comparison to the rest of the features that were intended for next week which I will detail below in this post.

Another change made, was utilizing the PIL library for image resizing as opposed to pygame’s internal resizing function. Now image quality is very much retained while displaying the music score in the application. Since I am at the stage where I will be implementing UI changes while also adding integration components, I decided to introduce one piece of functionality. In particular, the import pdf/png button now works upon a click. Initially, I had planned to use a python module for displaying a GUI of importing a file but I realized that this would get very computationally expensive and if it were to run in the foreground then the user would not be able to perform other UI actions (say pause the player robot immediately). For this reason, I did more digging into powerscript development to figure out how to launch the Windows file opening system and return the file name that was clicked and opened. From here, I modified the event loop code for the pygame graphics, that are displaying the app view I show above, to utilize multithreading to launch the windows powerscript job, maintain the GUI operation, and cancel user requests to import more files while the file opening system is already running.

As of now, once the user selects their file, the python code just prints the file returned, but given the previous work I have posted to our team Github of spawning an OMR job from python with the file string as input, this should hopefully be a straightforward integration hack. I may also want to multithread the OMR job appropriately.

According to our team Gantt chart I am on schedule. Tomorrow I should finish the home screen UI, which is pretty much done, and by next week all user actions should be implemented including the OMR integration. I have been on schedule every week so far, but just since next week is such a large task, I will have to work longer to get it done and still likely will fall short of finishing. (I say this because getting the file insertion view working properly took a few hours alone) Ultimately, I will try to get as much as I can finished and catch up as necessary.

Posted on

Aden’s Status Report for 3/18

This week I did not accomplish all the goals I set out for myself. Last week I stated that I wanted to have a full sketch of what the chassis will look like and possibly a Solidworks rendering of what I sketched. I was not able to get to the Solidworks rendering primarily because my Macbook cannot download it from the CMU software catalog, but I was running out of time this week trying to catch up on all the work I neglected to do over spring break. I was able to sketch a very rough idea of what we would like the chassis to look like. I also got some rough estimates on the dimensions of the solenoids and, using those measurements, the minimum dimensions of the chassis. My idea is to have a 3D-printed container with a wall down the middle for the solenoids to be mounted. With the solenoids mounted down the middle of the container, they will hover over the piano keys and press them when turned on. Additionally, two pipes will go through the container’s length and can be placed on the edges of the piano so that the chassis can hover over the keys.

 

As you can see above, this is a rough idea of what the chassis should look like. The top view shows how the solenoids will be placed in the container, with the top set of solenoids hovering over the black keys and the bottom set hovering over the white keys. Furthermore, the side view indicated roughly where the pipes will go through the container so as not to interfere with the solenoids.

This brought sketch has put me a little behind schedule since I wanted a little more detailed idea to be completed this week, but it is not the end of the world since we have so much slack time built into our schedule. Hopefully, this week I will be able to make up for lost time last week.

Some deliverables I plan to provide at the end of next week is a much more in-depth drawing of what our chassis will look like. Additionally, I would like to measure the keyboard dimensions to get a much more accurate idea of what the size of the chassis will need to be so that it fits on top of the keyboard. I also hope to get my hands on a copy of Solidworks so I can play around and figure out a good rendering to be sent to the 3D printer. I will also look into 3D printing filament that we will need to purchase to use the 3D printers in Techspark.

Posted on

Nora’s Status Report for 3/11

I spent a majority of the week working on the design report with my team members. As a result, I was not able to conduct thorough testing of the scheduling. I did implement additional code to handle rests in the music as well as setting the GPIO pins low when a note is finished playing. This will handle cases where the same note is played twice or more in a row.

Since I was not able to test the timing accuracy of the scheduling, I am slightly behind schedule. However, I plan to catch up by allocating time tomorrow to fully test out the new code and verify the timing accuracy of the current code using a metronome.

Next week, I plan on modularizing the GPIO pin mapping and working with Aden to integrate with the actuators. I will need to check if the timing still holds once the hardware is involved, since one bottleneck we encountered earlier on was that time it took to activate the solenoids limited our max frequency.

Posted on

Team Status Report for 3/11

As we noted in the design report, the main risk we face in our project is not being able to achieve exact accuracy in meeting timing and rhythm requirements that we laid out in our design requirements. While we have confidence in our ability to replicate common tunes, we foresee challenges for songs and pieces that are very complex in rhythm with very fast playing of notes. To mitigate this risk, we will not support playing songs that have notes with note values smaller than our system can support. In addition, we will preselect a set of songs for demo that we can ensure are playable by our system.

After establishing our model for the system architecture, each of us has been tasked to focus on different subsystems. Rahul, who is working on the GUI, will have to get more familiar with the syntax of pygame to flesh out the application design. Nora, who is emphasizing efforts in the note scheduling, will have to ensure type safety of the data as it is handled in the music21 data classes. Aden, who is leading the building of the accompanyBot player structure, may have to get familiar with the Solidworks and also learn circuit breaker operation to implement that portion of the circuit.

So far, we have been meeting the milestones on time. Our parts have all arrived as well, so we are not worried about any delays due to shipping. Thus no changes were made to our existing schedule or system.

Posted on

Aden’s Status Report for 3/11

In the past two weeks, I primarily worked on the design report. I spent the last few days of the week before spring break writing and editing until we were prepared to submit. Aside from writing the report, I have put together a circuit of five solenoids controlled by an Arduino to determine roughly how much power they consume. Five solenoids turned on at once is our worst-case scenario, so I wanted to ensure that our calculations were correct and that they it does not exceed 60W. I found that if we power the solenoids with their maximum voltage (12V), the five solenoids only draw 4A of current, placing their maximum power consumption around 48W. Additionally, I was testing it with PWM signals, and the solenoids draw even less current. If we are strapped for power, we can use PWM signals instead.

(The image is a bit blurry, but it says 12V and 4.009A.)

This progress is on schedule with our most up-to-date Gantt chart. After spring break, I will begin to put some serious thought into the design of the structure that will hold up the solenoid circuit.

In the coming week, I plan on having a well-thought-up sketch of the chassis. Additionally, since we were thinking about 3D printing the structure, I would like to play around a get a mockup on Solidworks. My goal for the week is to provide something substantial for the course staff on what our chassis will look like.

Posted on

Rahul’s Status Report for 3/11

Most of my time on the project this week was spent writing the design review report with my team. In addition to writing that paper, my task in the Gantt chart was to create the UI mockup for the accompanyBot application. Early on, I had made an initial design for the proposal. I decided to make modifications on that to make the colors more aesthetically pleasing as well as make the functionality more feasible.

For reference, this initial mockup had highlighted the note or rest that was currently being played or in queue on the accompanyBot device:

This functionality would require additional machine learning and synchronizing between the subsystems that would deviate from the rest of the project. Additionally, this design was lacking the tempo modifier that was specified in our use case requirements.

For our design review report, I updated our UI model to this version:

The new design incorporates a dark theme which will have less strain on users eyes, the tempo modifier which will be dynamic and prevent input speeds which are too fast, and a current measure display to replace the highlighted note(s) of the previous design. The current measure will be feasible as music21  allows image generation from segments of measures from the original score. So for a score with 100 measures, the system will create and store 100 small images of measures in local memory.

Inevitably, more changes will likely be made to the front-end design come implementation. To account for this, I decided to get started learning the pygame documentation and begin implementing. For now I have developed a proof of concept display:

Now, I know where to look to create the precise visuals such as rounded corners or underlined font. The main concern with visuals is that I found pygame’s image rescaler to very much degrade overall image quality. This results in missing music staff lines when displayed. I will likely delegate rescaling and other poor operations to be performed as OS jobs.

I will have to clean up the app code into modular sections to continue development, but so far I am in a good position to hopefully finish the bulk of the application by next week. Once we start integrating my team and I will have to figure out PyUSB and our data transmission protocol between subsystems so that I can successfully send signals to the RPi, and Nora and Aden can implement the reception of these signals to complete the control of the accompanyBot.

 

Posted on

Nora’s Status Report for 2/25

This week, I completed the first pass of the note scheduling on the Raspberry Pi using the music21 library to parse the notes, and the changes are now pushed to our new GitHub repository. The parse function is able to successfully convert XML files into Stream objects which hold hierarchical information on the Parts, Measures, and Notes (also music21 object types). Based off this info, I was able to extract values for the tempo and beatTypes which I used to calculate the number of ms each quarter note is worth. Dimensional analysis of calculation is shown below:

Additional Resource

From there, we can then scale it by our smallest note value to get the time delay that we will increment between each “timestamp” (a slice in time in which we can determine which notes are active to tell which keys are pressed or not) For simplicity, the current code assumes the smallest valued note we can play is a 16th note. This can be easily turned into a variable later on.

The function that does the main chunk of work is called

schedule

which iterates through the notes stored in the Stream and sets the bits in a bitmask based on the note being played (using mappings expressed through two changeable dictionaries) and then updates the notesToPlay dictionary that matches the bitmask to different timestamps.

As mentioned in the design review presentation, I utilized the pigpio library for controlling the GPIO pins simultaneously by setting and clearing a bank of bits. To visually see the results of the scheduling, I attached the GPIO outputs to LEDs arranged in a keyboard-like fashion. Pictured below is the setup, as well as the measured BPM from playing a simple quarter note C scale at 60 bpm. The code seemed to work pretty well!

Additionally, here is a video of it playing the following chromatic scale. It appears to capture the different note values pretty well.

The task I was scheduled to complete for this week was to “convert XML to scheduled GPIO inputs using music21.” As discussed above and with some help from Aden, I have met this goal satisfactorily, although I will need to work on ironing out a few extra details such as playing successive notes and accounting for rests. Therefore overall I am on schedule.

For the next week, I plan to continue improving the scheduling algorithm and conduct more thorough testing beyond using the built-in GuitarTuna metronome. However, I do anticipate spending a decent amount of time dedicated to writing up the Design Review Report with my teammates, so I may not be able to spend as much time on these tasks.

Posted on

Aden’s Status Report for 2/25

This past week I ordered the remaining parts, which included 14 solenoids and 6 transistors. This puts us at around half of our budget since solenoids that produce enough force for our purpose are expensive. Moreover, I ordered more parts than we needed to mitigate the risk that some parts could break when putting the project together. Hopefully, they will arrive soon, and I can assemble all the circuitry required for depressing the piano keys. Meanwhile, I have been helping Nora with the microcontroller portion of the project. We have been working on getting a proof of concept where we take a converted sheet of music from the OMR, parse it into a python object using music21, and schedule the notes on the correct gpio pins that will be connected to our solenoids. Instead of solenoids, we have been using LEDs in the configuration of piano keys to get a visual of which keys will be pressed by a solenoid at a given time. We successfully got the LEDs to light up in the order of a C and Chromatic C scale. Ultimately, we will need to refine the scheduling algorithm further to handle things like rests, chords, successive eighth notes, etc., but getting it to do a basic C scale was a big step. Additionally, when moving from LEDs to solenoids, we will have to account for factors that do not come with LEDs, such as the physical limit that the solenoids can move back and forth. Overall, aside from being unable to get my hands on the remaining parts, it was a productive week in terms of moving toward our final goal.

Not being able to receive our parts this week did not put me behind my current schedule. Although it would have been nice to get everything here this week, on our Gantt chart, I scheduled next week for building the circuitry we will need for the final version of our project. I would still like to keep some momentum going, so while I’m building the circuitry next week, I will also be thinking and drawing out rough sketches of what will be holding everything in place over the piano.

As I have already stated, next week, I hope to put together the circuitry for the final version of the project. Obviously, it may not be exactly what we need at the end of the semester, so I will not be soldering anything yet. Still, I hope to put something together so that we can integrate the microcontroller and solenoids to get an accurate feel for the timing. Additionally, I would like to aim for a rough design of our final chassis. This does not have to be what we end up going with, but just something to think about while I put together the circuitry.

Posted on

Team Status Report for 2/25

This week, we made it past the Design Review Presentation and have entered a major portion of the development stage this week. Our team’s GitHub has been set up, and initial contributions have been made with respect to the OMR parsing scripts and Raspberry Pi microcontroller scripts. Additionally, we discussed and moved forward with ordering the remaining 14 solenoids and 6 NMOS transistors to meet our hardware specifications.

Integration of the accompanyBot apparatus with the raspberry pi may have a hidden difficulty. Through Nora’s initial GPIO code, we were able to see that LEDs and smaller components respond more quickly compared to the bulkier inductance based solenoids. This may affect our ability to reach the requirement of 6 key presses per second. To mitigate the risk of not meeting this requirement, we may need to rescope the threshold metric so that the hardware is able to meet it. No further changes were made to our design and requirements.

Teaming

One strategy that we have employed for working together effectively is maintaining constant communication. Apart from our Slack channel, we have a group chat where we discuss issues and different ideas for our respective aspects of the projects. For example, this past week we made sure to get one another’s approval before ordering the remaining parts for our build. We also meet up at least once a week outside of class time to update each other on our individual progress, write our team reports together, and work on any deliverables that we aimed to complete that week. Furthermore, we have made the following adjustments to fill in gaps related to challenges that have come up:

  • Rearrangement of scheduled tasks – Due to the introduction of music21 to aid in our scheduling process, we adjusted the work assignments so that more time could be allocated to familiarizing ourselves with the library. Additionally, Rahul joined Nora in setting up the notes scheduler algorithm.
  • Waiting for parts to arrive – While the remaining solenoids and transistors are being shipped, Aden has been helping Nora with her microcontroller scheduling tasks. This was anticipated and accounted for before ordering parts.