Posted on

Nora’s Status Report for 4/8

This week, I continued serial integration with the local application. We were able to demonstrate end-to-end communication at the interim demo which was a promising milestone.

Early in the week, I collaborated with Rahul to confirm how we would define the byte commands sent between the Raspberry Pi and local application. These were the commands we decided on:

“S\n” → Start
“P\n” → Pause
“C[measure_number]\n” → Current measure number
“T[tempo_number]\n” → Current tempo
“F[file_name]\n” → File incoming

On the Raspberry Pi side, start and stops received from the serial port and update the state of the scheduler (namely whether or not the piece is paused and if it is started, what time and measure number it starts at). The current measure number received will update the current measure played, while current measure number packets are sent back to the computer after every measure change to let the GUI know to change the measure as well.

Once this specification was in place, I tested the roundtrip latency for serial communication. Using the serial port on my computer, I timed how long it took to send a test command to the Raspberry Pi and receive a response back. This resulted in around a 3 ms delay which was well below the requirements we had for starting and stopping the playing. Using a similar method, we tested sending an entire XML file over USB but found that the latency for one page was already reaching 13 seconds. The time it takes to send longer files would scale linearly, so we decided that serial communication was too slow from a user’s perspective. Thus we are deciding to use scp over ssh to copy files onto the Raspberry Pi.

In addition to running these latency tests to validate the system, I have been running tests as I have developed the scheduling algorithm. In terms of scheduling accuracy, I have been generating custom sheet music that targets specific features, converting them into XML files, and running them through the scheduling algorithm to see the outputs. I ran two chromatic scales with sharps and flats through the system to make sure that all the pins were accurately mapped and that all the possible pitches were activated correctly. I also have one piece that is a repeated quarter note at 60 bpm and another at 120 bpm that I have used to test the tempo accuracy. Using a metronome app, I measured the tempo to match the target tempo. Another piece attempted to play a six note chord to ensure that only five pins max were high at one time.

Currently I am on schedule since our updated Gantt chart has me finalizing the serial communication up until Carnival.

Next week, I plan to finish the serial communication to incorporate opening the files based on the file name sent through the serial port. I will also need to add support for changing the tempo, which should be relatively straightforward since I just need to update the variable that keeps track of the measure duration in milliseconds.

Posted on

Rahul’s Status Report for 4/8

This week I was able to accomplish end to end connection of the accompanyBot with all of my team members. This was very exciting! I also discovered a bug while doing this. If the user opens the insert file menu while Audiveris is processing another file and then hits cancel on the new file menu the program crashes. This has been resolved now. Another issue that I have been aware of but cannot resolve is if Audiveris is busy processing and the user wishes to cancel the process or just decides to close the app preemptively. The Audiveris process dialog will continue running until it fully finishes. Considering that this OMR software is built upon very old toolchains and I already tried many options to get it configured for MacOS at the beginning of the semester, I will not dig into how I can send it a SIGKILL or similar termination signal. The easy work around is just to close the window that appears which terminates the process manually.

Earlier in the week, the interim demo took place. After showing the application, my team’s course advisor Tom suggested that I should add a digital player to demo the parsed music as a midi output before the accompanyBot starts to play. I was able to address this suggestion and implemented the digital player which converts the xml into midi first and then begins playing an endless loop. This loop can be paused and unpaused via the spacebar key.

Here is a demonstration of it playing and pausing (there should be audio playing along with the video)

In terms of completing my side of the serial communication I set up sender code and patched up the receiver thread code to get measure data from the notes scheduler running on the RPi. I also eliminated one piece of data that was initially being sent by the RPi. Previously, the number of measures in the song was getting determined by music21 and sent over the serial connection to the GUI app. I decided to process the number of measures algorithmically from the raw xml structure.

Overall, the project is in a good position. I was slightly worried last week that I would get behind, but after getting things working I believe I am on schedule again. Going into next week, I will have some more time to confirm the workings of the communication between subsystems. In particular I need to set up the ssh command to deliver the xml file from the GUI to the RPi. As mentioned earlier, this is necessary to avoid the one-time heavy load over the serial connection. If this gets done quickly then I can begin stress testing early and identifying problems that the GUI or notes scheduler may need to have fixed.

The tests on the application side will involve many different variations of quality of music inputs, randomization of button presses to break application, manual disconnection of serial cable while app is still running (I think I tried this already, and since I’m using threads for communication, they terminate with an error printed to the console, but the main app thread keeps running fine, so this is not unreasonable behavior I suppose), etc. I anticipate the app to crash when tests fail and will analyze the error logs closely to determine the point(s) of failure. Also, I will try to perform actions that spawn and terminate different threads at a time while checking the task manager to make sure all python processes get cleaned up when they should. Part of this effect on the use case requirements is to maintain the functionality and latency requirement between subsystems. In terms of design, the GUI should be user friendly. So any odd errors that prevent intended usage must be addressed.

 

Posted on

Aden’s Status Report for 4/8

Heading into the final weeks of the semester, and thus this project, I have run one primary test for my part of the project, and that was ensuring the power consumption of the circuitry does not exceed 60 W. This goal has been met since we limited the number of solenoids that can be turned on at once to five, since each solenoid draws about 0.9 A and uses 12 V this requirement has been met and tested. Additionally, if this proves to be troublesome in the coming weeks, the voltage can be lowered, reducing the current since each solenoid doe not need 12 V to operate. I will also be helping measure our maximum tempo since the circuit will be a limiting factor due to its physical characteristics. Hopefully, we will be able to meet our preset quantity, but that is probably unlikely.

In addition to the interim demo, the image above is what I have accomplished this week. I realized that there were a couple of issues with the previous version of the chassis design. One of the major issues was that the black and white keys were not actually aligned properly in my previous design. Another major issue was that we wanted to allow the system to be placed over any octave on the keyboard, but after some more careful measurements, I realized this would not be possible for what we had in mind, so I had to redesign the chassis nearly from scratch. Additionally, the previous design was under the assumption that we would just secure the solenoids to the inside of the chassis, but to plan for problems arising for demo day, we switched to a design that would allow for solenoids to be replaced easily. Overall, I believe this is a much more robust and easy-to-use design, and hopefully, it will be suitable for demo day.

I am still technically on track, but I would’ve liked to 3D print a prototype tomorrow, and I’m not sure if I will be able to since I am currently having difficulties with Solidworks right now. Hopefully, I will be able to figure out my problem with enough time to 3D print tomorrow, but if I don’t, I will try to print something out Monday. I hope I can print it out tomorrow, though.

Next week I hope to have a prototype of the chassis completed, and if it does not function properly, I hope to update the design and print another one before the week ends. The end of the semester is coming soon, so I need to finish this by the end of next week.

Posted on

Team Status Report for 4/1

For our interim demo next week, we will demo the working parts of our subsystems. We currently have a functional sheet music parser and UI on the user input side. We also have the scheduler which can actuate solenoids based on a manually inputted XML file. Our main efforts tomorrow will be focused on integrating these two subsystems, although we cannot guarantee that it will be completed by our demo next Wednesday due to reasons outlined below.

This week, we encountered a roadblock when trying to integrate the Raspberry Pi with the local application. Our team did not foresee the RPi serial communication to the computer requiring extra hardware to convert from USB to UART. Hence, we were not able to make much progress and thus may not have a working integration between the GUI app and the hardware controller come interim demo. We are going to offset the risk of ordering and dealing with new parts by adding an Arduino MKRZero to our system design as a middle controller to pipe information between the computer and RPi. This could pose a risk by increasing the overall latency between app and solenoids. We are accepting this risk with the knowledge that the typical payload of information traveling should be very small, <10 bytes. Additionally, we discussed previously the usage of pySerial library for communication over usb. Python comes with a serial library that has a simpler syntax and seems to be meeting our system requirements. So for now, we are making this minor framework switch.

Next week we will be going into our planned slack time, so no changes will be made to the schedule. All remaining tasks deal with integration so we will be working on integrating pieces of the software to controller to hardware in descending order of urgency.

Posted on

Nora’s Status Report for 4/1

This week, I worked with Rahul on integrating the Raspberry Pi with the computer’s local application. Since the computer will be sending and receiving measure numbers as points of reference for where in the piece to start playing and what part of the piece to display on the screen, I had to restructure the code so that the measure numbers were the keys in the dictionary that holds all the notes. This would make it easier for the RPi to look up which notes to play at specific times. With Rahul’s guidance, I re-implemented the scheduler to be compatible with this data structure change. Thanks to Rahul’s suggestion, I also changed the code so that the RPi loops continuously while checking the system time instead of sleeping unnecessarily as was done in the previous version. I was able to alter the code and get it back to a working implementation.

Another area I worked on was introducing the serial communication to the Raspberry Pi. As mentioned by Rahul, we did not have the required USB to UART converter that would allow us to interface with the RPi’s TX and RX pins. To avoid the extra cost and delay to our schedule that would result from purchasing an adapter, we decided to use Arduino microcontrollers that have built in serial ports. In tandem with Rahul, I experimented with using the Arduino MKRZero which has two serial ports: one between the Arduino and the computer and another to the on-board TX and RX pins. By using these two pins, and having the Arduino read the data from the USB and write to the TX/read from the RX and write back to the USB, I was able to set up the system to send and receive strings between my computer and the RPi. While one risk of this implementation is that there may be added latency, the fact that the serial communication is buffered should allow the Arduino to pass the information through at roughly the same baud rate. Tomorrow I will write a more comprehensive test using the system time to find the round trip latency between sending bytes from the computer and receiving a response from the RPi.

At the moment I am behind schedule since we have not completed the integration. However, we did anticipate that this part of the project would take more time and had incorporated slack time for it accordingly. Thus this is on schedule when considering that slack time.

Next week I will focus on further developing the serial communication. I will be meeting with Rahul to agree upon the command bytes that will be sent for various situations that are required by the application.

Posted on

Aden’s Status Report for 4/1

This week finally got around to playing with Solidworks. I was able to generate a basic rendering of what we had in mind for the chassis design. I am still working out how much wiggle room we have between solenoids, but otherwise, I should be set to 3D print soon.  Unfortunately, I am not finished with the Solidworks rendering and would like to add some additional features to it, so I will focus on that tomorrow. Here is the 3D rendering that I am currently working on:

From my last status update, I am more or less on schedule. I was not able to get a copy of Solidworks until yesterday, and even then, I had to get a new license from the software catalog and sit through an hour of download times. Tomorrow, however, I will be cleaning up the rendering and adding some features to the design to hopefully improve the overall performance of the accompanyBot.

In addition to the work I will be doing tomorrow, I also hope to figure out how to 3D print, how expensive it is, and how to generate the files necessary to print it out. I also hope to help Nora and Rahul with the serial communication if they do not finish it before the week ends.

Posted on

Rahul’s Status Report for 4/1

Early in the week I worked with Nora to devise data structures for our notes scheduler. The primary functionality we determined would be to take a measure of notes at a time and play through them sequentially unless it receives a measure change or pause signal from the app. I then proceeded to try and work out the communication methodology between the RPi and Windows computer. Unfortunately, my team did not have a usb to usb-c cable on hand, so I placed an order request for one. While waiting, I was able to borrow one from Alex, my group’s TA. I was able to connect the RPi to the Windows computer, but this amounted to nothing since the RPi is not a peripheral and does not get recognized immediately. We would need to order a UART to facilitate the data communication. To avoid ordering more parts and having to deal with UART, I decided to just plug in an Arduino into my laptop and test serial communication from python to the Arduino. I had to relearn some of the arduino intricacies but got a reasonable proof of concept working.

The accompanyBot GUI application will have to spawn a thread from the start that communicates with the hardware. This communication thread and the app’s main thread should be allowed to modify specific mutable objects such as the playing state variable or the current measure number. I ran a test to verify this functionality and it was successful .

In the print log from the test, the list [“nothing”, “here”] gets modified to [“winner”, “here”], by the thread doing communication the data “winner” also gets received by the arduino which was configured to only send this data if it received the “hello mr arduino” message. The main thread is printing the list, while the communication thread is printing the accurate sent and received messages.

The experienced engineer might be thinking right now that I will have a lot of debugging to do if do not mention using locks for the shared data. Though I did not try that out in the test (as it was not necessary), I will keep in mind to use mutex locks in the integration.

The next steps will be to implement all of the communication signals that will be necessary. The variety needed could change as we flesh out our app, but for now I can get started on some of the essential ones. This is what I will be working to do, going into next week. And in general for the weeks to come, it will all be integration work. My team and I are facing some design changes that we will outline in the team status report so our Gantt schedule will likely be very volatile in the coming weeks. I feel like I am slightly behind schedule, but I am sure I can complete the necessary work without cause for concern. We have set up the slack time just for this reason and hopefully will have a working product by final demo.

 

Posted on

Nora’s Status Report for 3/25

This week I worked on adding code to incorporate chord playing support in the RPi. Since the chords are parsed differently from the notes, I had to figure out how to deal with them separately. Another case I had to consider was flats and sharps, since I had to add mapping that mapped equivalent sharps and flats to the same GPIO pin since the name of the note may change due to key signatures and accidentals, but the physical note stays the same. I also worked on making the overall process for scheduling the piece from XML file format to the scheduled notes more smooth.

I am behind schedule according to the previous Gantt chart, but after discussion with Rahul we decided that he would aid with scheduling and integration after his tasks for the GUI are completed. We will

Next week I hope to be able to have the RPi successfully take in inputs from the application. I will be working alongside Rahul to handle communication between the local application and the RPi to handle uploading the XML files, starting, stopping, and changing the tempo. This will put us in a good spot for the interim demo that is during the following week.

Posted on

Team Status Report for 2/25

This week we met in pods with other teams to discuss potential ethical issues associated with our project while also providing feedback on our peers’ projects. Some points that were brought up in the discussion were considerations for possible areas that could go wrong and groups of people for whom our product is not designed for. In terms of problems that could arise, if the product fails during a performance, this could result in embarrassment for users. Additionally, if the timing is off then users will have difficulties when they go on to practice and play with other people. Finally, issues with the product could potentially break the piano if the force is too high or harm people’s hands if they stick their fingers under the solenoids. These considerations all pose risks to the success of the accompanyBot.

Since we are in the integration phase now, another big risk to the project is that we encounter roadblocks with joining the different subsystems. To address this risk, we are going to discuss clear requirements for the interfaces before implementing. We are also redistributing the attention of team members to focus on areas that are at risk.

There will have to be more changes to the design of the chassis since the dimensions of the solenoids did not match up with the dimensions of the piano keys. There is enough space on the black keys, but the white keys may prove to be difficult to build around. We will likely have to  offset the solenoids inside the chassis so that they are not directly adjacent to one another allowing them to fit along the length of the octave. Hopefully it does not prove impossible to construct, but I believe it is still viable. There was a minor change to our system, specifically the image displaying done in the GUI application. This required the incorporation of poppler and pdf2image modules for python. The model of our system from the design report still functions the same however.

In terms of our schedule, some changes will be made. In particular, Rahul will be moving attention to getting the notes scheduling finalized. Aden is primarily taking up the building of the chassis, and will be working on this solo next week instead. Nora will be focusing on integrating with the computer with the help of Rahul in the coming week.

This is the change being made to the Gantt Chart tasks for week 8: