Nora’s Status Report for 4/29
This past week I presented our final set of slides for the Final Presentation. Thus last Sunday I worked on creating the powerpoint and preparing for the oral presentation. Overall, the presentation went smoothly. We received one question about how we handle pieces where more than five notes at a time to which I answered that they are iterated through sequentially in the scheduling algorithm, and any notes after five are not played. If we have more time after the system is built, we may implement a more complex method that analyzes the chords and schedules accordingly.
During the week, I also worked on adding functionality in the code to prevent the program from crashing when it encounters errors and exceptions. This adds a layer of robustness for demo day in case unexpected errors appear that we don’t catch during testing. Going into next week we plan on conducting more testing of the entire system so that we catch those bugs before our demo.
I also looked into having the program run on startup. I have tried editing the rc.local file as well as placing the program in the init.d folder on the Raspberry Pi. I have not been able to do this successfully, but I will continue trying so that we can just plug in the Pi directly so that setup is more seamless on demo day. However, if it ultimately does not end up working, it will not impact the functionality of our overall system.
Next week, I will work with Aden and Rahul to complete the assembly process of our final accompanyBot and then carry out end-to-end testing so that we do not encounter hiccups during demo day. We will also be working together to complete our final poster, report, and video.
Team Status Report for 4/22
All of us have had success in producing our respective components of the accompanyBot. We are confident that we will be able to showcase the whole project come demo day. That being said, we have not tested the whole system as intended with real users selecting files and clicking buttons for a real robot to press down on keys. This is the biggest risk now, as there could be something that we had not considered which would impair the accompanyBot’s intended functionality. To mitigate this risk, we will aim to complete the integrated system by the start of finals week so that we can spend all of finals week testing the robustness of the system before demo day. Additionally, getting the chassis to house the solenoids without them getting stuck has proven to be more difficult than originally anticipated. This is one of the primary focuses at the moment to ensure accompanyBot performs smoothly on demo day. Another major concern is suspending the chassis over the keyboard because it requires the solenoids to be offset depending on if they press a black key or white key. Once again, this is a top priority for the coming two weeks to ensure that it will work on demo day. Thus we will be reprinting parts and re-iterating on the design as frequently as our personal schedules allow.
No major changes have been made to the system. We have mostly been testing and refining the project. We each got a chance to inspect the app, notes scheduler, and first rendition of the solenoid chassis. Upon inspection, we understood what could be improved before demo day and are readily integrating those improvements. For the app there is room to enhance the useability. For the notes scheduler we will need to ensure that the program on the RPi runs on startup and that any errors that might pop up do not cause the program to crash, since it will be difficult to rerun the program while the system is in production. The chassis needs slightly wider holes and more restrictive walls to hold the solenoids steady and prevent them from getting stuck when they retract. These refinements pertain to our current tasks in the Gantt chart, so no changes to the posted work schedule are to be made.
Nora’s Status Report for 4/22
Since the last status report, I have finalized the serial communication integration with Rahul. One issue that popped up when we were testing the integration was that the Arduino MKRZero we were using sometimes had difficulties connecting to the computer. We had to reset the board often and faced trouble with the bootloader. Thus we decided to switch to an Arduino Uno since it connected more reliably. Since the Uno only had one hardware Serial port, I had to look into a software serial library to do the same forwarding that we had on the MKRZero. This proved to be successful. The tradeoff here was that the Uno is slightly larger than the MKRZero, so we will need to design the casing around this constraint. However, we thought this was a necessary tradeoff since the user needs to be able to consistently connect to the hardware without having to touch the electronics in any way.
This week I have also hammered out additional features that enhance the user-friendliness and robustness of our product. Specifically, I added support for octave checking so that the scheduler takes into account not just the pitch of the note but also the octave it lands in. This makes the music more accurate to the original sheet music. Due to our one octave constraint, the accompanyBot must be placed over a fixed octave throughout the duration of a song. Thus during the scheduling algorithm, I added code to find the octave that is most frequently played. We return this octave number to the local application so that we can notify the user to move the accompanyBot over that specific octave. Then, during the piece, only notes falling in that range will be played. I also added code to dynamically calculate the max tempo of a song based on the song’s smallest note value, its time signature, and the physical limitation of our solenoids (which can only play four notes a second). If a user tries to update the tempo or upload a song that has a higher tempo than this maximum value, then the scheduler will cap the playing at this max tempo and also send the max tempo back to the application so that the user can be notified.
During the week, I also helped Aden with the 3D printing. I offered some suggestions for design changes after the initial prototype finished printing. Next week I will continue to help with the design and construction of the physical system.
Currently I am on schedule according to the Gantt chart. In the coming week, I will work on adding code that catches errors while the program is running, since we have discovered that connecting and disconnecting the serial communication results in certain bytes sent to the UART which are not able to be decoded properly and thus crash the scheduler program.
Team Status Report for 4/8
The main risk we have at the moment is getting the dimensions of the chassis correct. We do not have very accurate measuring equipment, so we had to resort to mostly eyeballing the dimensions of the keys/keyboard using a measuring tape. The measurements appear accurate enough when looking at the Solidworks rendering, but we will not be sure until we print it out. Additionally, placing the chassis on the keys may prove to be a challenge, and we may need to adjust the design of the chassis, but otherwise, the physical portion of the system seems to be coming along smoothly.
One change we have made to our design is the decision to use secure copy through ssh to transfer the XML files to the Raspberry Pi. We decided on this route because we found that the latency for sending one page of an XML file through serial communication was too slow (over 13 seconds) to be acceptable. We tested manually scp-ing files to the Raspberry Pi with success. This may require some extra work in running a powershell script to scp to the Raspberry Pi. We believe this will not incur a heavy cost since one of our team members, Rahul, has gotten some experience writing powershell scripts earlier in the semester. From the user’s standpoint, this may require the user to configure the Raspberry Pi on startup to connect to new networks, but we believe the tradeoff of a one time setup in exchange for much faster file uploading speeds is justified because the user will want to upload many files for all the different pieces they will play during the product’s lifetime.
Another change we have made to the design is to no longer support placing the accompanyBot on any starting key. Originally we wished to have the capability of placing the solenoids over any configuration of white and black keys by adding extra solenoids and remapping the GPIO pins. However, based on the physical measurements and differences between the spacing of the black keys, we found that it is physically impossible to account for that, so instead we will have to fall back on our original constraint of a fixed starting key. However, we will still be able to move the accompanyBot up and down octaves, so there is still the ability to play all the notes on the keyboard.
Lastly, a change was made to the pipeline of how the xml file gets processed. Previously we had intended for it to get immediately shipped off to the notes scheduler. After feedback from our interim demo, we allowed for digital playback of the xml via midi signals before the accompanyBot solenoids receive the information to begin playing. This requires operation of the computer speakers which were previously not factored into our design.
For the interim demo, we created an updated Gantt chart showing the tasks that we will be working on for the last three weeks of the semester. The updated schedule is attached here.
During our interim demo, we were able to successfully show the end-to-end communication for starting the music, pausing the music, and updating the current measure number (both from the computer to the RPi and sending the measure number back so that the GUI could update the state). Here is a video of the current implementation in action.
Nora’s Status Report for 4/8
This week, I continued serial integration with the local application. We were able to demonstrate end-to-end communication at the interim demo which was a promising milestone.
Early in the week, I collaborated with Rahul to confirm how we would define the byte commands sent between the Raspberry Pi and local application. These were the commands we decided on:
“S\n” → Start “P\n” → Pause “C[measure_number]\n” → Current measure number “T[tempo_number]\n” → Current tempo “F[file_name]\n” → File incoming
On the Raspberry Pi side, start and stops received from the serial port and update the state of the scheduler (namely whether or not the piece is paused and if it is started, what time and measure number it starts at). The current measure number received will update the current measure played, while current measure number packets are sent back to the computer after every measure change to let the GUI know to change the measure as well.
Once this specification was in place, I tested the roundtrip latency for serial communication. Using the serial port on my computer, I timed how long it took to send a test command to the Raspberry Pi and receive a response back. This resulted in around a 3 ms delay which was well below the requirements we had for starting and stopping the playing. Using a similar method, we tested sending an entire XML file over USB but found that the latency for one page was already reaching 13 seconds. The time it takes to send longer files would scale linearly, so we decided that serial communication was too slow from a user’s perspective. Thus we are deciding to use scp over ssh to copy files onto the Raspberry Pi.
In addition to running these latency tests to validate the system, I have been running tests as I have developed the scheduling algorithm. In terms of scheduling accuracy, I have been generating custom sheet music that targets specific features, converting them into XML files, and running them through the scheduling algorithm to see the outputs. I ran two chromatic scales with sharps and flats through the system to make sure that all the pins were accurately mapped and that all the possible pitches were activated correctly. I also have one piece that is a repeated quarter note at 60 bpm and another at 120 bpm that I have used to test the tempo accuracy. Using a metronome app, I measured the tempo to match the target tempo. Another piece attempted to play a six note chord to ensure that only five pins max were high at one time.
Currently I am on schedule since our updated Gantt chart has me finalizing the serial communication up until Carnival.
Next week, I plan to finish the serial communication to incorporate opening the files based on the file name sent through the serial port. I will also need to add support for changing the tempo, which should be relatively straightforward since I just need to update the variable that keeps track of the measure duration in milliseconds.
Nora’s Status Report for 4/1
This week, I worked with Rahul on integrating the Raspberry Pi with the computer’s local application. Since the computer will be sending and receiving measure numbers as points of reference for where in the piece to start playing and what part of the piece to display on the screen, I had to restructure the code so that the measure numbers were the keys in the dictionary that holds all the notes. This would make it easier for the RPi to look up which notes to play at specific times. With Rahul’s guidance, I re-implemented the scheduler to be compatible with this data structure change. Thanks to Rahul’s suggestion, I also changed the code so that the RPi loops continuously while checking the system time instead of sleeping unnecessarily as was done in the previous version. I was able to alter the code and get it back to a working implementation.
Another area I worked on was introducing the serial communication to the Raspberry Pi. As mentioned by Rahul, we did not have the required USB to UART converter that would allow us to interface with the RPi’s TX and RX pins. To avoid the extra cost and delay to our schedule that would result from purchasing an adapter, we decided to use Arduino microcontrollers that have built in serial ports. In tandem with Rahul, I experimented with using the Arduino MKRZero which has two serial ports: one between the Arduino and the computer and another to the on-board TX and RX pins. By using these two pins, and having the Arduino read the data from the USB and write to the TX/read from the RX and write back to the USB, I was able to set up the system to send and receive strings between my computer and the RPi. While one risk of this implementation is that there may be added latency, the fact that the serial communication is buffered should allow the Arduino to pass the information through at roughly the same baud rate. Tomorrow I will write a more comprehensive test using the system time to find the round trip latency between sending bytes from the computer and receiving a response from the RPi.
At the moment I am behind schedule since we have not completed the integration. However, we did anticipate that this part of the project would take more time and had incorporated slack time for it accordingly. Thus this is on schedule when considering that slack time.
Next week I will focus on further developing the serial communication. I will be meeting with Rahul to agree upon the command bytes that will be sent for various situations that are required by the application.
Nora’s Status Report for 3/25
This week I worked on adding code to incorporate chord playing support in the RPi. Since the chords are parsed differently from the notes, I had to figure out how to deal with them separately. Another case I had to consider was flats and sharps, since I had to add mapping that mapped equivalent sharps and flats to the same GPIO pin since the name of the note may change due to key signatures and accidentals, but the physical note stays the same. I also worked on making the overall process for scheduling the piece from XML file format to the scheduled notes more smooth.
I am behind schedule according to the previous Gantt chart, but after discussion with Rahul we decided that he would aid with scheduling and integration after his tasks for the GUI are completed. We will
Next week I hope to be able to have the RPi successfully take in inputs from the application. I will be working alongside Rahul to handle communication between the local application and the RPi to handle uploading the XML files, starting, stopping, and changing the tempo. This will put us in a good spot for the interim demo that is during the following week.
Team Status Report for 3/18
One risk that may jeopardize the project is 3D printing the chassis. Aden has not used 3D printing technology in 3 years, so relearning it may take more time than expected. Similarly, he has not used Solidworks in two years. Rahul is less familiar with such tools and will likely spend more time learning once he is done developing the software for the accompanyBot and begins aiding with hardware construction. Clearly, there will be a learning curve to building the chassis. However, if there are significant difficulties with 3D printing then the alternative plan is to use a plank of wood with the solenoids secured on the plank and placed on top of the keyboard. Another concern is that the solenoids are quite heavy, so the support material must be able to withstand the weight. Therefore this would factor into our design choices as well.
No significant design changes were made to the physical portion system. Details are still being worked out in regard to the chassis, but it will more or less look similar to the existing design from the design report. The solenoid circuit also remains the same from the design report. The application design will remain the same as our specification. As for the integration of the app with the RPi, we may use PySerial but if that does not suit our needs in a reasonable amount of time we will rethink this system communication step. Due to slight fluctuations in the timing of the solenoids found during running the RPi code to control the MOSFET switching. Additional calibration may be needed to make sure the tempo meets our system requirement. However, since the variations have resulted in values slightly off from the targets (1bpm difference), we may consider loosening the 100% accuracy measure to allow some tolerance, since we did see Professor Sullivan’s feedback about the metric being a high bar to achieve.
No significant changes need to be made to the schedule. If there do need to be changes, they will take up the slack time that is in place.
Nora’s Status Report for 3/18
This week, I worked on integrating the RPi GPIO outputs with the physical circuitry by attaching them to the gates of each MOSFET. A video of the chromatic scale that we showed in a previous status report being played on the solenoids can be found here
One thing we have not mentioned so far is that, from the rough piano layout seen on the paper, you can see there is an extra solenoid between the E and F spaces. This solenoid is placed in the area where there would be a black key in between the white keys. Another will be placed between the B and C spaces. This part of the design is so that we are able to move the accompanyBot up and down the scale. Since different starting keys of the octave will require different orders of black and white keys, we will just remap the solenoids to the correct keys. Thus we will be able to play more contiguous notes for certain melodies that do not start on a C. The downside of this design is that at any given time, two of the solenoids will be useless. However, the solenoids must be arranged like this if we do not want to have the user have to move the solenoids themselves each time they adjust the starting position.
To make the pin mapping more modularized, I have written functions that change the dictionary that maps the notes to pins based on the starting note passed into the function. This will allow the user to place the accompanyBot at different spots on the keyboard.
I also added additional XML files that can help in testing the tempo and the case where the piece tries to play more than five keys at a time
So far, I am on schedule according to the Gantt chart, although tomorrow I will need to add support for chords since they are not supported in the current implementation due to the fact that I did not realize they are a different data structure in the music21 library.
In the next week, I will work on making the process for playing a piece more streamlined since at the moment, I must manually change the files and rerun the code every time I want to switch the piece. I will also work with Rahul on accepting the user inputs to start and stop.