Nora’s Status Report for 4/29
This past week I presented our final set of slides for the Final Presentation. Thus last Sunday I worked on creating the powerpoint and preparing for the oral presentation. Overall, the presentation went smoothly. We received one question about how we handle pieces where more than five notes at a time to which I answered that they are iterated through sequentially in the scheduling algorithm, and any notes after five are not played. If we have more time after the system is built, we may implement a more complex method that analyzes the chords and schedules accordingly.
During the week, I also worked on adding functionality in the code to prevent the program from crashing when it encounters errors and exceptions. This adds a layer of robustness for demo day in case unexpected errors appear that we don’t catch during testing. Going into next week we plan on conducting more testing of the entire system so that we catch those bugs before our demo.
I also looked into having the program run on startup. I have tried editing the rc.local file as well as placing the program in the init.d folder on the Raspberry Pi. I have not been able to do this successfully, but I will continue trying so that we can just plug in the Pi directly so that setup is more seamless on demo day. However, if it ultimately does not end up working, it will not impact the functionality of our overall system.
Next week, I will work with Aden and Rahul to complete the assembly process of our final accompanyBot and then carry out end-to-end testing so that we do not encounter hiccups during demo day. We will also be working together to complete our final poster, report, and video.
Team Status Report for 4/29
This past week, we presented our Final Presentation. During the presentation, we explained the technical challenges we faced and the design tradeoffs we made throughout the design process.
Currently, the main risk we face is being able to 3D print the complete casing within the time left before demo. To mitigate this risk, we will be printing the finished casing in smaller sub-parts and assembling them together. This allows for each individual part to take less time to print, meaning we can more easily adjust the design and reprint the part if there are issues with the physical dimensions.
We carried out an entire system functional test from application to solenoids. This worked out fine. We also have conducted tests to verify each subsystem’s functionality. Our results from this testing in previous weeks have helped us to avoid making any new design changes at this stage of the project. The majority of experimental findings was done in determining latency timings as well as OMR parsing accuracy.
These were the methods we used to test specific requirements:
-
- OMR Accuracy → Reconvert back to pdf and compare visually
- Tempo accuracy → Measure tempo with a metronome
- Computer/RPi Latency → Function records system time when start/stop signals are sent and received
- Power Consumption → Measure DC power supply output voltage and current to calculate power
The table below summarizes the results we found through our testing
These metrics show that we pass on all four requirements.
Rahul’s Status Report for 4/29
For the final week of classes, I was able to accomplish the tasks I outlined for myself last week. This includes building a UI accessible cache prompt, enabling a disconnection mechanism from the rest of the system, as well as creating disappearing messages to provide an informational feedback to the user on application events. The cache prompt works by checking to see if the music score that the user is inputting has already been processed. If so, the user is prompted with a message saying whether they would like to use the existing xml or create a new one (which would take a while). Since Audiveris takes several seconds to translate each input page of music into xml, it would be beneficial for the end user to save on computation time here. If no matching xml is found in the cache, the cache prompt is not displayed and the application goes straight to running an OMR job.
Connecting to device was made possible through a large sky blue button in the app, but if there are complications for connecting such as the USB wire coming loose, then the application should recognize and report such things. Previous print statements that I had in the code for when the device connected or python exceptions for the connection breaking have now been replaced by color coded disappearing messages that show as an alert at the top of the app. Additionally, any misinputs such as non pdf/png files or files that the OMR fails to process are also reported in this fading out alert message style. Lastly, just for the sake of having it the “connect to device button” now becomes a “disconnect from device” button after having connected. This was more complex under the hood as I had to clean up the communication threads each time the USB port was closed, and then generate new threads for a reconnection so as not to run into problems writing or reading from a closed port.
Here is a view of the essentially finished accompanyBot GUI showcasing the new cache message along with a fading alert at the top:
This simplicity of the updated display hides away all the complexity from the end user which was the intended goal of this design. To verify the new additions I tried connecting and disconnecting an arduino multiple times, sometimes unplugging, sometimes disconnecting through the software button. Everything there seems to run smoothly. Performance tests integrated with the notes scheduler and OMR software were detailed in previous status reports.
Thats all for my work into this project. Thanks for reading.
Aden’s Status Report for 4/29
This past week I did not accomplish as much as I had hoped due to having multiple exams and assignments due. The main thing I completed was adjusting the 3D-printed model to fit snuggly around the solenoid rather than loosely fitting and getting stuck after retracting. Using this new design/information, I can replicate it across the entire chassis design so that the solenoids do not move as much and get stuck. Aside from this adjustment, I did not accomplish as much as I had hoped to.
In the coming week, I hope to finish much more. Now that classes have finished and I have almost all of my other assignments completed, I hope to be able to dedicate much more time to completing the chassis. Other than replicating the new dimensions mentioned above, I have to add some support to hold the chassis up while it is suspended over the piano keys. Additionally, we went out and bought metal rods so that the chassis could rest on top of them and suspend over the keys. With these two additions, the chassis should be successful in hovering over the keys, and the only thing we need to ensure after that is that it presses the keys when needed. If that is not successful, I plan on making cylinders to extend the solenoids’ reach so they can press the keys. Once all of this is integrated, it should successfully press down on the keys!
I hope to have a new Solidworks model as well as a freshly printed chassis model by Wednesday. If both of these are successful, we should be set for demo day. If not, we will continue iterating up until demo day.
Team Status Report for 4/22
All of us have had success in producing our respective components of the accompanyBot. We are confident that we will be able to showcase the whole project come demo day. That being said, we have not tested the whole system as intended with real users selecting files and clicking buttons for a real robot to press down on keys. This is the biggest risk now, as there could be something that we had not considered which would impair the accompanyBot’s intended functionality. To mitigate this risk, we will aim to complete the integrated system by the start of finals week so that we can spend all of finals week testing the robustness of the system before demo day. Additionally, getting the chassis to house the solenoids without them getting stuck has proven to be more difficult than originally anticipated. This is one of the primary focuses at the moment to ensure accompanyBot performs smoothly on demo day. Another major concern is suspending the chassis over the keyboard because it requires the solenoids to be offset depending on if they press a black key or white key. Once again, this is a top priority for the coming two weeks to ensure that it will work on demo day. Thus we will be reprinting parts and re-iterating on the design as frequently as our personal schedules allow.
No major changes have been made to the system. We have mostly been testing and refining the project. We each got a chance to inspect the app, notes scheduler, and first rendition of the solenoid chassis. Upon inspection, we understood what could be improved before demo day and are readily integrating those improvements. For the app there is room to enhance the useability. For the notes scheduler we will need to ensure that the program on the RPi runs on startup and that any errors that might pop up do not cause the program to crash, since it will be difficult to rerun the program while the system is in production. The chassis needs slightly wider holes and more restrictive walls to hold the solenoids steady and prevent them from getting stuck when they retract. These refinements pertain to our current tasks in the Gantt chart, so no changes to the posted work schedule are to be made.
Nora’s Status Report for 4/22
Since the last status report, I have finalized the serial communication integration with Rahul. One issue that popped up when we were testing the integration was that the Arduino MKRZero we were using sometimes had difficulties connecting to the computer. We had to reset the board often and faced trouble with the bootloader. Thus we decided to switch to an Arduino Uno since it connected more reliably. Since the Uno only had one hardware Serial port, I had to look into a software serial library to do the same forwarding that we had on the MKRZero. This proved to be successful. The tradeoff here was that the Uno is slightly larger than the MKRZero, so we will need to design the casing around this constraint. However, we thought this was a necessary tradeoff since the user needs to be able to consistently connect to the hardware without having to touch the electronics in any way.
This week I have also hammered out additional features that enhance the user-friendliness and robustness of our product. Specifically, I added support for octave checking so that the scheduler takes into account not just the pitch of the note but also the octave it lands in. This makes the music more accurate to the original sheet music. Due to our one octave constraint, the accompanyBot must be placed over a fixed octave throughout the duration of a song. Thus during the scheduling algorithm, I added code to find the octave that is most frequently played. We return this octave number to the local application so that we can notify the user to move the accompanyBot over that specific octave. Then, during the piece, only notes falling in that range will be played. I also added code to dynamically calculate the max tempo of a song based on the song’s smallest note value, its time signature, and the physical limitation of our solenoids (which can only play four notes a second). If a user tries to update the tempo or upload a song that has a higher tempo than this maximum value, then the scheduler will cap the playing at this max tempo and also send the max tempo back to the application so that the user can be notified.
During the week, I also helped Aden with the 3D printing. I offered some suggestions for design changes after the initial prototype finished printing. Next week I will continue to help with the design and construction of the physical system.
Currently I am on schedule according to the Gantt chart. In the coming week, I will work on adding code that catches errors while the program is running, since we have discovered that connecting and disconnecting the serial communication results in certain bytes sent to the UART which are not able to be decoded properly and thus crash the scheduler program.
Aden’s Status Report for 4/22
This week, my main focus has been finally printing out a prototype of our chassis structure that will hold up the solenoids above the piano keys. This has taken so long because we have been trying to sort out the logistics of 3D printing due to the cost of printing on campus. We finally have a prototype that has given me insight into what needs to change for the final solution.
One major issue was the holes that the solenoids protruded from were too small for them to return to. We also gave them too much wiggle room where the walls held them in place. Thus, I increased the diameter of the holes and made the walls tighter around the base of the solenoids. Another issue that we are having with this design is the difference in height between the black and white keys. Therefore, we somehow have to create an offset between the black and white keys so that all of the solenoids are close enough to their respective keys.
Going into the final week of classes, my progress is a little behind schedule. It would have been nice to get this done a week earlier, but we were constrained by logistical reasons.
This week, however, I plan on finishing a new design and hopefully being able to print it out before the week ends so we can squeeze in one or two more alterations that may need to be made before demo day.
Rahul’s Status Report for 4/22
Since there was a gap from the last post, I completed a good amount of the work pertaining my last two tasks in the updated Gantt chart. This includes finishing the communication protocols between subsystems and running some tests for the project requirements. Last report, the missing piece of intercommunication was mentioned to be the ssh/scp format for relaying xml files from the user application to the notes scheduler RPi. This subprocess has been resolved with no issues assuming both devices are present and connected to the campus internet. I had to do some research on how to specify a timeout parameter for making the scp connection in powershell, and incorporated this into the design as well. Otherwise, the application would freeze while trying to connect to ssh for much too long. Additionally, setting up scp required both Nora and I to store the appropriate keys in the ssh_config to avoid having manual password entry every time an xml file is sent.
Regarding the testing, I am mostly done with performance tests. I was able to make qualitative and quantitative measurements for the Audiveris accuracy on reading notes. The quantitative measurements involved me counting the actual number of note heads and rests in a piece, generating the xml from the notes, reconverting this xml back into sheet music, counting the number of correctly placed notes/rests, and taking the ratio of correct markings to total markings. For the most part, results meet the design requirement of 95% accuracy on average for basic to moderate difficulty pieces. This is how the OMR testing results look:
Medium had either 4 note chords in one of the two piano staves and Hard tended to be more of a solo (this is supposed to be the “accompany”Bot) . These pieces were just tested to find the limits of the OMR solution. I also tried taking a picture with my phone of a sheet of music in okay lighting and the OMR seemed to have around 50% accuracy, but this too goes out of the appropriate inputs the accompanyBot should receive. Also note the first two pieces had too many notes for me to manually count so I just took the ratio of time where the song sounded correct after playback through MIDI. Sometimes MIDI playback would expose singing parts generated in the XML as opposed to only piano parts. This would not be problematic for the solenoids device as it ignores the type of part, however, is an issue for the sample audio which Tom suggested I incorporate. I tried correcting the xml to find and replace all singing parts with piano, but as the xml file gets longer, this job requires significant time that slows down the app. Ultimately, the digital player is just a secondary feature and plays more of a debugging role for the user in case something goes wrong once the solenoids start playing.
Additionally, with Nora I ran a latency test from application to scheduler and back, sending play and pause commands while the solenoids were active. Through our tests we determined the maximum full cycle communication time to not exceed 72ms. This successfully meets our 100ms limitation.
Regarding bug testing, I realized there were too many print statements in my code for a finalized project as well as the fact that the serial port from the GUI was never being closed at any point. I want to add in features to the GUI that replace console logs with more user friendly text or buttons, as well as introduce a disconnection mechanism from the UI for the user to disconnect without unplugging the device. This is by no means a setback, as I have had much time to work on the app and believe I can develop these features quickly now over the final week of classes.
Overall I believe my progress is on schedule. I’m looking forward to finishing my last tasks and operating the project on an actual piano.
Team Status Report for 4/8
The main risk we have at the moment is getting the dimensions of the chassis correct. We do not have very accurate measuring equipment, so we had to resort to mostly eyeballing the dimensions of the keys/keyboard using a measuring tape. The measurements appear accurate enough when looking at the Solidworks rendering, but we will not be sure until we print it out. Additionally, placing the chassis on the keys may prove to be a challenge, and we may need to adjust the design of the chassis, but otherwise, the physical portion of the system seems to be coming along smoothly.
One change we have made to our design is the decision to use secure copy through ssh to transfer the XML files to the Raspberry Pi. We decided on this route because we found that the latency for sending one page of an XML file through serial communication was too slow (over 13 seconds) to be acceptable. We tested manually scp-ing files to the Raspberry Pi with success. This may require some extra work in running a powershell script to scp to the Raspberry Pi. We believe this will not incur a heavy cost since one of our team members, Rahul, has gotten some experience writing powershell scripts earlier in the semester. From the user’s standpoint, this may require the user to configure the Raspberry Pi on startup to connect to new networks, but we believe the tradeoff of a one time setup in exchange for much faster file uploading speeds is justified because the user will want to upload many files for all the different pieces they will play during the product’s lifetime.
Another change we have made to the design is to no longer support placing the accompanyBot on any starting key. Originally we wished to have the capability of placing the solenoids over any configuration of white and black keys by adding extra solenoids and remapping the GPIO pins. However, based on the physical measurements and differences between the spacing of the black keys, we found that it is physically impossible to account for that, so instead we will have to fall back on our original constraint of a fixed starting key. However, we will still be able to move the accompanyBot up and down octaves, so there is still the ability to play all the notes on the keyboard.
Lastly, a change was made to the pipeline of how the xml file gets processed. Previously we had intended for it to get immediately shipped off to the notes scheduler. After feedback from our interim demo, we allowed for digital playback of the xml via midi signals before the accompanyBot solenoids receive the information to begin playing. This requires operation of the computer speakers which were previously not factored into our design.
For the interim demo, we created an updated Gantt chart showing the tasks that we will be working on for the last three weeks of the semester. The updated schedule is attached here.
During our interim demo, we were able to successfully show the end-to-end communication for starting the music, pausing the music, and updating the current measure number (both from the computer to the RPi and sending the measure number back so that the GUI could update the state). Here is a video of the current implementation in action.