Posted on

Danny’s Status Report for 10/26

For this week, I spent a good amount of time doing the ethics assignment. Aside from the ethics assignment, I have created the Django project that I will be using to slowly implement the feature and test my implementation. Currently, I have looked into understanding how video files will be displayed to our users. We are currently not planning on streaming the videos to our users which makes the project easier to work with. After some research, my current plan is to store the video files on the local file system on the RPi. If needed, we can attach an external hard drive for better performance and reliability. The metadata for these videos will be stored inside the database. This will contain information such as the number of videos for a particular user, the file path to find the specific videos, the date when the video was processed, etc. I have started writing some rough code that will model this behavior in my Django project but ran into some slight issues that I have not fully debugged. This will be part of my work for next week. Lastly, we received the micro SD card so I booted the RPi and set up the necessary components to run the web server on the RPi. Once the code for the web framework has been written then it can simply be run on the RPi with minimal issues ideally. 

My progress is currently still behind but I believe I am slowly catching up. One thing that will force me to catch up quicker is having a stricter deadline. I am currently planning on having some basic functionality working for the web app before we meet Professor Dueck’s students again. This will give me a more definitive goal to complete which I believe will help me catch up.

For next week, I hope to get some of the basic functionality working on the web application and running on the RPi. At the bare minimum, I want to figure out how we will interface between the two RPis and implement the start button feature. This would allow us to begin controlling our project through the web application and would be a good starting point for the rest of the web application. For a further goal, I would like to further look into how the video files will be stored and displayed to our users. I believe I got most of my questions answered but I would like to have some sort of example functioning by the end of next week.

 

Posted on

Shaye’s Status Report for 10/26

This week I focused mostly on the tension algorithm. During our previous testing session before fall break, we noticed that our algorithm isn’t great for detecting tension in scales/ Hanon. In layman’s terms, our algorithm doesn’t work when the fingers are closer into the hand/ aren’t reaching as far. 

During our session this week, I tried to mess around with values in our current algorithm to produce different results during scales/ Hanon and had little success. One specific issue did come up consistently with these two exercises—the algorithm would call playing not tense once the pianist has been playing for long enough, regardless of whether they were playing tensely or not. As a result, I have been rewriting/ writing several versions of the algorithm. One version I’m optimistic about automatically records a new neutral vector every so often, hopefully solving the issue above. I’ve also asked Dueck’s students to come up with some features of tension for me to incorporate into other versions of the algorithm. I’ll continue working on these versions this week. Note that this issue doesn’t come up with chords and arpeggios. 

I also recorded video snippets to test with during the session. I’ll be writing some test routines to help resolve the algorithm issues.

I’ll also be working on reorganizing the code in the coming week. Mainly, I’ll be separating out the main camera landmark routine & tension detection. Currently, tension detection is embedded into the landmark detection loop. Separating the two would allow for multithreading, potentially enabling us to run our system faster. The code will also be more modular & easier to follow. 

Currently, I’m not behind on writing/ testing the CV/ tension parts. However, our group is starting to fall behind on integration—I’ll be supporting Jessie with RPi setup & integration as needed. 

Videos from this week’s testing are here.

Posted on

Team Status Report for 10/26

General updates:

  • Team trip to Home Depot to buy parts to connect the webcam gooseneck to the tripod. The connecting part of the gooseneck was a wider tube (⅝”) and not threaded while the connecting part of the tripod was a smaller threaded screw (¼”) so it was difficult to find a connector for the 2. However, with assistance from the Home Depot staff we landed on using multiple spacers. We screwed the tripod connecting part with the help of a heat gun to create threads in the spacers. In the future, we might have to glue the spacers to the gooseneck tube to make sure it stays in there.
  • We met with Dueck’s students and tested our newly built stand. Images are attached below. The stand worked great! It was stable (didn’t require a counterweight) and was tall enough to capture all the keys on the keyboard. Additionally, the gooseneck was able to stretch enough horizontally such that the pianist’s head did not block the view of their hands. The webcam’s USB cable was a bit short so it was inconvenient to hold the laptop close to the camera stand. Additionally, the USB cable kept bumping into the pianist’s head. We have ordered velcro for the purpose of wire management and a USB extension cable. 
    • We also gather video of scales & Hanon specifically to test with. More details can be found in Shaye’s report 

  • As per Joshna and Prof Bain’s suggestion we met with Varun to ask for advice on using the KV260. More details can be found in Jessie’s report. 

Design Changes: 

Based on the feedback we received from Varun, we have decided to drop using the KV260 and commit to pivoting to the RPi with an accelerator to accelerate our hand landmark model. 

Updated Schedule:

We are slightly behind our originally planned schedule of finishing the CV and FPGA integration by the coming Wednesday; additionally, our plans have shifted to using the RPi instead of the FPGA. Here is our updated schedule with the RPi:

Due 11/01 – 

  • Jessie: finish moving the model to RPi and research how to connect the buzzer and display and web app hosting RPi using UART 
  • Shaye: write many versions of the tension detection algorithm for RPi (see Shaye’s status report)

Due 11/06 – (next meeting with Professor Dueck’s students)

  • Shaye: finalize tension detection algorithm using output of RPi model (see Shaye’s status report for more)

Due 11/08 –

  • Jessie: finish implementing buzzer feature and set up UART connection with web app hosting RPi

(11/08-11/11) – buffer time

Due 11/11 – start full system integration

Posted on

Jessie’s Status Report for 10/26

This week’s tasks:

  • I started the week with the intention of putting the model onto the KV260. I downloaded the model and dataset (for the quantization step) suggested by Shaye. I parsed through the dataset and selected a subset that I thought would be best for the quantization step. Note that the model we chose is not same the MediaPipe model Shaye has been using. This is because the Vitis AI workflow is only compatible with Pytorch and TensorFlow files while the MediaPipe model is a TFLite file. For now, we opted to use the Blaze model instead which is in the form of a PyTorch file, but planned to look into conversion scripts in the future if the accuracy of the Blaze model was too low. 
  • I met with Varun to discuss our concerns and difficulties while using the KV260. We learned that using the KV260 is unfavorable to the accelerated Raspberry Pi for our specific application.
    • The Raspberry Pi has a more powerful accelerator and processor. The  RPi can do 13 tera-operations per second (TOPS) while the FPGA can do around 1 TOP. 
    • The Raspberry Pi is easier to work with because there are more resources online. We found someone using the RPi with our exact model application who achieved a frame rate of 22-25 fps. The KV260 would take a lot of effort to reach our target frame rate (if it’s even possible). Varun said that the frame rate of processing the Restnet 18 model (from the quickstart tutorial), which I found to be 15-24 FPS from doing the tutorial a previous week, is likely the maximum capability of the KV260. 
    • Varun mentioned the following benefits of using the KV260; however, none of them apply to our project
      • The KV260 is more power efficient than the RPi. We plan to have users plug the device into wall power (no battery necessary), so this is not a concern for us. 
      • The KV260 has more I/O. It has more pins that can be used for anything while the RPi just has a small amount of GPIO pins and some USB ports. However our application has a limited number of I/O (display, buzzer, webcam, web app hosting RPi), so the additional I/O pins that the KV260 provides are not necessary for us. 
      • The KV260 can be useful if there are multiple things that need to be accelerated. However, our application only requires the model to be accelerated. Varun believes that software should be fast enough to run the tension detection algorithm. 
  • I read through the documented RPi workflow for putting MediaPipe’s hand landmark model onto the accelerated RPi to prepare to do it myself next week. 
  • I also worked on the ethics assignment that was due this week. 

Schedule:

I failed to complete last week’s assigned deliverable because we decided to shift from putting the model onto the KV260 to putting the model on an accelerated RPi. I’m currently behind schedule as Shaye and I originally planned to finish the FPGA/CV integration by the coming Wednesday; however, due to the pivot from KV260 to Raspberry Pi, our schedule has changed as noted in the team status report. 

Next week’s deliverables:

Next week I plan to integrate the MediaPipe model onto the RPi. I should also have a rough idea of how to implement the buzzer and display feature. I will also look into how to set up UART to communicate with the web app hosting RPi. 

Posted on

Shaye’s Status Report for 10/20

I spent the week before fall break focusing on testing the tension detection with Professor Dueck’s students and working on the design report. 

During our last session with the musicians, I had each musician play a variety of technique exercises including scales, chords, arpeggios, and Hanon exercises with relaxed and tense wrist positioning. Generally speaking, the default threshold values for tension detection worked on exercises that require more wrist movement (arpeggios & chords), but worked inconsistently with scales and Hanon. Once I tuned the values a little bit, the algorithm was more accurate with identifying tension with scales and Hanon. I’ll work more on finalizing these values this coming Wednesday. I’ll also standardize the set of exercises for the pianists to play while we’re working with them. 

We did run into a minor issue with the setup while testing. At times, the musicians’ heads would accidentally get in the way of the camera and block out a hand, causing the program to crash. I’ll add in more robust error catching prior to this Wednesday so that we’ll experience less crashing while testing. Additionally, we didn’t have a proper connector between our gooseneck and the tripod last session—getting this connector will help steady the camera feed and may also help resolve the issue. 

I’m still currently on schedule. In addition to what I’ve mentioned above, I’ll also begin running the CV pipeline on an RPi as a backup and assist Jessie with porting our existing CV pipeline as needed. 

Some clips of testing from last week are in this folder. 

Posted on

Team Status Report for 10/20

General updates:

  • Worked on and finished the design report (the vast majority of our time this week was spent here).
  • Worked with musicians on 10/9 to test out rough camera setup & tension detection algorithm—see Shaye’s status report for more info.

Product solution considerations: 

A was written by Jessie, B was written by Danny and C was written by Shaye.

Part A: 

We hope our product will allow pianists of all skill levels, from hobbyists and beginner students to professionals, to protect themselves from hand and wrist injuries related to piano playing– all players can benefit from injury prevention. The product’s utility, though intended for students with access to a teacher, could also be applied to those who are self-taught. However, the self-taught student would also have to self-learn how to identify correct positioning in order to properly set the initial calibration. Additionally, our system does not rely on a laptop to host our system, so users with only access to a phone or a tablet can also protect themselves. The system will also have simple and intuitive features, so it does not require the user to be technologically advanced. 

Part B:

Our product will hopefully encourage people to either learn or continue playing the piano. Whether you are completely new to the piano or perhaps have been injured in the past, our product will ensure that these players are avoiding positions that can cause injury. We hope that the peace of mind our product will provide will lead to an increase in piano players. If the amount of pianists increases due to our product then we believe that we are contributing to a growing and deeper culture. While we are targeting piano players, we believe that a more musically inclined population is able to both better appreciate the culture we currently have but also make contributions to the culture we all share. 

Part C: 

We account for environmental factors with our system in two main ways: by using less power-hungry hardware, and by decreasing overall medical interventions for wrist strain injuries. In terms of hardware, our FPGA-based system would use less power than a Jetson/ GPU based system, minimizing power consumption during longer practice sessions. If our proof-of-concept becomes more widely adopted, this difference in power consumption would have a large impact on reducing overall energy waste from our system. 

Additionally, by preventing wrist strain injury in pianists, our product will decrease the amount of medical resources spent attending to those injuries, thus reducing the environmental impact of those resources. This includes a variety of healthcare items, ranging from single-use wrist wraps to energy spent on wrist imaging. 

Posted on

Danny’s Status Report for 10/20

Most of my time this week was spent on the design report. I underestimated how long the design report would take us so there was little progress in terms of the web application. However, the design report helped solidify some of the ideas we had for our project. I have a deeper understanding of our overall project and have a clearer understanding of what we still need to do. The web application is also more fleshed out from working on the design report and it is more clear to me what parts I need to work on in order to complete the web application. Additionally, on Wednesday we met with the pianists again. This time it was a different group of pianists who were participating so we polled them on the UI and any web app features they might want. They seemed to really want to have more control over the different features and wanted to toggle certain features off and on. They were also interested in potentially having different types of feedback other than audio but we have not decided if we will pursue that route. The discussion with the pianists was helpful because it further clarified what they are expecting from the web application.

My progress is slightly behind schedule. One part of this is that we used the micro SD card that came with the RPi for the FPGA. Thus, I could not work on setting up the RPi to host the web application. We should be receiving another micro SD soon so I should be able to make some progress on that front. I should be able to catch up quickly as I have spent time while working on the design report to understand what I have to do. Setting up the RPi should be a fairly quick process. I would still need to work on the web application itself but that should not take a long time either. Overall, I think I am slightly behind but that it is not a major issue and that I should be able to quickly catch up.

In the next week I hope to set up the RPi for the web application hosting and to start working on the web application itself. This will involve using the Django web framework to begin writing out the different parts that we will need for our web application.

Posted on

Jessie’s Status Report for 10/20

This week’s tasks:

  • Worked on the design report, largely focusing on the FPGA-related and testing sections; however, writing and editing were collaborative and done synchronously.
  • Sent an email to AMD’s Andrew Schmidt as a follow-up to our meeting last Wednesday. I explained the main ideas of our project as they relate to what we plan to do on the KV260 and listed our concerns, some of which include whether meeting our target frame rate is feasible, fitting the model onto the FPGA, and advice on how to implement the kinematics portion on the FPGA.

Schedule:

I’m still on schedule, though next week I will have to make a lot of progress towards the CV and FPGA integration. I was unable to write a rough draft version of the Python kinematics code using the Vitis AI API as I planned last week as the design report took up the 12 hours I allotted towards capstone. 

Next week’s deliverables:

In order to stay on schedule, though ambitious, I plan to work closely with Shaye to finish putting the MediaPipe pose detection model onto the FPGA and write a rough draft of the kinematics code. These plans might change depending on the advice we receive from Andrew.

 

Posted on

Danny’s Status Report for 10/05

This week I spent a decent amount of time practicing for the presentation I had on Wednesday. I spent time going over it both by myself and with Shaye and Jessie so that they could give me feedback on the presentation.

I also spent some time helping Jessie with the Zynq UltraScale+ tutorial: https://xilinx.github.io/Vitis-AI/3.0/html/docs/quickstart/mpsoc.html. I mainly did the first half of the tutorial of setting up the necessary software that is needed and making sure the target was up and running correctly. There were a couple issues that we had while going through the tutorial. There were a couple of minor issues that got in the way such as incorrectly flashing the SD card and not having the correct micro USB to USB A cord (we only had a charging cable so had to run to the store to get one). Another issue was that the Kria KV260 needs internet connection in order to download certain packages but it can only access the internet through an ethernet port. I had to figure out how to provide internet to the Kria KV260 using the ethernet port on my laptop as we had no easy access to a router. After we solved these issues running the example model in the first half tutorial was relatively easy to do. Once the target was running and setup, Jessie took over and completed the second half of the tutorial.

Lastly, for the web apps I did not do as much work as I would have wanted on it this week. I was caught up with different things in capstone and in my different courses. However, I was still able to take a look at how different web app examples to get a better understanding of how Django works and is typically used.

I think I have not progressed on the web apps as much as I would have wanted. I didn’t expect this week to be as busy as it has been so that was an oversight on my part. I am not too worried thought because we will have more work sessions coming in the following weeks so I believe I should be able to catch up the. Additionally, there is no major urgency to finish the web app (before meeting Professor Dueck and her students) so I think it is acceptable to be slightly behind. Also, I think it was more important to help with the FPGA as that is a priority for us as we want to make sure we are able to get the CV models we want working on it ASAP.

Next week, I hope to catch up on the work that I was supposed to do this week for the web app. Additionally, I believe we should be obtaining another SD card for the RPi so I should be able to set that up and perhaps set up a test server on the RPi to understand how it works.

Posted on

Shaye’s Status Report for 10/05

This week I worked on fleshing out the tension detecting system I detailed last week. This was a little more involved than I thought it would be—the function ended up consisting of the following pseudocode: 

  1. Recording x number of past wrist angles
  2. Compute difference between each of the angles in the signal 
  3. Convolve the signal with a sliding window of 1’s to compute average range over that window
  4. If the result of the convolution is smaller than a threshold, then not enough wrist angle change has happened and that window is labeled as tense
  5. If the result of the convolution is larger than a threshold, then that window is labeled as not tense 
  6. If over half of the windows are tense, the function returns tense
  7. If not, then the function returns not tense. 

The bolded words in the pseudocode are adjustable thresholds throughout the function. After messing around for a bit, I settled on 20 recorded angles, a window size of 3, a convolution threshold of 0.5, and half of the windows being tense as my values. You can see the results of these values in the video linked here. True means tension and false means no tension. 

I’ll be testing the tension detection function on Professor Dueck’s students this coming Wednesday. I anticipate that I’ll need to tune some of the values as this will be the first time I’ll be testing this function on an actual piano. 

Additionally, I worked a little more with the webcam this week. The team report has photos of the webcam setup. Once we finalize the full setup next week I’ll work more with the webcam on the tripod configuration. 

I’m currently still on schedule—I’ve temporarily held back on conversion to give Jessie some more time to scope out the FPGA. We’re unsure if it’s necessary to fully port the pipeline or not yet. We’ll continue to progress on this in the following week. I’ve requested an RPi as a contingency if we’re unable to port the pipeline by fall break.