Posted on

Danny’s Status Report for 12/07

 

For this week, I continued my work of putting the videos and associated graph onto the web application. The graphs are very rudimentary but they display whether you have tension or not over the course of time. I investigated trying to sync the video with the graph but I think it is slightly too complex. I was not quite sure how to implement that and I didn’t think it was essential to the web application so I skipped it for now. I was able to fix all the bugs I had and the videos/graphs are able to be displayed onto the web application now. I also used an HTML video library to have a better display for our videos as the default one lacks some features. There is a small issue where the videos are not able to be played for some reason. I am currently working with the videos provided by the model and I have noticed that some of the videos are being corrupted. I am not sure if the model is not saving the videos correctly or if it is a problem with my code. I am going to test the model with regular videos I download off the internet and work with Jessie if the model is outputting corrupted/invalid videos. Additionally, I worked on improving the instructions so that our users are able to clearly understand what is being asked of them.

My progress is still on schedule. I am not worried about falling behind.

For next week, I hope to fix the video issue I was seeing where the videos were not properly being played. I also will begin testing the new UI with different users. These users do not need to be pianists as they will simply follow the instructions to set the stand up. Additionally, we will have one integrated test for the UI that we have planned out as well. For the UI itself, I will clean up some of the graphs associated with the video that display tension. If I have time, I will also work on improving the look of the web application such as changing the color scheme and moving components but these are not essential and more minor changes if the time allows.

 

Posted on

Team Status Report for 12/07

General updates:

  • At the beginning of the week, the team prepared for the final presentation by finalizing slides and doing some practice runs.
  • Jessie and Shaye worked together to adjust the live feedback code so the tension for 2 hands could be identified. Previously, our tension detection algorithm only worked for 1 hand. For more information, see Shaye’s status report.
  • Jessie continued to work on testing the post-processing and average FPS of various piano-playing videos. For more information view her status report. 
  • Danny continued working on improving the web application and displaying the videos on the website. For more information view his status report.
  • At the end of the week, the team also discussed the content we want to present on our poster. 

 

Unit testing:

 

UI/UX (Clarity of instructions, stand setup/take down)

  • We tasked one of our pianists to run through the instructions on the web application. This involved setting up the stand, placing the stand behind the piano bench and adjusting the camera, navigating to the calibration and recording page for a mock recording session, and then taking down the entire system.
  • We timed how long it took for the pianist to complete each of the steps and they were within most of our time limits. One step we underestimated was how long it takes to set up the stand so we will be working on that.
  • We received feedback afterwards that our instructions were not the clearest so the pianist was not sure what they had to do at each step. They also commented that providing pictures of each component would help clear up the instructions as well.
  • As the pianist was able to complete most of the tasks within time, we are not too worried about making big changes to our system. We have made our instructions more clear and are going to add pictures to our instructions to help our users. Additionally, we are not currently planning on changing anything related to our stand or how setup/takedown is done as changing the instructions is easier and was our main complaint.

Hardware (latency, frame rate)

  • We created a small data set of videos with varying types of pieces and of varying lengths. We used this data set to test both the average frame rate and post-processing time.
  • Frame rate
    • To find the average frame rate for each video, we have an array of the captured frame rate and then average this at the end of the video. The base code already included the code to find the frame rate. It uses a ticker to track time and after 10 frames, will find the frame rate by dividing 10 frames by the time that has passed. We collect and average these values.
    • We found that our average frame rate was about 21-22 fps, which is within the targeted range. Therefore, we made no changes to our design due to the frame rate.
  • Post-processing time
    • To find the post-processing time of each of the videos we create a time-stamp before the frames of the video are written into a video and afterwards.
    • We found that the post-processing time was way below our targeted processing rate of ½ (processing time is half of the video length). The rate that we achieved is around ⅙. However, we plan to test this more extensively with longer video lengths. It’s possible that our code will change if these longer videos fail, which could slow down our processing rate. However, since the rate is so far below our target, we are not concerned.
    • No major design decisions were made based on our findings. However, changes were made to our code based on the fact that the post-processing did not work for longer videos. Instead of storing the frames of the video in an array, the frames were stored in a file. This likely slowed down the frame rate and post-processing time due to the extra time to read and write to files; however, with this change, we are still within the target range.
  • Live feedback latency
    • To test the live feedback latency of our system, we inputted clearly tense and not tense movements into our system by waving and abruptly stopping. Our tension algorithm detects tension by horizontal degree deviation, therefore the waving is extremely non-tense and the abrupt stop followed by a pause is extremely tense. We had multiple iterations of waving and stopping and recorded when the waving stopped and when the audio feedback was produced. The difference between these 2 timestamps was effectively our system’s latency.
    • We found that the system had a latency of around 500ms, which is way below our target of 1 second. These results are from only testing tension in one hand as we are still finishing our tension detection algorithm; however, we expect the latency will go up with 2 hands as with 2 hands in the frame the frame rate drops. We don’t think this drop in frame rate will cause us to go over our target though since the drop is not very big (28 fps to 21 fps). Because we are meeting our target, no changes were made based on these results.

Model (accuracy of angles)

  • Compared Mediapipe to hand measurements & Mediapipe to Blaze. 
  • Was fine, no changes 
  • Graphs of angles & summary of percent match between every output

 

Tension detection algorithm (comparison to ground truth)

  • Run detection pipeline on clips. Compare to ground truth results 
  • Compiled two datasets the our tension form that we sent out—one where at least 4/6 respondents agree, and one where 5/6 respondents agreed
    • We sent out 21 video clips to be labelled
    • The 5/6 dataset contained 7 videos 
    • The 4/6 dataset contained 15 videos
  • Overall, the datasets are very iffy & tension detection is inconclusive. However, because of the mixed ground truth and feedback for a better angle, that could’ve been an approach if we had more time for this project.
  • Preliminary results are as summarized: 

 

Integrated tests:

 

Tension on RPI: 

  • To test whether the tension detection algorithm maintains a high accuracy when run on the RPi, we plan to use the same as tension detection testing from earlier (but on the RPI w/ full system instead). We plan to run this soon and currently have no results for this test.

 

Overall workflow UI/UX testing: 

  • Because we are not quite finished integrating our system, we plan to do this in the near future and have yet to collect results. However, to test our full system, we plan to first have Shaye, our local pianist, run through the workflow to see if there are any major integration issues. Once this works, we will ask a pianist who is unfamiliar with our system to try and go through the workflow to see if the UI is clear.

 

Posted on

Danny’s Status Report for 11/30

For this week, I spent most of my time integrating with Jessie and the overall system. This involved cleaning up the web application. This means designing the UI so it resembles more of a finished product. As a team, we thought about what instructions would need to go on the website. I was the one in charge of putting the instructions onto the website. Additionally, I continued interfacing with Jessie and changed the recording page to be able to pass in arguments to the model. We wanted to pass in a toggle for the display, a volume control, and the rate at which the buzzer goes off. The recording page is now able to properly start the model with the options specified. The user is also able to stop the model from the recording page as well. Once this is done, the model will save the recorded video into the filesystem which can be visible to the user in the analytics page. I also began looking at how to display the videos onto the website. I was asked to look into listing the videos into expandable tabs so that UI looked more clean. I have a rough idea of how to do it and will hopefully have it completed by the end of this week. For some verification, we met with one of Dueck’s students. The other pianists were not able to attend unfortunately. We were able to get some rough testing and verification from this student, however. We had them look at our website and run through the instructions provided. We timed them on how long it took for them to do each task. Unfortunately, it seems that our instructions were not the most clear and that there was some work to do to make our project easier to use. Finally, I have helped my team on the final slides. We drafted what we wanted on the slides beforehand. I then helped create some of the slides such as the block diagram  and software tradeoff slides. Additionally, I created the slide relating to the UI/UX verification which involves the setup, UI and takedown. 

 

I would say my progress is on track. I am not worried about finishing everything before the final demo.

 

For next week, I hope to finish the UI and finish integrating with the rest of the system. This involves updating the instructions and updating how the videos are displayed on the website. As a stretch goal, a more detailed analytics page will be implemented as well. 

 

I came into this project not knowing anything about web applications and how to design/create them. I had some incredibly basic HTML and CSS knowledge from a couple years ago and that was about it. I had to learn how to use the Django framework for my project and how it leads to a website being created. This involved learning how to interface with a database to store metadata information, how to set up a web server on the RPi, and how to create a more clean looking frontend by working with HTML, some small amount of JavaScript and Django. In order to learn these things, I read a lot of the Django documentation. I completed the small Django tutorial to setup a basic web application at the beginning of the semester which helped solidify the ideas for me. Additionally, I read from a lot of Django forums and StackOverflow posts to answer any questions I had about the features I had to implement. 

Posted on

Team Status Report for 11/30

General updates:

  • The team worked together at the beginning of last week to finish preparing for the interim demo. 
  • The team also worked together to write an initial draft of the setup instructions on the webpage. We then met with one of Professor Dueck’s students to test the setup and take down time of our stand using our instructions. We also wrote a Google form to gather feedback on the instructions for the setup/takedown process. 
  • This week, the team worked to create slides for the final presentation; each person largely worked on the slides relevant to their contributions. Jessie wrote the slides related to verifying hardware aspects (frame rate, post-processing time, and latency) and the design tradeoffs between the NVIDIA Jetson, AMD KV260, and an accelerated RPi. Shaye wrote the slides relating to verifying software (tensions algorithm, hand landmark models). Danny wrote the slides relating to verifying and testing the UI/UX (setup time, takedown time and web application)
  • Jessie continued to work on integrating features onto the RPi. She implemented the LED recording-feedback feature Joshna suggested during the interim demo, adjusted/debugged the buzzer feature, and enabled video recording in the live feedback code. She also wrote a Google form to gather ground truth on tension based on some clips we’ve recorded. Lastly, she worked on testing our system’s latency and started writing a script to find the average frame rate and post-processing time for various videos. Refer to her status report for more details. 
  • Shaye continued iterating on the tension algorithm & tracking different aspects of playing. Refer to their status report for more details. 
  • Danny continued working on integrating with the rest of the team. Time was also spent cleaning up the web application and implementing the different features that were asked for. Refer to his status report for more details.
Posted on

Danny’s Status Report for 11/16

For this week, as mentioned in the team status report I helped set up the RPi for use. One concern we had was that the 16 GB SD card we originally had would be too small as we would want to store video files on the RPi. We also had some slight concerns that our model might be too big for the SD card as well. Thus, I helped transfer the 16 GB SD card to a larger 128 GB SD card. As for the web application, I worked on the front end and decided the different pages we will have in the end. This involved trimming a lot of extra unnecessary content that was available on the template. I tried to stick with the bare minimum that we needed in order to ensure that our users will not be overwhelmed or confused. Next, I worked with Jessie to integrate the web application with the model. I created two separate pages, one for calibration and another for recording, that would be used when we interface with the model. For now the calibration page has two buttons that start and stop the small display we have. The recording page will also start and stop a separate script that will start recording from the webcam and store the output video file appropriately. This script was just to test the functionality of our buttons and it will be changed to start and stop the model we have on the RPi in our final product. I then started looking into how to store the video files within the web application. I encountered some small bugs that prevented me from finishing but I believe I have a good idea on how to fix the issue. 

 

I would say that I am on schedule now and I am not worried about my progress. I have completed the work we wanted to do before the interim demo and will be on progress for the final demo.

 

For next week, I hope to be able to show the video files on the website. There was a small issue with displaying it on the website before but I would ideally want to figure it out by the end of next week. I would also want to create the instructions page and figure out how to display the post-processed graphs of the videos as well. Finally, I would like to set up the web server so we do not have to run the command to start the web application. I did it previously but we ended up switching RPis so I would have to redo that as well. As a stretch goal, I would ideally also look into a better way of displaying the videos on the website. I think it does not look the best but obviously it is more important to get it functional at first.

Verification:

We currently have not tested the web application with any of our users. As mentioned in our design report, we plan to test the web application by having them use our website and giving us their feedback. This test will consist of having the users complete a series of tasks that we have outlined for them. We will want to measure how long it takes for each of the users to complete each of the tasks. The exact requirements were outlined in  our design report but each task should not take more than a couple minutes at most. Our users would ideally not get stuck at any step in the process. The measurements we will be taking will be compared to the requirements we have set and would ensure that the users are able to use the web application without difficulty. If any step takes longer than anticipated then we will try to figure out how to make that specific step more accessible and easier for our users. At the beginning and after they completed the tasks, we will ask the users for any feedback on the web application. Lastly, we will most likely poll our users on our overall system and that may include more web application feedback that they would like to give. 

Posted on

Danny’s Status Report for 11/09

For this week, we received the RPI accelerator. Jessie spent most of the time working on it and making sure the model was working. We also received our display that will be used during calibration. I spent some time helping Jessie to work with the display. We were attempting to follow the instructions provided but certain libraries weren’t available to download on the RPi OS version we had and it was unclear if they were required. However, after spending some time we were able to figure out how to properly communicate with the display. Jessie then took over again and figured out how to display the camera feed onto the display for use during calibration. As for the web application, I realized that with my limited experience with web applications it would be difficult to create a web application that looks good or is friendly to our users. Thus, I have decided to modify a Django template to serve our users instead. I spent some time looking at the available options and choosing the template that we would like to use. I tried a couple templates at first and tried to stick to using React as well but those templates would not work for me. I was unsure if I was following the instructions for how to compile and run the server incorrectly or if their template was just broken. After spending some time trying to debug the issue I eventually decided to choose a different template to use. Finally, I have settled on a specific template and was able to get it to compile and work. I will and have been making some modifications to it as we only need a simplified version of the template. Currently, we are planning on having 2-3 pages to use so I have been cutting down on the number of pages that we need. I will then be modifying the pages so that one page is the control page (has a button that runs the model code) and a video page that will display the videos. The video page will still require the same video database idea I came up with so the work I did last week should still be applicable. 

I would say my progress is still on track. I have decided to speed up the way we are doing our front end design so that should not be an issue anymore. The rest of my time will be spent on the backend design so I am not too worried about the schedule.

For next week, I hope to continue working with the template and implementing some basic functionality that will work with the full-system integration by the end of the week. Right now that is looking like a button that will start recording and a page that stores the videos that we have recorded. As a stretch goal, I will be looking into how we will modify the videos to display the incorrect positions. Shaye has suggested creating a graph and putting that below the video to display the tense playing throughout the video.

Posted on

Team Status Report for 11/09

General updates:

  • As a team, we set up the RPi case/fans. We have to take this apart and back it together in the future to attack the display and buzzer. 
  • This week the team largely worked independently. 
  • Jessie continued working on setting up the accelerated RPi. She started the week by registering the RPi with CMU-DEVICE and looking at how to have a program run automatically at boot. She continued to look into putting the model on the RPi, setting up the active buzzer, and interfacing the display. To find more information, refer to Jessie’s status report. 
  • Shaye worked on using footage from this week’s session with the pianists to help inform tension algorithm rewrites. Refer to Shaye’s status report for more info. 
  • Danny continued to work on the web application. Danny decided to refer to a template and modify that for our uses instead of starting from scratch. Refer to Danny’s status report for more information.

Pictures of the RPi Accelerator and the case:

Posted on

Danny’s Status Report for 11/02

For this week, I was going to spend some time figuring out how to interface between the two RPis using UART. However, Jessie spent the vast majority of the time trying to figure it out for us. I spent some time initially setting up the additional RPi as there were some ssh issues but we all worked together to get the RPi connected to the wifi and were able to ssh. We were considering using a hotspot from our phone as the internet connection for our RPi but for some reason that was not working for us and we were unable to ssh. As I mentioned, after the RPi was functioning and we were able to properly ssh into it Jessie took over and spent time looking into how to connect the two RPis. I spent some time figuring out how to store videos within the database. I tried looking into it last week but was unsuccessful because of some misunderstandings I made. However, I figured out how to properly upload, reference and display the videos stored within the database. For my test, the videos were stored on my laptop’s filesystem but in the future they will be stored on the RPi. If we get an additional drive then the videos will be stored on that as well. Additionally, I do not believe we need to add functionality for our users to upload videos so it is unlikely that feature will make it to the final version but it is relatively easy to do if required. 

My progress is still slightly behind but I believe that I am catching up. Again, as I emphasized before, it would be ideal if I was able to implement some basic functionality before meeting with Professor Dueck and her students next week. I believe figuring out the video portion of the web application was a good step towards that goal.

 

For next week, I would like to work with Jessie to figure out how to interface with the other RPi through UART. I would then want to create a simple button on my web application that can use UART to talk to the other RPi. This would be good as it could be a way to show basic communication between the two RPis and how the web application is involved. Additionally, I have figured out how to store the videos in the database but I would like to change how the videos are displayed in a more list-like format. I would like to investigate how to do that by the end of next week as well.

Posted on

Team Status Report for 11/02

General updates:

  • This week the team largely worked independently. 
  • Jessie worked on setting up the accelerated RPi. She looked into putting the model on the RPi, interfacing UART between the 2 RPis, using the buzzer, and ordering a display. To find more information, refer to Jessie’s status report. 
  • Shaye worked on different iterations of the tension algorithm and set up unit testing with the videos from last week. They also organized the code for storing information about hand position information and helped debug buzzer RPi issues. Refer to Shaye’s status report for more info. 
  • Danny continued to work on the web application. He worked on figuring out how to store the video file paths within the database and how to display the videos through our web application. He implemented some functionality to allow users to upload videos but it will probably not be part of the final design. Refer to Danny’s status report for more information. 
Posted on

Danny’s Status Report for 10/26

For this week, I spent a good amount of time doing the ethics assignment. Aside from the ethics assignment, I have created the Django project that I will be using to slowly implement the feature and test my implementation. Currently, I have looked into understanding how video files will be displayed to our users. We are currently not planning on streaming the videos to our users which makes the project easier to work with. After some research, my current plan is to store the video files on the local file system on the RPi. If needed, we can attach an external hard drive for better performance and reliability. The metadata for these videos will be stored inside the database. This will contain information such as the number of videos for a particular user, the file path to find the specific videos, the date when the video was processed, etc. I have started writing some rough code that will model this behavior in my Django project but ran into some slight issues that I have not fully debugged. This will be part of my work for next week. Lastly, we received the micro SD card so I booted the RPi and set up the necessary components to run the web server on the RPi. Once the code for the web framework has been written then it can simply be run on the RPi with minimal issues ideally. 

My progress is currently still behind but I believe I am slowly catching up. One thing that will force me to catch up quicker is having a stricter deadline. I am currently planning on having some basic functionality working for the web app before we meet Professor Dueck’s students again. This will give me a more definitive goal to complete which I believe will help me catch up.

For next week, I hope to get some of the basic functionality working on the web application and running on the RPi. At the bare minimum, I want to figure out how we will interface between the two RPis and implement the start button feature. This would allow us to begin controlling our project through the web application and would be a good starting point for the rest of the web application. For a further goal, I would like to further look into how the video files will be stored and displayed to our users. I believe I got most of my questions answered but I would like to have some sort of example functioning by the end of next week.