Tag: Danny’s Status Reports

 

For this week, I continued my work of putting the videos and associated graph onto the web application. The graphs are very rudimentary but they display whether you have tension or not over the course of time. I investigated trying to sync the video with the graph but I think it is slightly too complex. I was not quite sure how to implement that and I didn’t think it was essential to the web application so I skipped it for now. I was able to fix all the bugs I had and the videos/graphs are able to be displayed onto the web application now. I also used an HTML video library to have a better display for our videos as the default one lacks some features. There is a small issue where the videos are not able to be played for some reason. I am currently working with the videos provided by the model and I have noticed that some of the videos are being corrupted. I am not sure if the model is not saving the videos correctly or if it is a problem with my code. I am going to test the model with regular videos I download off the internet and work with Jessie if the model is outputting corrupted/invalid videos. Additionally, I worked on improving the instructions so that our users are able to clearly understand what is being asked of them.

My progress is still on schedule. I am not worried about falling behind.

For next week, I hope to fix the video issue I was seeing where the videos were not properly being played. I also will begin testing the new UI with different users. These users do not need to be pianists as they will simply follow the instructions to set the stand up. Additionally, we will have one integrated test for the UI that we have planned out as well. For the UI itself, I will clean up some of the graphs associated with the video that display tension. If I have time, I will also work on improving the look of the web application such as changing the color scheme and moving components but these are not essential and more minor changes if the time allows.

 

For this week, I spent most of my time integrating with Jessie and the overall system. This involved cleaning up the web application. This means designing the UI so it resembles more of a finished product. As a team, we thought about what instructions would need to go on the website. I was the one in charge of putting the instructions onto the website. Additionally, I continued interfacing with Jessie and changed the recording page to be able to pass in arguments to the model. We wanted to pass in a toggle for the display, a volume control, and the rate at which the buzzer goes off. The recording page is now able to properly start the model with the options specified. The user is also able to stop the model from the recording page as well. Once this is done, the model will save the recorded video into the filesystem which can be visible to the user in the analytics page. I also began looking at how to display the videos onto the website. I was asked to look into listing the videos into expandable tabs so that UI looked more clean. I have a rough idea of how to do it and will hopefully have it completed by the end of this week. For some verification, we met with one of Dueck’s students. The other pianists were not able to attend unfortunately. We were able to get some rough testing and verification from this student, however. We had them look at our website and run through the instructions provided. We timed them on how long it took for them to do each task. Unfortunately, it seems that our instructions were not the most clear and that there was some work to do to make our project easier to use. Finally, I have helped my team on the final slides. We drafted what we wanted on the slides beforehand. I then helped create some of the slides such as the block diagram  and software tradeoff slides. Additionally, I created the slide relating to the UI/UX verification which involves the setup, UI and takedown. 

 

I would say my progress is on track. I am not worried about finishing everything before the final demo.

 

For next week, I hope to finish the UI and finish integrating with the rest of the system. This involves updating the instructions and updating how the videos are displayed on the website. As a stretch goal, a more detailed analytics page will be implemented as well. 

 

I came into this project not knowing anything about web applications and how to design/create them. I had some incredibly basic HTML and CSS knowledge from a couple years ago and that was about it. I had to learn how to use the Django framework for my project and how it leads to a website being created. This involved learning how to interface with a database to store metadata information, how to set up a web server on the RPi, and how to create a more clean looking frontend by working with HTML, some small amount of JavaScript and Django. In order to learn these things, I read a lot of the Django documentation. I completed the small Django tutorial to setup a basic web application at the beginning of the semester which helped solidify the ideas for me. Additionally, I read from a lot of Django forums and StackOverflow posts to answer any questions I had about the features I had to implement. 

For this week, as mentioned in the team status report I helped set up the RPi for use. One concern we had was that the 16 GB SD card we originally had would be too small as we would want to store video files on the RPi. We also had some slight concerns that our model might be too big for the SD card as well. Thus, I helped transfer the 16 GB SD card to a larger 128 GB SD card. As for the web application, I worked on the front end and decided the different pages we will have in the end. This involved trimming a lot of extra unnecessary content that was available on the template. I tried to stick with the bare minimum that we needed in order to ensure that our users will not be overwhelmed or confused. Next, I worked with Jessie to integrate the web application with the model. I created two separate pages, one for calibration and another for recording, that would be used when we interface with the model. For now the calibration page has two buttons that start and stop the small display we have. The recording page will also start and stop a separate script that will start recording from the webcam and store the output video file appropriately. This script was just to test the functionality of our buttons and it will be changed to start and stop the model we have on the RPi in our final product. I then started looking into how to store the video files within the web application. I encountered some small bugs that prevented me from finishing but I believe I have a good idea on how to fix the issue. 

 

I would say that I am on schedule now and I am not worried about my progress. I have completed the work we wanted to do before the interim demo and will be on progress for the final demo.

 

For next week, I hope to be able to show the video files on the website. There was a small issue with displaying it on the website before but I would ideally want to figure it out by the end of next week. I would also want to create the instructions page and figure out how to display the post-processed graphs of the videos as well. Finally, I would like to set up the web server so we do not have to run the command to start the web application. I did it previously but we ended up switching RPis so I would have to redo that as well. As a stretch goal, I would ideally also look into a better way of displaying the videos on the website. I think it does not look the best but obviously it is more important to get it functional at first.

Verification:

We currently have not tested the web application with any of our users. As mentioned in our design report, we plan to test the web application by having them use our website and giving us their feedback. This test will consist of having the users complete a series of tasks that we have outlined for them. We will want to measure how long it takes for each of the users to complete each of the tasks. The exact requirements were outlined in  our design report but each task should not take more than a couple minutes at most. Our users would ideally not get stuck at any step in the process. The measurements we will be taking will be compared to the requirements we have set and would ensure that the users are able to use the web application without difficulty. If any step takes longer than anticipated then we will try to figure out how to make that specific step more accessible and easier for our users. At the beginning and after they completed the tasks, we will ask the users for any feedback on the web application. Lastly, we will most likely poll our users on our overall system and that may include more web application feedback that they would like to give. 

For this week, we received the RPI accelerator. Jessie spent most of the time working on it and making sure the model was working. We also received our display that will be used during calibration. I spent some time helping Jessie to work with the display. We were attempting to follow the instructions provided but certain libraries weren’t available to download on the RPi OS version we had and it was unclear if they were required. However, after spending some time we were able to figure out how to properly communicate with the display. Jessie then took over again and figured out how to display the camera feed onto the display for use during calibration. As for the web application, I realized that with my limited experience with web applications it would be difficult to create a web application that looks good or is friendly to our users. Thus, I have decided to modify a Django template to serve our users instead. I spent some time looking at the available options and choosing the template that we would like to use. I tried a couple templates at first and tried to stick to using React as well but those templates would not work for me. I was unsure if I was following the instructions for how to compile and run the server incorrectly or if their template was just broken. After spending some time trying to debug the issue I eventually decided to choose a different template to use. Finally, I have settled on a specific template and was able to get it to compile and work. I will and have been making some modifications to it as we only need a simplified version of the template. Currently, we are planning on having 2-3 pages to use so I have been cutting down on the number of pages that we need. I will then be modifying the pages so that one page is the control page (has a button that runs the model code) and a video page that will display the videos. The video page will still require the same video database idea I came up with so the work I did last week should still be applicable. 

I would say my progress is still on track. I have decided to speed up the way we are doing our front end design so that should not be an issue anymore. The rest of my time will be spent on the backend design so I am not too worried about the schedule.

For next week, I hope to continue working with the template and implementing some basic functionality that will work with the full-system integration by the end of the week. Right now that is looking like a button that will start recording and a page that stores the videos that we have recorded. As a stretch goal, I will be looking into how we will modify the videos to display the incorrect positions. Shaye has suggested creating a graph and putting that below the video to display the tense playing throughout the video.

For this week, I was going to spend some time figuring out how to interface between the two RPis using UART. However, Jessie spent the vast majority of the time trying to figure it out for us. I spent some time initially setting up the additional RPi as there were some ssh issues but we all worked together to get the RPi connected to the wifi and were able to ssh. We were considering using a hotspot from our phone as the internet connection for our RPi but for some reason that was not working for us and we were unable to ssh. As I mentioned, after the RPi was functioning and we were able to properly ssh into it Jessie took over and spent time looking into how to connect the two RPis. I spent some time figuring out how to store videos within the database. I tried looking into it last week but was unsuccessful because of some misunderstandings I made. However, I figured out how to properly upload, reference and display the videos stored within the database. For my test, the videos were stored on my laptop’s filesystem but in the future they will be stored on the RPi. If we get an additional drive then the videos will be stored on that as well. Additionally, I do not believe we need to add functionality for our users to upload videos so it is unlikely that feature will make it to the final version but it is relatively easy to do if required. 

My progress is still slightly behind but I believe that I am catching up. Again, as I emphasized before, it would be ideal if I was able to implement some basic functionality before meeting with Professor Dueck and her students next week. I believe figuring out the video portion of the web application was a good step towards that goal.

 

For next week, I would like to work with Jessie to figure out how to interface with the other RPi through UART. I would then want to create a simple button on my web application that can use UART to talk to the other RPi. This would be good as it could be a way to show basic communication between the two RPis and how the web application is involved. Additionally, I have figured out how to store the videos in the database but I would like to change how the videos are displayed in a more list-like format. I would like to investigate how to do that by the end of next week as well.

For this week, I spent a good amount of time doing the ethics assignment. Aside from the ethics assignment, I have created the Django project that I will be using to slowly implement the feature and test my implementation. Currently, I have looked into understanding how video files will be displayed to our users. We are currently not planning on streaming the videos to our users which makes the project easier to work with. After some research, my current plan is to store the video files on the local file system on the RPi. If needed, we can attach an external hard drive for better performance and reliability. The metadata for these videos will be stored inside the database. This will contain information such as the number of videos for a particular user, the file path to find the specific videos, the date when the video was processed, etc. I have started writing some rough code that will model this behavior in my Django project but ran into some slight issues that I have not fully debugged. This will be part of my work for next week. Lastly, we received the micro SD card so I booted the RPi and set up the necessary components to run the web server on the RPi. Once the code for the web framework has been written then it can simply be run on the RPi with minimal issues ideally. 

My progress is currently still behind but I believe I am slowly catching up. One thing that will force me to catch up quicker is having a stricter deadline. I am currently planning on having some basic functionality working for the web app before we meet Professor Dueck’s students again. This will give me a more definitive goal to complete which I believe will help me catch up.

For next week, I hope to get some of the basic functionality working on the web application and running on the RPi. At the bare minimum, I want to figure out how we will interface between the two RPis and implement the start button feature. This would allow us to begin controlling our project through the web application and would be a good starting point for the rest of the web application. For a further goal, I would like to further look into how the video files will be stored and displayed to our users. I believe I got most of my questions answered but I would like to have some sort of example functioning by the end of next week.

 

Most of my time this week was spent on the design report. I underestimated how long the design report would take us so there was little progress in terms of the web application. However, the design report helped solidify some of the ideas we had for our project. I have a deeper understanding of our overall project and have a clearer understanding of what we still need to do. The web application is also more fleshed out from working on the design report and it is more clear to me what parts I need to work on in order to complete the web application. Additionally, on Wednesday we met with the pianists again. This time it was a different group of pianists who were participating so we polled them on the UI and any web app features they might want. They seemed to really want to have more control over the different features and wanted to toggle certain features off and on. They were also interested in potentially having different types of feedback other than audio but we have not decided if we will pursue that route. The discussion with the pianists was helpful because it further clarified what they are expecting from the web application.

My progress is slightly behind schedule. One part of this is that we used the micro SD card that came with the RPi for the FPGA. Thus, I could not work on setting up the RPi to host the web application. We should be receiving another micro SD soon so I should be able to make some progress on that front. I should be able to catch up quickly as I have spent time while working on the design report to understand what I have to do. Setting up the RPi should be a fairly quick process. I would still need to work on the web application itself but that should not take a long time either. Overall, I think I am slightly behind but that it is not a major issue and that I should be able to quickly catch up.

In the next week I hope to set up the RPi for the web application hosting and to start working on the web application itself. This will involve using the Django web framework to begin writing out the different parts that we will need for our web application.

This week I spent a decent amount of time practicing for the presentation I had on Wednesday. I spent time going over it both by myself and with Shaye and Jessie so that they could give me feedback on the presentation.

I also spent some time helping Jessie with the Zynq UltraScale+ tutorial: https://xilinx.github.io/Vitis-AI/3.0/html/docs/quickstart/mpsoc.html. I mainly did the first half of the tutorial of setting up the necessary software that is needed and making sure the target was up and running correctly. There were a couple issues that we had while going through the tutorial. There were a couple of minor issues that got in the way such as incorrectly flashing the SD card and not having the correct micro USB to USB A cord (we only had a charging cable so had to run to the store to get one). Another issue was that the Kria KV260 needs internet connection in order to download certain packages but it can only access the internet through an ethernet port. I had to figure out how to provide internet to the Kria KV260 using the ethernet port on my laptop as we had no easy access to a router. After we solved these issues running the example model in the first half tutorial was relatively easy to do. Once the target was running and setup, Jessie took over and completed the second half of the tutorial.

Lastly, for the web apps I did not do as much work as I would have wanted on it this week. I was caught up with different things in capstone and in my different courses. However, I was still able to take a look at how different web app examples to get a better understanding of how Django works and is typically used.

I think I have not progressed on the web apps as much as I would have wanted. I didn’t expect this week to be as busy as it has been so that was an oversight on my part. I am not too worried thought because we will have more work sessions coming in the following weeks so I believe I should be able to catch up the. Additionally, there is no major urgency to finish the web app (before meeting Professor Dueck and her students) so I think it is acceptable to be slightly behind. Also, I think it was more important to help with the FPGA as that is a priority for us as we want to make sure we are able to get the CV models we want working on it ASAP.

Next week, I hope to catch up on the work that I was supposed to do this week for the web app. Additionally, I believe we should be obtaining another SD card for the RPi so I should be able to set that up and perhaps set up a test server on the RPi to understand how it works.

For this week, I have finished the Django tutorial. I should have a very simple and basic understanding of how the Django framework functions. I have a more definitive idea of how the web app should be structured so I will do some research into how I can implement those features. I will probably also spend some time looking at existing frameworks if they already exist. Additionally, since I will be running the web server on the RPi I will look into how to do that effectively. I do not think the RPi section will take very long though. 

This past Wednesday we met with Professor Dueck and some of her students. I asked them about the features we currently had planned and any features that they might want. The consensus seemed to be that they wanted either limited live feedback or to be able to control the amount of live feedback they receive. For example, they were against the idea of having a live video as they were worried it would be too distracting. We are now considering having a live video on a display only for calibration purposes. Additionally, the audio buzzer used as live feedback was also questioned. They seemed to think it would also be distracting depending on the piece they are playing. Thus, they wanted to be able to toggle when they receive the audio feedback or not. For the web app, this would mean I would need to add some feature that is able to toggle the audio feedback as well. This should not be hard to do but it was good to know. Lastly, they supported the idea of storing their previous recordings on the web app so that they can go back and see their progress. Thus, we are considering buying a dedicated hard drive to store the videos for our users but we will stick to using the RPi for now.

Thirdly, I briefly helped Jessie with the quick guide at this link: https://xilinx.github.io/Vitis-AI/3.0/html/docs/quickstart/mpsoc.html.   However, we were not able to get far because we realized that the power supply we found for the FPGA was not compatible.

I would say my progress is still on track overall. For this upcoming week I am worried about slightly falling behind on the web app so that will be my main focus.

Next week, I hope to have examined a couple Django example frameworks to see how it is typically used. I would also want to research a couple ideas of how to implement the features that are required for our project. This includes having a start/stop record button, toggling the audio feedback, storing and serving the video files from the FPGA. I would like to get a decent idea of how all this could be implemented.

This week I started looking into web app development. My part for this project will be to work on the web app but I have no experience working with web apps. I dedicated some time researching the different parts and frameworks that go into creating a web app. There seemed to be a many ways to go about creating a web application and many tools to help. I settled on learning Django at first because it is popular and seems to provide a lot of features for both the backend and the frontend. It also seemed that if I needed to create a better/prettier frontend I can add that onto Django. Thus, I thought Django would be a good place to start picking up web apps. I am currently completing the tutorial on how to create your first web app and will be making progress throughout the rest of the week.

My progress is currently on track with my schedule and I have no concerns with falling behind.

My current plan for deliverables is to finish the Django tutorial to catch myself up to speed on how implementing the backend of a web app works. I will then continue to look into the documentation that I feel is relevant for our project. Currently, our main feature that we are considering adding into our web app is to create a page that will play the video feedback. I will try to look into what would be necessary for this to work.

On this upcoming Wednesday, we will meet with Professor Dueck’s students. I will attempt to get some preliminary feedback on what type of features they might expect from a web app. This will hopefully provide more guidance on what type of documentation from Django I would need to read so that I can begin thinking about implementing the features early.