The past two weeks I have been working on creating the custom timed workouts as well as wrapping up the calculations for calories burned + average heart rate.
For calories burned I did a bit of research in terms of the MET (Metabolic equivalents) for various workouts. From my research I found that push ups are have a MET value of 7.55, leg raises have an MET 2.23 and Lunges have a MET value of 2.80. I used the following formula to calculate the aaverage calories burned as well as heart rate in the code:
I was able to complete this section and it will accurately reflect how many calories are burned. I also take into account during rest sections that the MET value will decrement at rate of 20% per minute.
In terms of the timed workout I was not able to fully finish them as I had anticipated as I had a very rough schedule this week (midterms etc.) and not too much time to finish. I have all the necessary media, models and templates at this point to finish but still have a few more buttons and pages to add. I will be finishing this by Monday (10/26) at the latest and will be almost ready for integration.
I have also begun researching the way that the data will be stored more in depth so that it can be tied into the application. SQLite will be adequate and be a safe way to store the data in a relation manner. I have the function headers for storing the workout data setup but will be adding that in, this upcoming week.
As I had a tough week I am about half week behind my schedule but will definitely be able to catch up, as my other classes had midterms and I will now be able to put my full effort/time into the application and UI. I anticipate some time will be required to integrate with the UART protocol as well as the posture analysis so I will also begin prepping for that.
The past week and the upcoming week are allocated for working on the HLS implementation. Unfortunately, I was roadblocked at the start of the week since I was having trouble with HLS and wasn’t sure how to proceed. I then met Zhipeng (who the Professor introduced me to) and he was able to address my questions and pointed me to use the MicroBlaze soft-core CPU that would be responsible for controlling the interactions that take place on the FPGA. I also updated the block diagram that exists on the FPGA to be the following.
Note: It is not complete as it still requires the IP core that is responsible for the image processing. This core will connect to the appropriate AXI ports that have been left unconnected.
I then started learning how to use the MicroBlaze core, which required adding a new piece of technology (Vitis) as the current version of Vivado does not have an SDK to program the MicroBlaze core. I was able to program the core and stream appropriate information via UART. I am looking into the other components in the block diagram and how to control them.
I am on track with the schedule. This week, I hope to be able to learn how to control the various components specified in the block diagram and also have the image processing IP core done.
We were finally able to take some demo pictures from the webcam since we ordered the item slightly later than the rest. I had been working with iPhone images. A noticeable problem when I first got the image is not the lighting is a completely different shade of color from the iPhone image. We hope that the change of saturation is due to the different webcams. However, it can also be due to the lighting of the room. I ran the newly captured images on my previous algorithm and they were not pinpointing the joints. I will do some further tests next week by capturing the images at night with the webcam and see if my new acquired bounds or old ones will be able to detect the trackers. Also, the webcam image seems to come in with a different dimension than a normal picture. Therefore, there are black lines to the side of it which is unnecessary for the joint tracking. Thus, I have made edits to my algorithm to crop out the black lines.
To aid me in finding the bounds, I have to manually find the center pixel of the trackers and capture the HSV bounds by finding the minimum and maximum within the area. I have to do this for every joint on a reference image. Then, I would run the same bounds on a different image to ensure that the joint locations that I returned are in a similar area. I have to redo my fine-tuning because of the different saturation of the image. I started with the pushups this week. I had to modify the morphological transform portion because an erosion followed by a dilation would remove a lot of the important pixels I track. Thus, I changed it to 2 dilations then followed by 2 erosions to better track the pixels. The picture below shows the output. The peach tracker on the elbow would have to be changed because it is too similar to the background; thus, it its not trackable. I hardcoded that position in for the posture analysis portion.
Purple Trackers represent the Joint Location. The image is the one I tested the bounds on.Purple Trackers represent the joint locations I find. This is the reference image.
For the posture analysis portion, I had to make edits to the code because due to latency issues we have decided that we will be only tracking the second or more important position of the workout. For a pushup, it is the downwards motion. Since I finally have the joint positions from the image processing portion, I could finally do some threshold fine-tuning. I adjusted the values of the slopes and angle comparisons to fit our model. The current model I got gives me the correct feedback when I track feed in the up position for the pushup analysis. It would output “Butt is too high” and “Go Lower” because it detects the hip joint and elbow joint not in the same slope and angle according to the pushup model. To make this better, we would have to capture more images of faulty pushups in order for me to fine-tune it even better.
I am on schedule in terms of the image processing portion, but is slightly behind on the posture analysis portion. Since the posture analysis portion is easier to implement and fine tune fully with the application and system set up, I will work on helping my teammates with their portions. I will get to the Lunges and Leg Raises posture analysis fine tuning when I have spare time from helping Venkata with the RTL portion. We seem to have found a library in HLS to help us do the image processing portion. However, the bounds that I find will still be useful for the HLS code. Since the joint tracking algorithm and posture analysis is very sensitive to real life noise, I envision the integration portion would constantly be updating, so I would keep on updating my algorithm on a weekly basis.
This week we took sample images with the webcam that recently arrived. In order to appropriately fine-tune the parameters for our posture analysis algorithm, we took multiple images at different positions of the various exercises that we plan on supporting. A major risk that we identified was that we would not be able to fully identify all of the colors. This risk is fully elaborated in Albert’s status report for this week. The contingency plans involve finding more distinct colors that would be easier to track and experimenting with different lighting and background conditions to find the optimal background for our project.
No major changes have been to the overall design and schedule.
In terms of team work this week mostly comprised of working on the design review and design report. We took the feedback from the presentation and updated our report accordingly. We also made a few updates to our schedule to make it more reasonable and up to date.
We also worked on designing and creating the tracker suit. We realized that there were a few colors that weren’t bright enough through the webcam so we had to change them and modify the location the trackers were placed.
This week I worked with Venkata/Albert to create the tracker suit that we will be using for our project. More details on the work done for this can be seen in the team status update for this week.
In regards to my own personal work I spent a lot of this week working on the design review report/presentation which ended up taking a major portion of my time. I was also able to make progress in creating the timed workouts. I now have each set timed and it will move onto the next set once the first set is finished. Pictures are being taken periodically as each rep finishes. The user input menu is completed now and the weight and height can be adjusted. In terms of the schedule I have updated it so that I have another week to work on the timed workouts as it ended up being a lot more work than I had anticipated. The timing for the schedule is still fine as I already had a slack week accounted for.
This week I started working on fine tuning the HSV bounds to do the color tracking. This was supposed to be done earlier, but we weren’t able to make the suit as there was some miscommunication in the ordering process. I had been fine tuning the alogirthm on sample images. However, fine tuning to the dark suit that we currently have took longer than expected because we would have to get it even more precise to the noise from a real life downscaled image. Instead of doing a relatively broad range, I have to manually pinpoint the joints first. Then, I would find the HSV values around those points and create an extremely precise bound to have better accuracy. I also finished implementing the posture analysis for pushups and lunges in python to provide feedback to the user. I plan to fine tune this portion after the image processing is set up on the FPGA. Also, since this week I was presenting for the design review presentation, I spent a lot of time fine tuning the presentation after everyone was done and practiced for more than 3 hours because I am not a good presenter.
In terms of schedule, I am more or less on track because I have started things that are due later but haven’t completed things the are already due. I have been doing a lot of verification of individual parts along the way, so integrating may take less time than we have planned. For next week, I will work with Venkata on converting the joint tracking algorithm to RTL using HLS.
This week was predominantly focused on learning HLS. I started reading this tutorial by Xilinx and have a rough idea of the workflow using HLS. I also talked to a couple of friends who are currently taking 18-643: Reconfigurable Logic and they pointed me to further resources for learning how to use HLS. After referring to a combination of these resources and some of the sample HLS projects that Vivado provides, I have an idea of how to go about implementing the design.
I am currently on track with the schedule. According to the schedule, this past week was for learning HLS and the upcoming week is for implementing the design. I have begun working on next week’s task for implementing the design by working with Albert to convert the existing Python code for image processing into C that can be passed into HLS.
This week I was able to make a bit of progress in terms of the application and user interface. I attempted to use the camera module of the pygame library but soon ran into issues, as it is only certified and optimized to work on linux. However we decided we wanted to develop our application on windows so I had to instead transition to using the open cv library. There was a bit of a learning curve so I spent some time understanding the intricacies the library.
After working with openCV I was able to create a basic application in pygame that now has the live video embedded. It also takes pictures periodically and saves them to a local folder which will be consumed by the FPGA. I’ve also been working on finding and converting model exercises to GIF’s that are displayed within the application. I’ve also begun creating a model page to get the user’s biodata that will be used to calculate things such as calories and heart rate. In terms of the schedule I am currently about half a week to a week behind as I haven’t fully finished creating the GIF’s for the model and am still working on calculating the user’s biodata page and creating the timed workouts.
This week we discussed how we would be providing feedback to the users for our workouts. We found an optimal time interval to satisfy the requirements of the time it takes for the FPGA to perform the calculations with the live workout. We finally got most of our equipment at the end of the week, and we are still waiting on the camera. This has been a slight roadblock to our progress because a lot of our work depended on these materials. We spent a lot of time on the Design Presentation, because we had to create new graphs, flowcharts, and an updated schedule to reflect our work and progress.
