Posted on

Zixuan’s Status Report for 3/27/2021

One piece of feedback we received from the design report was to resize the images before passing them into OpenPose to reduce latency. We initially tested it with images of size 432×368. The overall accuracy for most poses is about 90%, and the average runtime is about 1.1 seconds per image on a laptop (CPU only). After testing with several different sizes, I found that the size of 320×272 might be a good choice for now since the detection accuracy is above 85% and the average runtime is about 0.7 second per image. The accuracy is significantly lower if I keep downscaling the images.

I modified the pose comparison script to handle the situations that some key points are not detected. I wrote the script that captures frames at specific interval from the user’s camera input. The interval will depend on the specific workout.

I have also been working on creating a library of the angle values of the standard workout poses. For each workout, I took out 2 to 4 frames and extracted the angles from these frames, which will be used to compare with the user’s angles to determine the accuracy.

I think I am a little behind the schedule, since there are more details in the implementation than I previously thought. Next week, I will finish creating the library, test it with me doing some of the exercises, and try to catch up.

 

 

Posted on

Team Status Report for 3/27/2021

One suggestion we received was to add a live video of the user so they can see themselves during their workout. However, we were thinking this feature wouldn’t be necessary because of the group we are creating this product for. People are mostly working out at home right now, and using YouTube workout videos, but they mostly aren’t watching themselves do the exercises. I don’t think many home workouts are done in front of a mirror (at least according to people I’ve asked). We’re also concerned about how much latency this feature would add to our product.

Even if they did do the workouts in front of a mirror, we think that the ability to watch themselves during the workout will distract the users. To look up at the live video, the user will need to use incorrect form. This is something we don’t want to encourage the users to do, because incorrect form can lead to injuries over time because of strain caused to certain areas of the body. Also, during workouts, the users should be focusing on engaging the muscle groups that a particular exercise is targeting.

Maybe as a compromise, we could add an overview of the generated workout so that users can learn how to perform the exercises before the actual workout starts. This part of the product could be controlled by the user; as in, they could choose when to see the next exercise (press the next arrow key) so they can take as much time as they need to learn one exercise before seeing the next. This would also allow users to skip exercises they already know, so they can focus on learning only new exercise forms. In this, we could provide the same looped videos so the users can copy them. The users would be able watch the demo video, find a mirror (or we could provide the live stream video here), and try to mimic the demonstrator’s form correctly.

Posted on

Sarah’s Status Report for 3/27/2021

This week, I added 9 core and arm exercises (from 4 videos) to our workout library. I also updated the workout algorithm (finalized algorithm with Maddie, and started it last week) to include paired exercises. The only paired exercises are the ones that work on one side of the body at a time (ex: single arm side pushup). I think the workout algorithm is finished. I can always go back to modify this algorithm if we change our design later.

I also worked with Maddie on some of the UI (looping exercises using OpenCV). I’m still in the process of learning PyGame, so the progress on the UI is slow. However, according to our gantt chart, I’m still on schedule.

Next week, I hope to have more of the UI done. Maddie already looped the exercises using OpenCV, so I want to get the 2 screens for rest time and exercises done. I’m thinking each of these screens will just be their own functions/classes so they can be easily called multiple times. If possible, I also want to get the start screen created. I’m just going to model it off of YouTube videos as well, so the design is cohesive. I also want to get the key controls working (where the user can press certain keys to trigger certain actions).

Posted on

Zixuan’s Status Report for 3/13/2021

This week I gave the design review presentation and worked on pose comparison. By getting the coordinates of the key points detected by OpenPose, I am able to compute the angles (R1, L1, R2… L4) between limbs as indicated on the image below.

We can compare the poses by computing the differences between the standard angles and the user’s angles. I tested it with some images and this comparison method worked well overall for front-view poses. For side-view poses, since some of the limbs are hidden behind the body, I skipped the undetected key points and only compared the valid angles. I will do more tests to see if some specific poses need to be handled separately.

I have been trying to figure out details about pose alignment. I also did some research on dynamic time warping, especially its implementation and use cases.

I think I am on schedule. Next week I will continue to work on the design report. I will also work on pose alignment, try to use/implement DTW to accommodate workouts at different speeds and test it with videos.

Posted on

Maddie’s Status Report for 3/13/2021

This week (wk 6) I worked on board setup, though I did not get as far along as I had planned.  Since I hadn’t worked with a Jetson board in the past, I didn’t anticipate how much difficulty I would have with setup on a Mac OS.

I managed to get Ubuntu running on my Mac with VirtualBox (I had a lot of unforeseen issues with VirtualBox install that led to setbacks, but I was able to resolve them).  I also set up my Nvidia developer account to help me with the TX2 developer kit.

Though I fell behind on my personal schedule this week, I have laid out a stricter schedule for myself for this upcoming week to make sure I get caught back up to speed.  I plan to use Sunday through Tuesday specifically for finishing setup with OpenPose on the TX2.  (I will also be working on the design report with my team due Wednesday.)  Then I can dedicate the rest of the week to working with Sarah on the workout library and construction algorithm.

If I follow this plan for week 7, I will be back on schedule by the end of the week.

Posted on

Sarah’s Status Report for 3/13/2021

This week, we were able to decide that we will use AWS to store our exercise library. I still need to request those credits through the form. I will do that after this status report. I also realized that some of the gifs I made last week got corrupted somehow (didn’t loop correctly), so I remade them. I found more workout videos to use for our library, I just need to create the gifs for them. So, I didn’t actually end up parsing new videos this week.

My progress is behind schedule because I wasn’t able to devote as much time as I wanted to this class. I ran into issues with projects from other classes (Distributed, mainly), so that took a lot of time away from this class.

To catch up, I hope to actually start on the scoring algorithm, and meet with Maddie to work on the exercise library/construction algorithm. I have created a calendar on my phone that I hope will help me track my assignments better, and break my tasks down into smaller segments. I think this will help me devote a more equal amount of time to this project and my other classes.

Posted on

Team Status Report for 3/13/2021

This week we received feedback from our design review presentation. One of the most significant risk was comparing the poses at different speeds. Our plan was to select specific frames from the user input based on time and compare them with the poses from the workout library, which would not work if someone is doing the exercises much faster/slower. We are looking in to dynamic time warping as suggested by Prof. Savvides, and trying to apply that in our project to correct what we compare by looking for the best alignments between time series.

We also need to figure out more quantitive and detailed ways to measure accuracy of pose alignment and comparison. We planned to test it with different body types and poses, and check whether it can align the points and distinguish the poses correctly.

We started working on our design review report.

Posted on

Maddie’s Status Report for 3/6/2021

This week I started building our ‘legs’ video library.  I broke down clips from a few different YouTube videos into single repetitions of the exercises.  Originally we had planned to use the video URLs to fetch the exercise clips, but now we think it may be easier to save mp4 versions of the clips, so I am planning to convert some videos to mp4 for trial Sunday and Monday.

I also started initial setup of the Jetson TX2 that we picked up.  I am planning to download OpenPose onto the board over the next few days, per our schedule.  I also spent time working on the design review slides.

I am pretty much on schedule, though I had to add a few extra days in the schedule for board setup and library construction, since it is taking a little longer than I anticipated.

For next week, I want to install and begin testing OpenPose on the TX2.  Additionally, I am planning on working with my team to construct a small video library.  I would also like to get some of the UI created.

Posted on

Sarah’s Status Report for 3/6/2021

This week, I manually parsed 3 YouTube workout videos into separate exercise clips (1 repetition each). This gave us 31 individual core and arm exercises for our library. I saved both a gif version of each exercise (using a gif making website) and the start and end times of the exercise from the YouTube videos in a spreadsheet shared with the rest of my group.

I also worked with my teammates on our Design presentation slides and report. We tried to incorporate the feedback we got from our proposal presentation slides. We particularly wanted to make the slides more visual (“marketing visuals”), so I added gifs and more images to the slides.

I requested our TX2 board from the 18-500 inventory, which Maddie picked it up. I also ordered our camera, which is with me. We haven’t met to swap hardware yet. We may need to change the tasks around in our Gantt chart to reflect these changes in task leadership (Maddie for board set up, me more library construction).

I think we’re basically on schedule, though.

Next week, I hope to figure out how we’re going to store the reference exercise clips (either on board, on laptop, or in AWS?). I want to continue parsing videos, too. It takes longer than I thought to find parts of each exercise that loop well, and find the how long to make the clip that will loop. So, I’m planning on parsing a couple videos per week so it’s manageable. I also hope to start working on our scoring algorithm.

Posted on

Team Status Report for 3/6/2021

We decided to use a Jetson TX2 Xavier board instead of a Jetson NX Xavier or Jetson AGX Xavier. This is because The 18-500 parts inventory had both the AGX and TX2 boards from previous years. We didn’t want to buy another board if there were already 3 usable boards available. We originally wanted to borrow an AGX, but other groups requested the 2 in inventory earlier than we did, so we weren’t able to borrow them. The inventory still had a TX2, so we submitted a request to borrow that instead, and were able to do so.

Borrowing the boards instead of buying a new one allowed us to stay exactly on schedule. If we bought an NX, then we would’ve been at least 2 days behind because the shipping time is 2 days. It also decreased our costs, so we’re only paying for a camera currently.

To adjust to the board we chose, the TX2, we decided to use a different library: Tensorflow OpenPose. This library was able to analyze some sample images (on Zixuan’s laptop) much faster than the original OpenPose was. While she was testing OpenPose, she was able to verify that the images we took from YouTube videos are usable with Tensorflow OpenPose.

This week, we also worked on our design presentation slides and tried to incorporate the feedback we got from the proposal presentation. So, we added more pictures/visuals, and a testing plan (not actually in the slides, but will be talked about). We’re currently still working on our reasoning for our metrics, but this should be done by the time we present.