Category: Zixuan’s Status Reports

This week I worked on testing the threshold for more exercises and the pose comparison algorithm. Since some core exercises have relatively low detection accuracy, I removed them from the workout library. I recorded some demo videos for the final demo. We also discussed the structure of the final video.

We will continue to work on the video for the demo and the final report next week.

This week I added more exercises into the metrics library and did some tests with the pose comparison algorithm. I also worked on the integration of the comparison algorithm and UI. Now we can run pose comparison with the UI. We have the gif of the exercise on the left and user’s video tracked by OpenPose on the right, and we get a result back which reflects the level of completion for each workout based on the angles.

Next week I will continue to work on the integration of the project, add more exercises into the metrics library, and try to improve the overall comparison accuracy.

This week I fixed some bugs of the pose comparison algorithm. One of the bugs was that since I was looking for 10 frames around each timestamp, an error would occur if 2 timestamps we were checking are less than 10 frames apart. I fixed it by setting the range of frames we use for comparison based on the timestamps of the exercise. I received the camera and did some test with it. I also collected the data for more exercises and added them into the comparison metrics library. I wrote another script to make the comparison algorithm work for a sequence of multiple workouts.

We are on schedule. Next week I will continue to add exercises into the metrics library and do more tests with the pose comparison algorithm.

This week I finished the code for pose comparison. We can enter the exercise name and number of loops we want to do, and the program will return the angle differences for each loop, and the sum of angle differences after the exercise is done. Later in the week I also modified and finished the code for frame selection, which looks for the frame with best fit pose for each frame we are comparing. Thus the users are allowed to do the exercise at slightly different speeds. For example, if we are comparing frame 10 of an exercise, we will look at frame 5~15 of the user video, and find the frame with the most similar pose to the standard one. I have tested it several times by doing some of the exercises myself, and the output does reflect my performance.

We were able to run OpenPose on TX2 successfully, however, there were some issues related to memory allocation when running my code on TX2. The error would occur about half the time we run it, and we want to fix that soon.

We are on schedule. Next week I will do more tests with the pose comparison code and fix some bugs.

This week I continued working on the comparison algorithm and building the comparison metrics library. Since the library contains the key points’ coordinates of the standard poses, the workout duration, and the timestamps for the frames we will compare, I need to first select the frames, collect the data, and then put things together for each exercise. I also worked on the script that looks for the frame in the user video that has best match to the standard pose; in other words, the frame with the pose most similar to the standard one.

I think I am a little behind the schedule, but I will finish up the pose comparison part tomorrow and have my code ready for integration on Monday, thus I should be able to catch up. Our group will meet on Monday to put everything we have together on the board. I will try to modify/optimize the comparison algorithm after we have it tested on the board.

One piece of feedback we received from the design report was to resize the images before passing them into OpenPose to reduce latency. We initially tested it with images of size 432×368. The overall accuracy for most poses is about 90%, and the average runtime is about 1.1 seconds per image on a laptop (CPU only). After testing with several different sizes, I found that the size of 320×272 might be a good choice for now since the detection accuracy is above 85% and the average runtime is about 0.7 second per image. The accuracy is significantly lower if I keep downscaling the images.

I modified the pose comparison script to handle the situations that some key points are not detected. I wrote the script that captures frames at specific interval from the user’s camera input. The interval will depend on the specific workout.

I have also been working on creating a library of the angle values of the standard workout poses. For each workout, I took out 2 to 4 frames and extracted the angles from these frames, which will be used to compare with the user’s angles to determine the accuracy.

I think I am a little behind the schedule, since there are more details in the implementation than I previously thought. Next week, I will finish creating the library, test it with me doing some of the exercises, and try to catch up.

 

 

This week I gave the design review presentation and worked on pose comparison. By getting the coordinates of the key points detected by OpenPose, I am able to compute the angles (R1, L1, R2… L4) between limbs as indicated on the image below.

We can compare the poses by computing the differences between the standard angles and the user’s angles. I tested it with some images and this comparison method worked well overall for front-view poses. For side-view poses, since some of the limbs are hidden behind the body, I skipped the undetected key points and only compared the valid angles. I will do more tests to see if some specific poses need to be handled separately.

I have been trying to figure out details about pose alignment. I also did some research on dynamic time warping, especially its implementation and use cases.

I think I am on schedule. Next week I will continue to work on the design report. I will also work on pose alignment, try to use/implement DTW to accommodate workouts at different speeds and test it with videos.

This week I successfully installed OpenPose on my laptop and tested it with some images and videos. It took a very long time for OpenPose to run and the frame-rate was only 0.1 fps, so we decided to switch from OpenPose to TensorFlow OpenPose. TensorFlow OpenPose is a library based on the original OpenPose, which lacks some features (such as hand detection) but runs much faster. It is also compatible with TX2, which is the board we decided to use.

After installing tf-openpose, since there are 4 different models included in the library, I ran a set of 50 images with each model to test the runtime and accuracy. It appears that the CMU model is obviously the most accurate one; since we want high accuracy for our project, we will use the CMU model though it takes much longer than the other models.

I also tested tf-openpose with images from different workouts videos to get a better sense of the performance. These results give me a general idea of which poses are harder to detect, and I will try to improve the accuracy by preprocessing the images (rotate, resize, etc.).

I think we are on schedule. Next week I will present our design presentation and work on pose alignment and comparison.

This week I read some papers and tutorials about OpenPose and looked into some projects using it. I also started to get familiar with OpenPose. It took me a very long time to install all the prerequisites of OpenPose and fix the errors. When I tried to configure and build it, I encountered some issues because my mac os is outdated and some brew packages no longer support high sierra.

I think the progress is on schedule and I will resolve these issues next week (hopefully tomorrow). I will also run OpenPose with sample images and compare some poses.

This week I researched on OpenPose and the hardware parts we were planning to use. I read some papers related to our project to have a better understanding of the technical challenges we might encounter as well as possible ways to solve them. I also worked on the proposal presentation slides with my teammates.

Our progress is on schedule. Next week I will work with my teammates to incorporate the feedback we got from the proposal presentation into our project and finalize our design plans. I will also familiarize myself with OpenPose and try to test it with some images.