Team Status 04/27

Risks and Contingency

Our biggest risk at this point is unfavorable demo conditions, such as insufficient lighting. We will be addressing this with additional light sources. In addition, we need to ensure that we can properly articulate the engineering behind our project, so we are developing an engineering routine to showcase this during the final demo.

Schedule Changes and Updates

This week we will be conducting some user testing results but will have some initial results for our presentation this week. Otherwise, we are on track.

Sandra’s Weekly Status

Accomplished Work

This week I wrapped up my runtime testing for Open Pose, Pose Comparison + Verbal Instruction Generator, and the Overall Application. My results are outlined below and will be included in our report. In addition, I started to work on the slides for the final presentation this week and updated a number of diagrams. I began some work outlining what we need to complete for the final report.

Upcoming Deliverables

Next week, I will be finishing up the presentation, poster, and report.

Chelsea’s Weekly Status

Accomplished Tasks

This week I removed the level buttons since we don’t need it anymore. I worked on adding the demo page for the engineering demo that consists of a frame showing what we print out in terminal, so that we could demonstrate the backend of the project. I also changed the routine page to include a label that would show users the instructions.

Deliverables I Hope to Accomplish Next Week

I hope to finish some user testing (we are currently thinking 3 tests for each of us) and start working on the poster and report.

Tian’s Weekly Status

Accomplished Tasks

1. Implemented the overall scoring for user poses. We define for each pose several important angles. The score is calculated as the number of correct angles over the total number of important angles.
2. Did preliminary user testing with two participants. I let 2 participants observe me doing the entire yoga routine using our application and collected feedback from these participants. For each instruction given by our application, the following few questions were asked:
   1. Does this instruction make sense to you?
   2. Do you think this instruction is expressive enough for the current situation?
   3. Do you think this instruction has appropriate length and pace?
   4. Do you think this instruction is helpful in actual practice?
3. Completed the draft of final presentation slides with Sandra and Chelsea.

Deliverables I Hope to Complete Next Week

1. Refine our application based on one more round of user testing
2. Design the poster with Sandra and Chelsea

Team Status

Risks and Contingency

At this time most of the functionalities of our application have been completed. The major risk for now is the unstable environment for webcam demo. We encounter the following two problems while testing our application:

1. Unstable Lighting condition. We tested our application in the ECE coves, and the major light sources is natural light. During cloudy days, the webcam can only capture dim and blurry images, and these images don’t work well with OpenPose. OpenPose always reports missing important keypoints when performing pose estimation on these images. Since the accuracy of OpenPose relies heavily on the lighting condition of the input images, we need to find a stable and strong light source for our final demo.
2. Webcam cannot focus well on objects or human locating further than 1.5m from itself. The newly bought webcam cannot focus and thus cannot capture clear images for humans at a certain distance from the webcam. This affects the performance of OpenPose. We suspect that this is also caused by the weak lighting we had when testing the webcam. We will test this webcam with stronger light source and see its performance.

Schedule Changes and Updates

We are a bit ahead of our schedule and have most functionalities working correctly. There are no changes to our current schedule.

 

Tian’s Status

Accomplished Tasks

1. Finished static image testing on all poses. I collected incorrect and correct samples for each newly added yoga pose, and tested the instruction generator based on these samples.
2. Implemented Threshold Manager, a struct that dynamically adjusts the thresholds for each pose as user attempts each pose. For example, if the user keeps getting the same instruction “Jump Your Feet Apart” for Warrior 1, this implies that they might not be flexible enough to open their hips as in the standard pose. Then the Threshold Manager will loosen the threshold for that angle between the hips so the user can pass the pose after making some effort, instead of getting stuck at this pose.
3. Integrated and tested the entire application with Sandra and Chelsea. We worked together to integrate all the parts: controller, optical flow, UI, pose comparison and instruction generation. We also performed webcam testing on all the poses and adjusted thresholds accordingly.
4. Implemented functions that flip a pose horizontally and flip an instruction horizontally (i.e. exchanging all left and right counterpart in the instruction). In this way, for the poses with different facing directions such as Left-Facing Low Lunge and Right-Facing Low Lunge, we only need to implement one of them.

Deliverables I hope to complete next week

1. Implement and visualize overall scoring for the poses so that the user can use the real-time scoring as a reference
2. Improve the instruction generator by adding more detections. For example, the generator needs to detect if the user is touching the floor with correct body parts.
3. Start preparing for the final presentation and the poster

Chelsea’s Weekly Status

Accomplished Tasks

Since we added a lot more poses, I added more data structures and methods to the routine class for easy communication among the UI, the routines, and the verbal instruction generation classes.

I also made major modifications to the summary page because our routine sizes grew a lot from our first demo and poses could no longer fit in the original layout.

During our meetings to fully integrate everything, I got to fix a lot of minor UI bugs as well as specifying styles for all the components. I also added the visual feedback that overlays on top of the video feed, which consists of a single fixed size circle that indicates to the user where there’s a mismatch angle between the user and the standard pose. However, it’s not working so well and we suspect that it’s inaccuracy from the Open Pose side. We are conducting further tests with the visual feedback this weekend before the demo. As a backup or a plus, I will be adding a progress bar that shows the user how close he/she is to completing a pose based on the number of angles that are within the specified threshold.

Tasks for next week

Next week I hope to fix any remaining UI bugs and introduce features to improve user experience. Then I hope to start making the presentation with my teammates as well as drafting the report.

Sandra’s Weekly Status

Accomplished Tasks

In an effort to help reduce the number of unnecessary instructions, I implemented logic to help reduce repeated instructions. In addition,  the user now has to hold the pose for 5 seconds. I also implemented logging, which is keeping track of all of our metrics, for runtime. We have a log file that generates each time.

During our meeting this week, I worked with Chelsea and Tian to do lots of testing of the application. We tested some of the new poses and how they work with the new logic that we each implemented.

Tasks for next week

I need to make sure that the program allows the user to exit a routine and begin a new one. In addition, I will be continuing to get logging info and translate this into tables and other graphics. I will also begin working on the final report, presentation, and poster.

 

 

 

Team Status 04/13

Risks and Contingency

Right now our biggest risk is small scale bugs in our application that can add up. For instance, sometimes, the application will tell a user to recalibrate multiple times during the same sequence because it is unable to track points on the user’s body. In addition, we sometimes encounter an issue where the same instruction is given to the user but obeying it does not render a result. We need to account for these tiny bugs, just be extensively testing the application. In addition, we will be incorporating thresholds to help improve the yoga practice for beginner users. We will be testing and generating metrics for the following components of our application. In addition, we are going to lower risk for demo surprises by using a camera and tripod system for capturing images via the webcam

1. Optical Flow Runtime
2. Pose Correction and Verbal Instruction Generation Runtime
3. Overall Routine Runtime
4. Verbal Instruction Usefulness

Schedule Changes and Updates

To account for the final in-class demo on the week of April 22nd, we are going to speed up some of our application testing. In the upcoming week, we will be conducting tests in the areas outlined above. Furthermore, we finish up adding poses and developing thresholds that are dynamic for the user. We are hoping to get the bulk of testing done in the upcoming week, so we can spend the last few weeks, doing smaller additions on the applications and report, presentation, and demo.

Team Member Updates

Sandra Sajeev

Accomplished Tasks

This week I was able to do runtime testing on both the optical flow runtime and pose correction runtime. The chart below showcases the average Pose Correction runtime in an area with low lighting. I will need to repeat these tests in various lighting conditions to determine the true approximation of the runtime. This is slightly higher than our initial estimate of pose correction time, but it is still under our 5-second maximum time.

In addition, I did some testing with the updated optical flow workflow. The new optical flow workflow looks at a bunch of distances from frame to frame and then initiates pose correction if the difference between the min and max distances is greater than some percentage. Below is a summary of results for thresholds ranging from 0.25 to 1.0. From these results as well as the overall experience, it seems like 0.5 and 0.75 are the best performers. 0.75 has the most stable results as the time remained similar. The time period measures the time between pose corrections on the y-axis.

Deliverables

This upcoming week I will be continuing the pose correction runtime testing and beginning overall routine testing. I will be collecting data and generating graphics as well as comparing test results with our test results.

 

Tian Zhao

Accomplished Tasks

This week I filtered out new poses (Cobra and Hero) that didn’t work well with OpenPose after testing with many static pose images.

I also finished refactoring and writing code for the entire instruction generation logic for all 14 poses. Previously, we were using a lot of if statements to condition on different poses and decide on which instruction to give. However, this approach only works well with a small number of poses. Now after refactoring, each pose has a list of functions such as get_arms_instructions, get_back_instructions, and each of the function will output instructions on different aspects of the pose. Given a pose, our instruction generator will simply concatenate the outputs of the functions. This makes it easier to add special instructions to a pose and to reuse similar functions for different poses.

Deliverables I hope to complete next week

1. Finish static image and webcam testing for all poses with Sandra and Chelsea
2. Implement threshold manager and tune the thresholds for all poses with Sandra and Chelsea

 

Chelsea’s Status Report

Accomplished Tasks

This week I added a feature to allow users to select their yoga levels, level 1-3, when they click on a routine button. The level determines how leniently the program restricts the angle differences between the user’s pose and the standard pose.

Additionally, when the user’s mouse hovers over a routine, an overview carousel card is added to display the poses in that routine with pictures and verbal descriptions.

Lastly, since we decided on the poses for the three routines: beginner, intermediate, and advanced, I hardcoded these values into our program.

Deliverables I Hope to Complete next week

Next week I hope to detect any remaining bugs on the UI side by testing our app as a whole and fix them. I will also start working on the final presentation with my teammates.

Team Status

This week we improved our integration and prepared for the MVP demo. Our have completed the following functionalities by far:

1. Basic UI features including pre-determined yoga routines, interactive GUI and speech synthesis system
2. Calibrating, tracking and capturing user movements using Optical Flow
3. Calculating differences between user pose and standard metric and generating meaningful template text instructions

Risks and Contingency Plans

Currently our program applies static thresholds for all the users and all the poses, which means if the user cannot meet the preset standard, it will repeatedly output the same verbal instruction. This affects the utility of our application and can be discouraging for yoga beginners.

We will deal with the risk using the following approaches:

1. Implement a Threshold Tracker that will loosen the thresholds if the user has received the same instruction multiple times
2. Increase the interval between two same instructions
3. Perform user testing on our application to make sure that users are comfortable with the pace and the contents of the instructions

Changes and Schedule Updates

We decided to replace DownDog and Plank with other yoga poses since they don’t work well with OpenPose and are hard to demo. Our current yoga pose set includes Tree Pose, Mountain Pose, Chair Pose, Staff Pose, Half Forward Bend and 9 other poses.

Our schedule remains the same.

Tian’s Status

Accomplished Tasks

This week I worked on integrating more poses into our existing algorithm. I collected standard yoga pose images for 12 more poses: Bridge Pose, Cat Pose, Cow Pose, Cobra Pose, Chair Pose, Hero Pose, Staff Pose, Seat Forward Bend, Half Forward Bend, Low Lunge, Cobbler’s Pose and Easy Pose. I experimented running these poses with OpenPose and generated standard metrics for these poses.

I also summarized the commonalities shared among these yoga poses based on the outcome of the experiments:

Since now we have many more poses, I started refactoring the instruction generator for more generic use and implementing instruction generation logic for these new poses.

Deliverables I hope to complete next week

1. Finish basic integration of all yoga poses mentioned above
2. Start webcam testing for all poses

Chelsea’s Status

Accomplished Tasks

This week I added a tool bar to allow users to exit a routine and go back to the landing page, switch to the previous or next pose, and pause or resume the current pose. I also added a summary page summarizing the user’s workout.

The routine page and the summary page now looks like the following:

 

Deliverables I hope to accomplish next week

Next week I hope to fully define the routines with my teammates since we have added a lot more poses as well as will be able to reverse poses. Secondly, I hope to be able to display the standard pose skeleton over the user’s body in the video feed to provide visual feedback.

Sandra’s Status

Accomplished Tasks

This week I refactored optical flow to dynamically change thresholds to detect movement. We have kept noticing that optical flow distances varied depending on the environment we are demoing.  For instance at nighttime in my aparmtnet a large movement threshold of 40 was sufficient, however during sessions in HH, the distances never exeeded 10 pixel distance. This might be affected by the overall light in the room. So I tracked the last 50 frames to detect an average threshold for movement or stabilization. This will create a more tailored yoga experience.

 

Next Week Deliverables

Next week, I will be stress testing the application and looking into creating the User Statistics class. In addition, I need to implement logic to close threads and the application