Posted on

Ankita’s Status Report 2/25

Current Progress:

  • Helped research which monitors we could potentially acquire and that would fit the constraints of our ideal-length mirror (14 in x 48 in)
  • Submitted the material acquisition form after finalizing material details with the rest of the team
  • Created a block diagram of the Software and Hardware Architecture for the Design Presentation
  • Continued working on the Pose/Angle lookup table for the side poses (downward dog and cobra pose) and running the Pose Detection algorithm on the yoga instructor’s ideal poses to determine the ideal angles at key joint configurations
  • Working to implement the Pose Correction functions for determining whether the current pose is a downward dog or cobra pose

In regards to the schedule, we are on track as we are attempting to finish all the Pose Detection/Correction algorithms before Spring break. With this upcoming week being our scheduled slack time, we are planning to work on our backlogged tasks (mainly mirror construction).

Next Steps:

  • Finish the Pose Correction functions for the poses mentioned above
  • Possibly research other metrics that could aid in correcting a pose besides angle- could consult a yoga instructor again for this information?
  • Assemble the mirror hardware
Posted on

Ankita’s Status Report 2/18

Current Progress:

  • Met with a yoga instructor to take reference pics for each of the 6 specified poses from the TOP and SIDE view
    • Also inquired about how a professional yoga instructor judges whether a pose is held correctly and what metrics she uses.
    • Based on her feedback, we are now emphasizing whether the joints are aligned with each other (“straightness”) of the limb
  • From these pics, created a pose/angle lookup table by quantifying the ideal yoga pose images from the instructor into angles formed from 3 of the 33 possible MediaPipe landmarks using the PoseEstimation function we previously wrote.
  •  Wrote the getAngle2D function that calculates the angle from 3 inputted nodes. This was used above to calculate the angles at notable joint combinations and analyze how the instructor’s pose angles varied with our hypothesized pose angles. This could also help us identify a valid error threshold as angles could vary slightly from person to person.
  • Read and watched tutorials on how to construct a wooden frame that will go around the mirror and LCD screen -> Will finalize the wood dimensions once we decide whether we will use one or two monitors.

In regard to the schedule, we are slightly behind on the hardware construction as we haven’t ordered our materials yet. In regards to the Pose Correction, we have made significant progress and will be most likely finished by the end of the week.

Next Week’s Deliverables:

  • Continue working on the Pose Correction algorithm and research other metrics besides angles and node distances that could be used to further improve our algorithm
  • Order the LCD monitor, 2-way mirror, and wood
  • Participate in a yoga class to gain further insight into what other metrics could be helpful to include

Relevant Courses:

  • 16-385: Computer Vision
  • 16-467: Human-Robot Interaction
  • 15-281: Artificial Intelligence: Representation and Problem Solving
  • 16-384: Robot Kinematics and Dynamics
Posted on

Sruti’s Status Report 2/18

Progress Update:

  • Worked on creating pose correction library:
    • Built functions to calculate node distances
    • Built functions to check for node visibility and thresholding
  • Processed ideal yoga pose images that were taken during the yoga instructor meeting.
    • Identified important nodes that should be used for correction
    • Identified what the node values should be
  • Spent a lot of time fleshing out the software and hardware workflow for the overall system and creating graphics for the design presentation
  • Looked into the physics behind how we will identify where the reflection of the user will show up on the mirror.

Even though we are behind on the hardware acquisition, I feel like we are on track for the schedule and should be able to get things done on track.

Relevant Courses Taken:

  •  Intro to Machine Learning
  • Computer Vision
  • Gadgets, Sensors, and Activity Recognition
  • Interactive Design Studio (HCI class for accessibility and usability design)
  • Physics II

Next Week:

  • Next week I hope to have the pose comparison aspect of the software working
  • I also hope to have the hardware bought and start assembly.
Posted on

Youssef’s Status Report 2/18

Progress Update:

  • Met with yoga instructor
    • Collected yoga reference images for fixed set of poses from 2 different angles (side and frontal view) to ensure all critical elements of pose are captured to be used downstream in the pose correction component
    • Interviewed them to gain better understanding of appropriate error signals to look for as guides for corrective suggestions and also know if the metrics we plan to use in PosePal are comprehensive enough/will result in an effective yoga-learning tool
  • Helped with pose correction library implementation
    • Researched more/collected information about the algorithms and metrics we would use to represent poses/detect errors
    • Tested functionality of implemented library to validate results of computed angles on set of examples

Still need to actually order hardware, but given the fact that the hardware we need to buy isn’t particularly niche/hard to acquire and that we are on track with the other tasks/don’t have pressing work that requires hardware within the next couple weeks, we should be able to get the hardware this week and still be on track with our planned schedule.

Relevant Courses Taken:

  •  Intro to Machine
  • Into to Deep Learning
  • Concepts in AI
  • Pattern Recognition Theory
  • Advanced algorithm course (15-210, 15-451, etc.)
  • Physics
  • Experience working with CV/rendering from computer vision swe internships

Next Week:

  • My goals next week are to finish the pose correction component and map errors internally to corrective suggestions
  • We also need to have the hardware ordered by the end of the week
Posted on

Team Status Report For 2/18

Progress Updates:

  • Had a meeting with the yoga instructor we are working with to get ideal poses and learn about typical errors and metrics to correct yoga poses
    • Collected reference pose images for all the 6 poses we want our system to have
    • Currently working on analyzing the poses to determine which nodes and what angles should be used for pose comparison. We are also computing pose detection values and storing them so it does not need to be recomputed each time.
    • Yoga expert helped us understand additional error signals used in in-person instruction that should be present in our pose correction system, most notable of these being the concept of alignment along axes on a person’s pose (for example, arms and shoulder should be collinear)
  • Implemented library for Pose Correction component to compute:
    • Euclidean angles given 3 joints on reference pose
    • Distance between 2 joints (to check if they are in contact)
    • Check if all the required joints are visible (to make sure they are in frame, and that there joints are not one behind the other)
  • Developed our design review presentation:
    • Fleshed out our system’s design
    • Better understood how to implement sub-systems
    • Practices and created slides

 Our Major Risks so far include:

  • Monitor and hardware acquisition: We are hoping to acquire 2 monitors to construct a full-length mirror, but are not sure how to acquire at a reasonable price and integrate them together along with the two way mirror. 
    • Contingency plan is to have one smaller monitor that acts like a mirror and have the user stand further away so their entire image can still be seen. 
  • Mirror UI: As mentioned in our presentation, another huge risk is how to display our Pose Correction results on the mirror itself and whether any further mapping may be required. We have spent some time working through the physics of it so it seems more plausible now but is still a risk until we acquire the hardware.
    • Contingency plan is to simply render the camera feed on monitor

Guiding Engineering/Scientific Principles For Our Design

  • Machine Learning/Computer Vision:
    • Core components of our project like the pose estimation capabilities rely on computer vision ML models. The model we’ve selected to use is a convolutional neural network (CNN) model called BlazePose that is optimized for real-time, on-device inference
  • Computer Graphics:
    • Need to use efficient and lightweight computer graphics libraries/methods in order to render pose corrective suggestions in real-time in a visually understandable way
  • Software engineering:
    • Working with several python modules and writing a custom pose comparison library
  • Geometry:
    • Pose comparison is based on angles and distances, which are rooted in 2D and 2D geometry
  • Physics of light and mirrors:
    • Using the behavior of light and principles of reflection to understand where a person’s reflection will appear on the mirror
  • Firmware and hardware-software interface:
    • Integrating camera input with the computer
    • Working with speakers for audio output
  • Human-Computer Interaction (HCI):
    • Have HCI as an additional guiding principle to make sure that the final integrated system is easily usable and accessible for users.
    • Ensure that the UI and corrective feedback are delivered in a way that is understandable and doesn’t compromise the yoga learning experience (deliver audio cues to avoid user physically interacting with the screen for example)

Timeline Update:

For our software algorithm, we are on track. However, we were supposed to buy the hardware last week. We spoke to the TAs about using monitors that are already present in the lab and are waiting on that information to go ahead with the purchase of materials. We hope to make that purchase this week, which will push back our hardware timeline by a week.

Next Steps:

  • Complete error comparison this week
  • Order hardware and begin assembly of hardware
  • Complete design presentation
Posted on

Youssef’s Status Report 2/11

Proposal Work

  • Completed requirements and software solution sections of proposal presentation slides
  • Prepared for and gave actual presentation for PosePal proposal to the class

Project Work/Progress

  • Synched with and organized meeting with yoga instructor for key project-related tasks
    • Will collect selected example pose models from scheduled meeting with instructor
    • Also prepared questions to ask instructor to identify what error signals our pose correction system should look for to best match how in-person classes are run and give the best user experience possible
  • Conducted research on different possible approaches we could use for the pose estimation component of PosePal
    • Collected benchmark performance data of candidate models for comparison and to understand how to objectively define the requirements that we can and should meet to satisfy the outlined use case for the project
    • Ran example models on video footage and using real-time camera to make sure that we can run and use them in the project itself
    • Example notebook link: https://colab.research.google.com/drive/13N94cPLcETxEstvvLbQO3jxYsmyNdbN4?usp=sharing
  • Code snippet for extracting the model and setting up needed environment to run it:

Next week deliverables include the following:

  • Meeting with yoga instructor on Monday and collect the reference yoga pose models
  • Implement the pose correction library that will include the geometric computations like computing the Euclidean angle given two x,y,z coordinates for use in pose comparison
  • Implement functionality to actually suggest corrections given two points, the reference and user’s, use this information downstream for the correction rendering and suggestions which will be the target task starting in the following week

Schedule Status Check

We have accomplished all the deliverables that we’ve outlined for up until this point on the schedule per the Gantt chart for the first week. Our upcoming tasks for the previous week are all included in the above next week deliverables section, so we expect to still be on track and in line with our original schedule by the end of the week.

Posted on

Ankita’s Status Report 2/11

Key status updates include:

  • Researching different methods of pose correction (OpenCV, BlazePose, and Colab)
  • Researching different ways of integrating the LCD display with the 2-way mirror and the Magic Mirror library
  • Tested out BlazePose on a sample video to see if the software produced the accuracy requirements we aimed for and whether the skeleton could be drawn as the video shifted at different angles

With respect to our current schedule, we are on track with the project deliverables as we plan to order materials this week and work on identifying the error metrics as well as continue implementing the pose correction code.

Next Steps:

  • Work on creating a library and functions to measure the node-to-node distance to help calculate node pose error
  • Finalize Mirror hardware and order soon
  • Meet with the yoga instructor to gain more insight into possible error correction metrics

 

Posted on

Sruti’s Status Report 2/11

Progress Update:

  • Tested out Mediapipe’s pose detection algorithm to see the results it would give.
  • After the team decided to use mediapipe, worked on creating a real time 2D pose detection algorithm which rendered the pose over the person’s video stream real time as shown below.

  • Spent some time understanding all the possible outputs from the media pipe pose detection.
  • Created a 3D rendering of the real world poses based on the video camera input as shown below.

  • Spent time deciding the different values we want to look at when doing pose correction with the rest of the team and created a clear plan and approach

Based on the progress, I feel like we are on track for the schedule and should be able to get things done on track.

Next Week:

  • Next week I hope to have a library of functions that handles pose correction completed.
  • I am participating in the yoga instructor meeting and hope to have ideas poses by the end of that meeting.
  • I hope to have decided on what hardware we need to buy and made that purchasing order.
Posted on

Team Status Report for 2/11

Progress Updates:

  • Scheduled a meeting with the yoga instructor to get pictures on the ideal poses. We also created a plan on questions to ask, including what the typical errors a person makes, what areas to focus on for each pose, etc.
    • The meeting is scheduled for Monday
  • Worked on pose detection algorithm with blaze pose using media pipe’s pose detection algorithm
    • Spent a lot of time understanding the various outputs of the mediapose’s library
    • Created a detailed plan of all the information we need to extract from the poses for the pose correction aspect
    • Build 3D renderings of the poses so we can have a better spatial understanding of the poses

 Our Major Risks so far include:

  • The Pose Correction algorithm does not perform as well as we outlined in our requirements section (>90%)
    • Contingency plan is to have an overall pose accuracy rather than error detection at each node or limb
  • Monitor and hardware acquisition: We are hoping to acquire 2 monitors to construct a full-length mirror, but are not sure how to acquire at a reasonable price and integrate them together along with the two way mirror. 
    • Contingency plan is to have one smaller monitor that acts like a mirror and have the user stand further away so their entire image can still be seen. 
  • Mirror UI: As mentioned in our presentation, another huge risk is how to display our Pose Correction results on the mirror itself and whether any further mapping may be required. 
    • Contingency plan is to simply render the camera feed on monitor

Design Updates includes:

  • From our Proposal Presentation feedback we rescoped our project by 
    • Removing hand gesture input and profile features to focus on creating a robust and reliable Pose Detection and Correction algorithm
    • Testing the algorithm on the computer for now and removing the Raspi integration
    • Solidified the workflow of the project and further divided out tasks

Ethical and Non-Engineering Considerations:

  • Device must be inclusive of all body types, shapes and sizes and provide the same objective user experience to all people irrespective of these physical differences
  • PosePal should not collect or store personally identifiable information to protect user data and establish user trust in using the system solely for their own personal fitness
  • Corrective suggestions should be delivered positively in a way to enhance the learning experience, with positive feedback given by the device to effectively support users
  • As we hope users can rely on PosePal to learn yoga, we need to ensure that the corrections given are actually correct and match what a human yoga instructor would give to avoid leading users incorrectly as they learn different poses

Next Steps:

  • Build a pose correction library based on the discussed plan from this week
  • Complete meeting with yoga instructor to get reference yoga pose models and validate our methodology of error detection and correction (i.e. do the signals we look for to correct poses match what the instructor would look for in an in-person class?)