Tag: Team Status Reports

Progress:

• Completed most of the mirror to camera integration. We were able to compute the scale factor in the x and y direction to make the mirror reflection be drawn over with error renderings
• We wrote out the code to show error renderings for all of the poses, and not just the one that we were testing on before
• We began working on creating audio feedback and have all the individual corrections as audio files.
• Continued to work on building the mirror as we had damaged the monitors and are working on fixing it.

After the interim demo, we scoped out what else we need to get done and the plan seems very doable in time!

Testing:

• So far, we have tested the latency of our system and we are getting real time results which passes our requirements for the latency for the system.
• We have scheduled a meeting with our yoga instructor for after spring break to test the accuracy of our system
• We plan to do the final user tests during the last week of our project.

Next Steps:

• We need to complete building the mirror in a robust manner so it will not constantly break
• We need to build an end to end yoga routine
• We need to integrate the second camera

Progress:

• Wrote the scaling algorithm and created a plan, as well as did a preliminary proof of concept test, to calibrate the camera-mirror scaling factor.
• Designed the hardware and finalized dimensions
• Bought the wood and supplies from home depot and marked their dimensions
• We were supposed to do the wood working and building today but techspark’s woodshop was closed and so have to push that out to early next week.
• Wrote the error rendering code but it needs to be tested some more.

Next Steps:

• We are focusing on the Mirror mapping software system this week, which will entail us recording measurements of key distances such as the distance from the user to the mirror, the depth recorded by our Pose Estimation algorithm, the width of their shoulders or arm span in the mirror reflection, and the actual width/arm span of the user. Once we have these measurements, we will be able to perform a linear regression/interpolate between these data points to produce a line of best fit that represents a correlation between the distance from the mirror and how the reflection shows up on the mirror.
• Any remaining hardware assembly will also be wrapped up this upcoming week.
• We hope to have an MVP of PosePal by the end of the week!

Challenges

• Following up from last week, the major hurdle we are facing is completing the integration of the mirror-mapping logic to successfully render pose correction suggestions directly onto the user’s reflection.
• For this mirror-mapping task, we have finalized the methodology for how to dynamically scale coordinates of the user from the camera feed coordinate space to the coordinate system of the monitor/mirror such that corrective suggestions get rendered in the right position on the reflection
• We went through an example flow of collecting this data and testing the resulting scale factor measured from it through our algorithm

Timeline

• We are in the middle of wrapping up hardware construction and integration of the product itself
• We’ve begun assembly of the frame of the mirror/monitor, and will need to complete final assembly this week to begin testing with the fully integrated product and verify the validity of our mirror-mapping logic and algorithm most importantly

Current Progress:

• Tested the acrylic two way mirror with the monitors. Through the testing, we realized that the acrylic is a much better solution that the mirror film and so decided to stick to using this for the hardware.
• We spent time doing research into the wooden frame construction and the different tools we have at CMU (like techspark and what is needed to do wood working there) to build the frame. We are now designing the wooden frame to build next week.
• We have had extensive ideation sessions on how to map the camera coordinates to the mirror coordinates, which included a lot of trigonometry and physics for mirrors. We now have a much better idea of how we want to implement this and have broken them down into sub tasks.
• We have written a function to calculate the amount by which an object appears smaller as they move away from the mirror, which we will use for understanding which parts of a body is visible on the mirror.

Next Steps:

• Since we only have one work session scheduled during class time due to the ethics discussion, we are planning to meet 2 more times throughout the week to discuss our progress in our respective sections
• We divided out the work into actionable sections that should be completed by the end of next week
  • Youssef: Working on how to render the error corrections (highlight/circle nodes with openCV or using tkinter)
  • Sruti: Working on the Camera to Mirror mapping by scaling different nodes and transforming the coordinates using a camera projection matrix
  • Ankita: Working on planning out the hardware assembly, ordering parts, and potentially creating a CAD model of our product

Challenges/Timeline

• One major challenge is to complete the implementation of our mirror mapping methodology per our outlined approach and verify that it works as expected, tuning as needed
• We also need to complete the pose correction rendering and ensure that users can suitably understand their errors and correct based on these suggestions
• We are on track with our original timeline, but are beginning to have tasks with sequential dependencies, particularly on the hardware construction of the mirror itself. We need to have this completed in order to most accurately test the mirror mapping work and implementation as well as the final pose correction rendering

Current Progress

• Pose detection and correction sub-section completed: Completed pose comparison algorithm for all the poses
• Tested and tuned parameters of five out of the six yoga poses
• Received the monitors and two way mirror sheet. Tested the smart mirror with the mirror sheet and monitors. The results are pretty good, we can see the monitor’s display quite clearly through the two way mirror. However, since the sheet is thin, the reflection of the person gets blurred as the person moves further from the mirror. So, while we think this is a good back up option, we want to try the acrylic mirror and pick the best of the two. We ordered the acrylic mirror which we will be able to test after spring break and finalize our hardware.

• The pose correction algorithm can identify which parts of the body have errors and what needs to be done to fix them, which will help the error rendering sub-system.

Challenges

• The major upcoming challenges our team will face are related to translating the corrections generated from our completed Pose Correction component into visually and auditorily helpful corrective suggestions for users of the final product.
  • We currently associate each possible pose error with a text description on the backend but will need to find an extendable and flexible way to translate these into the visual and auditory feedback users receive to make sure the use case/design requirements addressing usability are satisfied
• We also need to complete the actual hardware integration and try out using the higher-quality acrylic mirror

Timeline

• Our original timeline expected hardware assembly for the mirror itself to be completed by now, but using one of our scheduled-in slack weeks before spring break, we are still on track with our original timeline.
• We should complete hardware setup of the mirror and camera by the end of the week and begin working on the mirror-camera coordination as the next major task for the following week

New tools for Sub-Systems

• Mirror Hardware System: Table saw, Wood glue, Nails, Sandpaper, Drill, Framing square
• Mirror UI: Magic Mirror library, tkinter

Next Steps

• The acrylic mirror should arrive within the next week, so we are aiming to finish the hardware assembly for the mirror.
• The rest of the poses for our Pose Correction component still need to be tested with the parameters for the Downward Dog pose especially needing a lot of tuning as the angles we specified are slightly too strict.
• The Mirror UI and mapping component is our next major software component after we wrap up the Pose Correction algorithm and according to our Gantt chart, we are planning to start working on the error display component next week.

Progress Updates:

• We continued working on the Pose Correction algorithm and completed the functions to check whether the current pose is within the defined threshold of the reference pose (as determined by extracting the notable node configurations from the yoga instructor’s ideal images)
• We researched different monitors that we could potentially use to construct a full-length mirror and decided to use a seamless, dual monitor set-up resulting in a final length of 48 in, which we determined was sufficient enough to account for most heights
  • We decided to use the flexible 2-way mirror sheet instead of the acrylic as an initial prototype as it was cheaper, thin (so the LCD screen would shine through), and had a variable length (as we were initially not sure which monitors we would end up using).

Risks:

• The primary risk so far is the Mirror hardware construction as we are not sure whether the 2 LCD monitors will be able to stick together using just the wooden frame or if additional measures will need to be taken
• If the wooden frame is not sufficient enough, we could also attach both monitors to an external piece of wood and overlay the mirror on top of that subassembly.

Changes to schedule and team work:

• No other schedule changes have been made and we will attempt to finish our backlogged tasks this week
• The pose correction is our top priority and all teammates are working on that right now but we will continue with our task split up after this
• There are no issues with our teamwork and accountability

Next Steps:

• Continue testing and refining the threshold parameters for whether the current pose is within error margins and can be classified as correct
• After receiving feedback on our Design Presentation, we will adjust our verification and validation procedures to create more quantitative and robust checks to ensure our design meets our stated requirements
• Contingent upon our materials arriving this week, we will start assembling the wooden frame encapsulating our LCD screens, attach the LCD screens together, and apply the 2-way mirror sheet to complete our hardware assembly
• For software, we will completely finish the Pose Correction algorithm and start testing it at various angles and distances to ensure it accounts for those discrepancies

Progress Updates:

• Had a meeting with the yoga instructor we are working with to get ideal poses and learn about typical errors and metrics to correct yoga poses
  • Collected reference pose images for all the 6 poses we want our system to have
  • Currently working on analyzing the poses to determine which nodes and what angles should be used for pose comparison. We are also computing pose detection values and storing them so it does not need to be recomputed each time.
  • Yoga expert helped us understand additional error signals used in in-person instruction that should be present in our pose correction system, most notable of these being the concept of alignment along axes on a person’s pose (for example, arms and shoulder should be collinear)
• Implemented library for Pose Correction component to compute:
  • Euclidean angles given 3 joints on reference pose
  • Distance between 2 joints (to check if they are in contact)
  • Check if all the required joints are visible (to make sure they are in frame, and that there joints are not one behind the other)
• Developed our design review presentation:
  • Fleshed out our system’s design
  • Better understood how to implement sub-systems
  • Practices and created slides

 Our Major Risks so far include:

• Monitor and hardware acquisition: We are hoping to acquire 2 monitors to construct a full-length mirror, but are not sure how to acquire at a reasonable price and integrate them together along with the two way mirror. 
  • Contingency plan is to have one smaller monitor that acts like a mirror and have the user stand further away so their entire image can still be seen. 
• Mirror UI: As mentioned in our presentation, another huge risk is how to display our Pose Correction results on the mirror itself and whether any further mapping may be required. We have spent some time working through the physics of it so it seems more plausible now but is still a risk until we acquire the hardware.
  • Contingency plan is to simply render the camera feed on monitor

Guiding Engineering/Scientific Principles For Our Design

• Machine Learning/Computer Vision:
  • Core components of our project like the pose estimation capabilities rely on computer vision ML models. The model we’ve selected to use is a convolutional neural network (CNN) model called BlazePose that is optimized for real-time, on-device inference
• Computer Graphics:
  • Need to use efficient and lightweight computer graphics libraries/methods in order to render pose corrective suggestions in real-time in a visually understandable way
• Software engineering:
  • Working with several python modules and writing a custom pose comparison library
• Geometry:
  • Pose comparison is based on angles and distances, which are rooted in 2D and 2D geometry
• Physics of light and mirrors:
  • Using the behavior of light and principles of reflection to understand where a person’s reflection will appear on the mirror
• Firmware and hardware-software interface:
  • Integrating camera input with the computer
  • Working with speakers for audio output
• Human-Computer Interaction (HCI):
  • Have HCI as an additional guiding principle to make sure that the final integrated system is easily usable and accessible for users.
  • Ensure that the UI and corrective feedback are delivered in a way that is understandable and doesn’t compromise the yoga learning experience (deliver audio cues to avoid user physically interacting with the screen for example)

Timeline Update:

For our software algorithm, we are on track. However, we were supposed to buy the hardware last week. We spoke to the TAs about using monitors that are already present in the lab and are waiting on that information to go ahead with the purchase of materials. We hope to make that purchase this week, which will push back our hardware timeline by a week.

Next Steps:

• Complete error comparison this week
• Order hardware and begin assembly of hardware
• Complete design presentation

Progress Updates:

• Scheduled a meeting with the yoga instructor to get pictures on the ideal poses. We also created a plan on questions to ask, including what the typical errors a person makes, what areas to focus on for each pose, etc.
  • The meeting is scheduled for Monday
• Worked on pose detection algorithm with blaze pose using media pipe’s pose detection algorithm
  • Spent a lot of time understanding the various outputs of the mediapose’s library
  • Created a detailed plan of all the information we need to extract from the poses for the pose correction aspect
  • Build 3D renderings of the poses so we can have a better spatial understanding of the poses

 Our Major Risks so far include:

• The Pose Correction algorithm does not perform as well as we outlined in our requirements section (>90%)
  • Contingency plan is to have an overall pose accuracy rather than error detection at each node or limb
• Monitor and hardware acquisition: We are hoping to acquire 2 monitors to construct a full-length mirror, but are not sure how to acquire at a reasonable price and integrate them together along with the two way mirror. 
  • Contingency plan is to have one smaller monitor that acts like a mirror and have the user stand further away so their entire image can still be seen. 
• Mirror UI: As mentioned in our presentation, another huge risk is how to display our Pose Correction results on the mirror itself and whether any further mapping may be required. 
  • Contingency plan is to simply render the camera feed on monitor

Design Updates includes:

• From our Proposal Presentation feedback we rescoped our project by 
  • Removing hand gesture input and profile features to focus on creating a robust and reliable Pose Detection and Correction algorithm
  • Testing the algorithm on the computer for now and removing the Raspi integration
  • Solidified the workflow of the project and further divided out tasks

Ethical and Non-Engineering Considerations:

• Device must be inclusive of all body types, shapes and sizes and provide the same objective user experience to all people irrespective of these physical differences
• PosePal should not collect or store personally identifiable information to protect user data and establish user trust in using the system solely for their own personal fitness
• Corrective suggestions should be delivered positively in a way to enhance the learning experience, with positive feedback given by the device to effectively support users
• As we hope users can rely on PosePal to learn yoga, we need to ensure that the corrections given are actually correct and match what a human yoga instructor would give to avoid leading users incorrectly as they learn different poses

Next Steps:

• Build a pose correction library based on the discussed plan from this week
• Complete meeting with yoga instructor to get reference yoga pose models and validate our methodology of error detection and correction (i.e. do the signals we look for to correct poses match what the instructor would look for in an in-person class?)