Posted on

Ray’s Status Report for 11/11/2023

I had made substantial progress on frontend this week. The first accomplished task is the dynamic insertion of widgets into the selection page is finally up and working. I designed a custom widget to hold both the previews and the names of the postures. Then all the widgets are added into the menu using python code instead of hard coding it with kv language. Below is the result:

Clicking the row that previews the pose will bring you to the training page and begin the training on that pose. This is, in my opinion, optimal in terms of both aesthetics and intuition.

The other thing I implemented is the mode page. Since we have both integrated postures of 24-form Tai Chi and custom postures from users, I created the mode page where the user can choose which set of postures they want to play with.

Clicking the upper block will bring the user to the same page as what the first image in this post shows. Clicking the lower block will bring the user to this page instead:

Here, the custom user poses imported through Jerry’s pipeline are displayed. Implementing this new page took a while since a lot of the page connection logic needs alterations.

I also started the frontend-backend integration with Eric and we have cooperated and agreed on variables and features that needs introduction into the frontend. This will extend into next week and will be my focus on the project.

I believe I’m well on schedule and our team would see completion of the MVP quite soon. Still, a lot of effort is still required for reaching that goal.

ABET:  On design requirements,  we need to make sure the system meets the minimum performance in detecting body part degree differences, the fps of the application, the accuracy of the verbal instructions, and the correctness of the user interface. Shiheng Wu and Hongzhe Cheng have conducted tests on the backend to make sure the first three design requirements are met. For the correctness of the user interface, Jerry Feng and I have checked the interconnection of the pages and made sure that the user are able to easily navigate through each of the pages, and  has ensured that when users uploaded images through the customization pipeline, they are displayed correctly in selection page and can be used as references in the training page. All tests for the subsystems went well.

We will redo all of the above tests on the full system with communication between the frontend and the backend correctly and robustly integrated. We will further conduct tests on user requirements, including testing the posture detection accuracy, verbal instruction latency, and accessibility of the system. In particular, we will use the systems ourselves and see if it meets our expectation for the system’s functionality and smoothness. We will also each invite three of our aquaintances to try the system out and receive feedback on all three user requirements from them. We can then improve the user experiences of our system based on our own evaluations and the testers’ suggestions.

Posted on

Shiheng’s Status Report for 11/11/2023

This week I continued developing the backend part of our software.
There were some back and forth between the group members on project details, but they were soon resolved and we were quickly back on track.

Following Monday’s interim demo session, I worked with Eric on hand posture detection as our demo did not detect them. We oversimplied the model and made the comparision inaccurate.
Fortunately it got resolved on the afternoon of the same day and we are now able to detect user’s hand posture. User will be reminded that their hand is not in the correct posture, e.g doing a fist instead of flat hand.

I also worked on more functionalities this week on the backend, including dividing and identifying body parts into ‘head’, ‘torso’, ‘arms’, ‘legs’, ‘feet’, communicating with Ray and Jerry about feedbacks needed on the frontend. The functionaility is mostly finished for normal picture (i.e user with full body included in frame with clear reference picture).

The plan for next week and following weeks is to implement the cases where unclear pictures are provided, or user is not posing their body inside the frame. I have been working closely with Eric’s test cases to identify potential issues from incoming rendered images and json files. Functions that need to implement are:
Calibration for one’s body and identifying their missing keypoints -> Prompt them to change their posture and include the full body in frame.
Multiple people in frame -> Use similiarity scoring to identify the actual user
Instruction wording -> Clear consise commands to user, and priortize lower body parts in training
Potential Integration issues -> Fix them as the project progresses
I also need to work with members on frontend about progress and implementation.

ABET question:
1. Unit testing on backend code: Lots of pre and postconditions are currently implemented for unit testing and debugging purposes
2. Image testing: Test on the go, using user inputs and reference images handpicked to test the functionality of the code
3. User feedback testing: Voice instructions will be played to user which will be evaluated on their quality and clarity
4. Edge case testing: Cases where improper images are uploaded? User not/partly in frame? User doing a completely different pose?

Posted on

Team Status Report for 11/11/2023

Following the interrim demo, our group continued to work on the frontend and backend of our system to further polish the functions and begin the process of integration.

The imminent problem mentioned during the interim demo about palm angles raised by professor Tamal was resolved in this week, as Shiheng and Eric worked through the openpose system and figured detecting keypoints on hands to determine hand postures.

We continued to discuss details for the future weeks of the project and made various important design decisions on certain details of our software.  We synced up our progress in Saturday meeting with Eric explaining integration steps of the software, Shiheng explaining the approach on the backend about pose comparison, Ray and Jerry working on the pipeline and communicating with the needs and ideas that backend needs to pass onto the frontend and UI.

Steps to get the application working:

  • Everybody work on main
  • Git clone from main branch into folderX
  • Copy everything from folderX into openpose root directory (C:/xxx/openpose/)
  • C:/xxx/openpose/ – Python3 main.py
  • Remember to recursively copy all the things, and remember to have bin/openposedemo.exe ready

Things to Do:

Front end:

    • Calibration (Integrated)
    • Pretty UI?
    • Sequence (Integrated in Training Iteration)
    • Skeleton Drawing – Ray (Discuss with Roland(shiheng))
      • Parse JSON to get coordinate/angle
        • Roland: JSON from openpose contains all coordinates of bodypose being captured, will just connecting the dots get an appropriate skeleton?  Normalization choice: Scaling the whole body by a factor (rough match)/tailoring limbs and body parts (exact match)?
      • Redraw the skeleton
      • Mark red for incorrect body parts
        • Roland: Pass back an error_list containing all the incorrect body parts above the threshold. No prioritizing body parts here, all incorrect body parts will be passed back.
    • Custom Upload renaming – Done by Jerry
  • Training Iteration – Jerry (Discuss with Eric for integration)
    • User tries to start training by clicking “Start” button in the training page
    • Camera get user image and send to backend (10s countdown)
    • User image processed by OpenPose to get user coordinates
    • Case 1: user coordinates contains every body part
      • Front end receives correct status code/return value
      • Display skeleton, verbal instructions is reading
      • ***If the user is correct, change to the next pose – Sequence working!!!
    • Case 2: user coordinates does not contain every body part
      • Front end receives bad return value
      • Display a flash warning saying your body is not in camera
      • Verbal speak “Change your position”
    • Iterate back to step 2
    • “Stop” button clicked to stop process or change pose…

Back end:

  • Error Priority – Roland
    • Feet->Leg/Thigh->Upper Body(Torso)->Arms, Instructions should be issued from this sequence (e.g Errors in Arms shouldn’t be mentioned before errors in lower body is fixed)
    • One instruction/error at a time to not confuse users learning Taichi
  • User Selection (Multiple people in frame chose the user in training) – Roland
    • Choosing the most similar person in frame (Similarity scoring)
    • Openpose input formatting for multiple people?
  • Instruction Wording – Roland
    • Dividing body parts into ‘head’, ‘torso’, ‘arms’, ‘legs’, ‘feet’, giving different instructions for corresponding body parts
  • Macro connection – Eric
    • Continue to integrate the frontend and backend, work with rest of the team

Later:

  • Verification & Performance Evaluation, Data collection for the final report (Second Last week)
  • User interview and UI test, report, slide, presentation. (Schedule for those events and start right after the integration is finished and functionalities achieved)
Posted on

Hongzhe’s Status Report on 11/11/2023

For this week, we were able to demonstrate our work in the interim demo to the staff member and show some of our progress. Given that we started late, I believe our team has made great efforts up until now. 

Personally, I achieved the following. Given the precious feedback from Professor Tamal, we found the bug where we were not capturing the angle for wrists. I was able to find the support for OpenPose on hand posture processing and update the backend model with Shiheng together to fix the issue. In order to do that, I also reprocessed all the reference dataset for future usage. At the same time, since OpenPose serves right in between the front and back end, I am now in charge of integration. I was able to integrate the real-time pipeline such that after hitting the capture button, the user image would be saved and the back end comparison procedure is called.

For the future weeks, I will keep integrating changes and parameters that go between the front and back end. Just to mention here that I will mostly work on integration on the real-time pipeline and Jerry would control his own.

ABET: Tests that I have created include Full back end functionality test and OpenPose functionality test. Basically the ones shown in interim demo.

Tests to be implemented particularly for my work would be the following (related to OpenPose and integration – full pipeline):

  1. Full pipeline functionality test: to test that the integration works fine and front and back ends are well connected
  2. OpenPose performance test: to see how much processing time is taken by OpenPose
  3. User Latency Test: to obtain the latency from the second data is captured from camera to when user first get verbal/visual feedback