Posted on

Hongzhe’s Status Report for Week 11/18/2023

For this week, we are trying our best to finalize all the functionality portion of the project given there is no scheduled task from the course schedule.

Personally, in charge of integration, I did a lot of communication all over the team to finalize the details for each part. We came up with the idea that since users need more distance in order to be fully captured by the built-in camera (assuming they are not buying another one), the training screen needs to be reformatted so that all visual data could be better seen. Thus, I manually reprocessed all the reference poses. At the same time, I was able to work with Ray to finish the settings page, where we use scroll bars to control setting parameters such as pre-train hold duration etc. I am working on a different logic for the training page and get front support with Ray to change the way users start and end the training/instruction procedure, avoiding redundant position changes to access their PC.

For the next week during Thanksgiving, we will try our best to finish the application and move on to schedule interviews and start organizing verification.

Posted on

Ray’s Status Report for 11/18/2023

This week, our team mainly worked on frontend/backend integration, and I took most of my time working on the skeleton draw functionality.

In order to make our system as intuitive as possible, we will have a screen showing direct comparison between the reference pose and the user pose on the training page of our application. Specifically, we want to draw the skeleton of the reference pose and then overlay the user pose on top of it, so that the user can intuitively see which of their limbs have incorrect angles.

I implemented the algorithm for drawing the reference skeletonthis week. It’s much harder than I expected since I need to conducts a couple of transforms on the data derived from openpose. The coordinate system of openpose is very different from Kivy’s, and I need to find a way to center the drawn skeleton. Also, to process the data outputed from the backend, I read through the entire backend code and pointed out a few places where the naming of the variables does not fully represent what they are. For now, the drawn reference skeleton looks good, and I will attempt the drawing of the user skeleton this weekend.

Also, in order to maximize the portion of screen used so that users can see their poses from far away, I changed the formatting of the training screen, which now has the look shown by the image above.

Meanwhile, I also helped a bit with Jerry’s frontend pose sequence task where we both tried to understand how relative layout works.

I think we are still on schedule with our project and our application is in good shape. Still, we will probably need the thanksgiving break to fully integrate the system and test out the bugs. After finising the skeleton drawing algorithm, I will help Jerry work on the pose sequence functionality.

Posted on

Ray’s Status Report for 11/11/2023

I had made substantial progress on frontend this week. The first accomplished task is the dynamic insertion of widgets into the selection page is finally up and working. I designed a custom widget to hold both the previews and the names of the postures. Then all the widgets are added into the menu using python code instead of hard coding it with kv language. Below is the result:

Clicking the row that previews the pose will bring you to the training page and begin the training on that pose. This is, in my opinion, optimal in terms of both aesthetics and intuition.

The other thing I implemented is the mode page. Since we have both integrated postures of 24-form Tai Chi and custom postures from users, I created the mode page where the user can choose which set of postures they want to play with.

Clicking the upper block will bring the user to the same page as what the first image in this post shows. Clicking the lower block will bring the user to this page instead:

Here, the custom user poses imported through Jerry’s pipeline are displayed. Implementing this new page took a while since a lot of the page connection logic needs alterations.

I also started the frontend-backend integration with Eric and we have cooperated and agreed on variables and features that needs introduction into the frontend. This will extend into next week and will be my focus on the project.

I believe I’m well on schedule and our team would see completion of the MVP quite soon. Still, a lot of effort is still required for reaching that goal.

ABET:  On design requirements,  we need to make sure the system meets the minimum performance in detecting body part degree differences, the fps of the application, the accuracy of the verbal instructions, and the correctness of the user interface. Shiheng Wu and Hongzhe Cheng have conducted tests on the backend to make sure the first three design requirements are met. For the correctness of the user interface, Jerry Feng and I have checked the interconnection of the pages and made sure that the user are able to easily navigate through each of the pages, and  has ensured that when users uploaded images through the customization pipeline, they are displayed correctly in selection page and can be used as references in the training page. All tests for the subsystems went well.

We will redo all of the above tests on the full system with communication between the frontend and the backend correctly and robustly integrated. We will further conduct tests on user requirements, including testing the posture detection accuracy, verbal instruction latency, and accessibility of the system. In particular, we will use the systems ourselves and see if it meets our expectation for the system’s functionality and smoothness. We will also each invite three of our aquaintances to try the system out and receive feedback on all three user requirements from them. We can then improve the user experiences of our system based on our own evaluations and the testers’ suggestions.

Posted on

Shiheng’s Status Report for 11/11/2023

This week I continued developing the backend part of our software.
There were some back and forth between the group members on project details, but they were soon resolved and we were quickly back on track.

Following Monday’s interim demo session, I worked with Eric on hand posture detection as our demo did not detect them. We oversimplied the model and made the comparision inaccurate.
Fortunately it got resolved on the afternoon of the same day and we are now able to detect user’s hand posture. User will be reminded that their hand is not in the correct posture, e.g doing a fist instead of flat hand.

I also worked on more functionalities this week on the backend, including dividing and identifying body parts into ‘head’, ‘torso’, ‘arms’, ‘legs’, ‘feet’, communicating with Ray and Jerry about feedbacks needed on the frontend. The functionaility is mostly finished for normal picture (i.e user with full body included in frame with clear reference picture).

The plan for next week and following weeks is to implement the cases where unclear pictures are provided, or user is not posing their body inside the frame. I have been working closely with Eric’s test cases to identify potential issues from incoming rendered images and json files. Functions that need to implement are:
Calibration for one’s body and identifying their missing keypoints -> Prompt them to change their posture and include the full body in frame.
Multiple people in frame -> Use similiarity scoring to identify the actual user
Instruction wording -> Clear consise commands to user, and priortize lower body parts in training
Potential Integration issues -> Fix them as the project progresses
I also need to work with members on frontend about progress and implementation.

ABET question:
1. Unit testing on backend code: Lots of pre and postconditions are currently implemented for unit testing and debugging purposes
2. Image testing: Test on the go, using user inputs and reference images handpicked to test the functionality of the code
3. User feedback testing: Voice instructions will be played to user which will be evaluated on their quality and clarity
4. Edge case testing: Cases where improper images are uploaded? User not/partly in frame? User doing a completely different pose?

Posted on

Team Status Report for 11/11/2023

Following the interrim demo, our group continued to work on the frontend and backend of our system to further polish the functions and begin the process of integration.

The imminent problem mentioned during the interim demo about palm angles raised by professor Tamal was resolved in this week, as Shiheng and Eric worked through the openpose system and figured detecting keypoints on hands to determine hand postures.

We continued to discuss details for the future weeks of the project and made various important design decisions on certain details of our software.  We synced up our progress in Saturday meeting with Eric explaining integration steps of the software, Shiheng explaining the approach on the backend about pose comparison, Ray and Jerry working on the pipeline and communicating with the needs and ideas that backend needs to pass onto the frontend and UI.

Steps to get the application working:

  • Everybody work on main
  • Git clone from main branch into folderX
  • Copy everything from folderX into openpose root directory (C:/xxx/openpose/)
  • C:/xxx/openpose/ – Python3 main.py
  • Remember to recursively copy all the things, and remember to have bin/openposedemo.exe ready

Things to Do:

Front end:

    • Calibration (Integrated)
    • Pretty UI?
    • Sequence (Integrated in Training Iteration)
    • Skeleton Drawing – Ray (Discuss with Roland(shiheng))
      • Parse JSON to get coordinate/angle
        • Roland: JSON from openpose contains all coordinates of bodypose being captured, will just connecting the dots get an appropriate skeleton?  Normalization choice: Scaling the whole body by a factor (rough match)/tailoring limbs and body parts (exact match)?
      • Redraw the skeleton
      • Mark red for incorrect body parts
        • Roland: Pass back an error_list containing all the incorrect body parts above the threshold. No prioritizing body parts here, all incorrect body parts will be passed back.
    • Custom Upload renaming – Done by Jerry
  • Training Iteration – Jerry (Discuss with Eric for integration)
    • User tries to start training by clicking “Start” button in the training page
    • Camera get user image and send to backend (10s countdown)
    • User image processed by OpenPose to get user coordinates
    • Case 1: user coordinates contains every body part
      • Front end receives correct status code/return value
      • Display skeleton, verbal instructions is reading
      • ***If the user is correct, change to the next pose – Sequence working!!!
    • Case 2: user coordinates does not contain every body part
      • Front end receives bad return value
      • Display a flash warning saying your body is not in camera
      • Verbal speak “Change your position”
    • Iterate back to step 2
    • “Stop” button clicked to stop process or change pose…

Back end:

  • Error Priority – Roland
    • Feet->Leg/Thigh->Upper Body(Torso)->Arms, Instructions should be issued from this sequence (e.g Errors in Arms shouldn’t be mentioned before errors in lower body is fixed)
    • One instruction/error at a time to not confuse users learning Taichi
  • User Selection (Multiple people in frame chose the user in training) – Roland
    • Choosing the most similar person in frame (Similarity scoring)
    • Openpose input formatting for multiple people?
  • Instruction Wording – Roland
    • Dividing body parts into ‘head’, ‘torso’, ‘arms’, ‘legs’, ‘feet’, giving different instructions for corresponding body parts
  • Macro connection – Eric
    • Continue to integrate the frontend and backend, work with rest of the team

Later:

  • Verification & Performance Evaluation, Data collection for the final report (Second Last week)
  • User interview and UI test, report, slide, presentation. (Schedule for those events and start right after the integration is finished and functionalities achieved)
Posted on

Hongzhe’s Status Report on 11/11/2023

For this week, we were able to demonstrate our work in the interim demo to the staff member and show some of our progress. Given that we started late, I believe our team has made great efforts up until now. 

Personally, I achieved the following. Given the precious feedback from Professor Tamal, we found the bug where we were not capturing the angle for wrists. I was able to find the support for OpenPose on hand posture processing and update the backend model with Shiheng together to fix the issue. In order to do that, I also reprocessed all the reference dataset for future usage. At the same time, since OpenPose serves right in between the front and back end, I am now in charge of integration. I was able to integrate the real-time pipeline such that after hitting the capture button, the user image would be saved and the back end comparison procedure is called.

For the future weeks, I will keep integrating changes and parameters that go between the front and back end. Just to mention here that I will mostly work on integration on the real-time pipeline and Jerry would control his own.

ABET: Tests that I have created include Full back end functionality test and OpenPose functionality test. Basically the ones shown in interim demo.

Tests to be implemented particularly for my work would be the following (related to OpenPose and integration – full pipeline):

  1. Full pipeline functionality test: to test that the integration works fine and front and back ends are well connected
  2. OpenPose performance test: to see how much processing time is taken by OpenPose
  3. User Latency Test: to obtain the latency from the second data is captured from camera to when user first get verbal/visual feedback
Posted on

Ray’s Status Report for 11/04/2023

This week I was preparing for the interim demo components for my part on the project.  In particular, I specifically worked on the training page to make sure the camera and the preparation time function for our application works.

The training page is now connected to the seleciton page and its camera is now working. Pressing the Capture button will start a countdown, and after the countdown, the footage of the user will be captured. Now the captured footage will be saved as an image file, but the actual pipeline we want to realize is to send the image to backend for comparisons. The connection with backend is my major task for next week.

Meanwhie, I managed to dynamically add widgets to the selection page. Now the main task is to create custom widgets so that each pose selection button has a preview of what it looks like. This will also be part of my work next week.

I also reached out to professionals for interviews on Taichine Training and searched for papers. This is to support our argument that the user’s feet postions should be used as the root for postures in comparison functions. I managed to get consent to interview for next week, and I also found a few relevant papers supporting our arguments.

Also, I set up a repository for our project to help our team integrate the full application. The UI code has worked successfully on all team members’ laptops, which is good news!

Overall, I am back on schedule and we can finally start our overall integration next week. I’m looking forward to it!

Posted on

Shiheng’ Status Report for 11/04/2023

I dedicated additional time to refining the comparison algorithm in our project, which focuses on capturing single image inputs with local reference posture. During our Friday meeting, the rest of the team conducted a code review of the algorithm and highlighted several areas that needed optimization. This was necessary because some parts of the code had originally been hardcoded for testing purposes, Hongzhe helped me pointed out some of the issues in my code. After addressing and enhancing the script based on our team members’ feedback, we managed to align it with most of the initial design goals. This included functionality for scoring, generating joint angles as outputs, and integrating our voice engine. I worked closely with Hongzhe this week in developing scripts and practiced some unit testing in different environments based on his output from the Openpose script.

 

As for the voice module implementation, we have not yet settled on a specific approach. I will continue collaborating with the rest of the team to determine how to incorporate the voice component into our program. I am currently attempting to write a separate script to call on and test generated voice files, but I would need to do some testing before integrating that part possibly on Sunday or during the next week. If the separate script could not be integrated in time, we will do a manual approach for the interim demo on this part and just showcase that the generated .wav files could be played.

Posted on

Hongzhe’s Status Report on 11/04/2023

In the past week, each team member is making substantial progress on building on some visible and usable components in the project and starting to integrate for the interim demo. At the same time, during the lecture on Monday, we learned a lot from the ethics lecture on what we should consider when being an engineer creating technology that makes not only technical impact. On Wednesday, we also got some valuable feedback from Professor Byron to check the status of what we would show on the interim demo.

Personally, in the past week, I accomplished the following things. First of all, utilizing the interface I created last week, I was able to process all the reference data we have on the default 24 Tai Chi Poses and generate the coordinate data. The data will be further used by Roland on testing comparison algorithms and for reference in the final product. Secondly, providing advice to my teammates, now everyone has OpenPose successfully built on their Windows PC. Last but not least, in preparing the interim demo, I was able to create a program that connects OpenPose and the comparison algorithm. The subunit test takes in a user image and reference pose name, processes the user image to get coordinates, compares the user and reference coordinates to generate text feedback. Note that this is the middle portion of the final product, we will further connect the real front-end app and camera interface and the text-to-speech app in the back to perform the functionality in the future. We might be able to integrate more functions tomorrow before the demo, but this is what I have been working with Roland to achieve.

In future weeks, given that the OpenPose infrastructure is mostly done, I will be focusing on integration, and some sub parts of app infrastructure such as skeleton drawing etc. We will also try to reach out to some interviewees to get some further confirmation on project details.

Posted on

Team Status Report for 11/4/2023

Our group is making progress according to schedule this week and preparing for the upcoming interim demo.

Ray and Jerry are collaborating on the Kivy (UI) design pipeline, there have been various implementation issues including environment switches (Linux to Windows) and learning the different methodology that Kivy provided compared to traditional Python. Kivy also provided various widgets that seemed easy to implement and apply to our program, but they turned out to be a lot more complex than we previously estimated. Fortunately, we collaborated during our usual meeting on Friday to debug and discuss elements of the UI design, which fixed most of the issues we previously had in development. Ray also worked on the file management part in creating a relatively primary file management system for us to store reference and user poses.

In addition to the work previously mentioned, Jerry has been making significant contributions to the development of his pipeline for single file uploads. His efforts have proven to be highly successful, as his code has been integrated into the main program developed by Ray. This integration marks a crucial step forward in the application’s functionality, allowing for the straightforward acceptance and processing of basic single images. With this foundation in place, Jerry is poised to continue his work on expanding the application’s capabilities. His next focus will be on further enhancing the application’s features by enabling it to accept and utilize pose sequences.

Eric (Hongzhe) continues to work on the Openpose system and its integration into our application. Eric is also learning Kivy along with Ray and Jerry to speed up the application development and integration of pose recognition. Continuing his progress from last week, Eric also did extensive testing on various Taichi pose pictures to make the skeleton from Openpose to overlay on the original picture. He is also working on file directory for output json files and with Shiheng for him to accept the inputs for comparison algorithm from our designated directory. Eric also helped with debugging throughout the programing both on Kivy and communicating demands with Shiheng’s comparison algorithm, which he would gladly provide postures for Shiheng to do unit testing for his algorithm.

Shiheng worked more on his own about the comparison algorithm for our design of capturing single image inputs. However, during the Friday meeting we held, the rest of the group did code review on the algorithm and pointed out various issues to be optimized, since some parts of the code were originally being hardcoded for testing purposes. After fixing and improving the script from team member’s advice, it could achieve most of the designs we planned initially, including scoring, producing joint angles as outputs, and invoking our voice engine. Since we have not decided how to implement the voice module into our program, Shiheng will continue to work with the rest of the group about playing generated voice instructions and polishing on the details will be the key of achieving a robust Taichi instructor tool.