Posted on

Jerry’s Status Report for 11/18/2023

This week I am excited to announce that I have finished putting together pose sequence support for the custom images upload page.  I was able to have dynamic image removal, and reordering from the user.  The user is also able to name the pose they would like to use, with the default name of the pose being the name of the first image in the sequence.  Users upload sequences by selecting multiple files from the computers native file manager that displays as a popup.  Full functionality can be seen in the video below:

https://youtu.be/ZXbXMUYbDSE  (In the video I accidentally say custom image upload support, I meant custom pose sequence upload support.)

I believe I am right on track with an adjusted schedule that my team has put together, where we wanted to get custom pose sequence support finished before Thanksgiving.  Since the meat of custom pose sequence support (functionality of dynamic changes to poses in upload phase and upload in correct ordering) is done, I believe I am well on track with our adjusted schedule.

In terms of deliverables, I hope to be able to finish tweaking the button sizes so that the buttons do not overlap with images when only 1 image is being uploaded.

ABET Question:

Many of the ways we have grown as a team is that as we have worked together more, our communication as a team has gotten better and better.  Originally, before I joined, the team communicated predominately through WeChat but they have been very gracious in switching to Slack to better accommodate me.  We have also gotten better at adjusting to using Slacks functionalities to organize questions we ask each other to coordinate the integration of our project.  We have also gotten better at actively maintaining an open atmosphere in our team meetings so everyone can voice their opinions and have a say in the direction we want to take our project as initially I think we were a little reticent to voice our opinions, but now everyone is comfortable sharing their opinions and discussing differences in view points to ensure a better project.

 

Posted on

Shiheng’s Status Report for 11/18/2023

My work this week reflects the efforts I made on the backend on integration with Ray and continued progress in implementing vocal instructions.

I have been actively driving progress in the project, specifically focusing on enabling pose selection from multiple persons. My work involves extensive Python scripting to develop a flexible system that is automated to choose poses from different individuals and pick out the Taichi practitioner. User will be able to train in environment not so private (e.g. gyms) without the need of having the room cleared to prevent the system from capturing other body parts.

Additionally, I have taken on the responsibility of building the backend support for skeleton drawing. Through passing angles with a reference frame, I enabled Ray to pinpoint and draw the vectors for the reference skeleton. The user skeleton follows the similar logic, and I have lay down the foundation for comparison through passing a boolean list for verfication purposes and creating visual cues on the front end.

I am still working and researching on creating good vocal instructions while putting the sequence and body parts priority in mind, which should be done by the end of this weekend and ready for testing. For following weeks, I will focus on testing and fixing bugs from our final product with all the functions done.

Posted on

Team Status Report for 11/18/2023

As we are approaching the end of semester, our team worked on integration and finetuning the infrastructure for the upcoming testing and verfication.

Jerry has been focused on enhancing the user experience by introducing custom pose sequence support. He’s also succeeded in integrating the app with file manager to handle multiple files, providing users with an efficient way to manage their custom pose uploads. Additionally, Jerry has improved user interaction by enabling the removal and reordering of files to be uploaded, directly within the app’s display.

Shiheng has taken on the challenge of allowing pose selection from multiple persons, a crucial feature for users within more complex environments and handling different outputs when multiple bodies are found in frame. In addition, he is concentrating on backend support for skeleton drawing for Ray, implementing a new and different approach for measuring angles that could be handled better on the frontend without the need for processing the raw data from Openpose.

Hongzhe (Eric) is dedicated to refining the visual aspects of the application. His work involves processing reference images to enhance their display quality. He is also focusing on establishing parameter connections, which will contribute to a more cohesive and user-friendly experience. Additionally, Hongzhe is also involved in fine-tuning the logic on the training page to ensure an optimized user journey.

Ray is playing a pivotal role in consolidating the efforts of the team. He is synthesizing key points and reference angles provided by Shiheng into the overall skeleton drawing process. Ray is also leading the redesign of UI components for improved aesthetics and usability, working on cleaning up the code and optimization. Furthermore, he is involved in refactoring reference pictures to enhance user experience with bigger picture components for better feedback experience.

ABET Question (How we have grown as a team?):

Throughout the semester, our team has experienced substantial growth and cohesion. We have established a robust framework for collaboration and decision-making, utilizing Google Drive for weekly reports, design documents, Github repo for codes, and Slack channel for communication and collaboration efforts.

Our mandatory lab sessions on Mondays and Wednesdays have provided consistent opportunities for team members to discuss progress and align our efforts. Additionally, our Friday or Saturday sessions over zoom have been dedicated to refining plans and syncing up progress during the week.

The use of Gantt Charts has proven instrumental in visualizing our project timeline and ensuring that tasks are allocated efficiently. This strategic planning has allowed us to stay on track and adapt to unforeseen challenges effectively. By regularly revisiting and adjusting our goals based on progress and feedback, we have fostered a dynamic and adaptive team culture.

Our commitment to an open and inclusive environment has been foundational to our success. We actively seek input from all team members, ensuring that each voice is heard and valued. Though we faced changes in group composition and change of plan in the beginning of the semester, everyone in the group was equally treated.

Thanks to everyone’s effort in creating an equal and exclusive environment, we have been able to make substantial progress from scratch as a group and advance through the hardships we faced throughout the semester.

Posted on

Hongzhe’s Status Report for Week 11/18/2023

For this week, we are trying our best to finalize all the functionality portion of the project given there is no scheduled task from the course schedule.

Personally, in charge of integration, I did a lot of communication all over the team to finalize the details for each part. We came up with the idea that since users need more distance in order to be fully captured by the built-in camera (assuming they are not buying another one), the training screen needs to be reformatted so that all visual data could be better seen. Thus, I manually reprocessed all the reference poses. At the same time, I was able to work with Ray to finish the settings page, where we use scroll bars to control setting parameters such as pre-train hold duration etc. I am working on a different logic for the training page and get front support with Ray to change the way users start and end the training/instruction procedure, avoiding redundant position changes to access their PC.

For the next week during Thanksgiving, we will try our best to finish the application and move on to schedule interviews and start organizing verification.

Posted on

Ray’s Status Report for 11/18/2023

This week, our team mainly worked on frontend/backend integration, and I took most of my time working on the skeleton draw functionality.

In order to make our system as intuitive as possible, we will have a screen showing direct comparison between the reference pose and the user pose on the training page of our application. Specifically, we want to draw the skeleton of the reference pose and then overlay the user pose on top of it, so that the user can intuitively see which of their limbs have incorrect angles.

I implemented the algorithm for drawing the reference skeletonthis week. It’s much harder than I expected since I need to conducts a couple of transforms on the data derived from openpose. The coordinate system of openpose is very different from Kivy’s, and I need to find a way to center the drawn skeleton. Also, to process the data outputed from the backend, I read through the entire backend code and pointed out a few places where the naming of the variables does not fully represent what they are. For now, the drawn reference skeleton looks good, and I will attempt the drawing of the user skeleton this weekend.

Also, in order to maximize the portion of screen used so that users can see their poses from far away, I changed the formatting of the training screen, which now has the look shown by the image above.

Meanwhile, I also helped a bit with Jerry’s frontend pose sequence task where we both tried to understand how relative layout works.

I think we are still on schedule with our project and our application is in good shape. Still, we will probably need the thanksgiving break to fully integrate the system and test out the bugs. After finising the skeleton drawing algorithm, I will help Jerry work on the pose sequence functionality.

Posted on

Ray’s Status Report for 11/11/2023

I had made substantial progress on frontend this week. The first accomplished task is the dynamic insertion of widgets into the selection page is finally up and working. I designed a custom widget to hold both the previews and the names of the postures. Then all the widgets are added into the menu using python code instead of hard coding it with kv language. Below is the result:

Clicking the row that previews the pose will bring you to the training page and begin the training on that pose. This is, in my opinion, optimal in terms of both aesthetics and intuition.

The other thing I implemented is the mode page. Since we have both integrated postures of 24-form Tai Chi and custom postures from users, I created the mode page where the user can choose which set of postures they want to play with.

Clicking the upper block will bring the user to the same page as what the first image in this post shows. Clicking the lower block will bring the user to this page instead:

Here, the custom user poses imported through Jerry’s pipeline are displayed. Implementing this new page took a while since a lot of the page connection logic needs alterations.

I also started the frontend-backend integration with Eric and we have cooperated and agreed on variables and features that needs introduction into the frontend. This will extend into next week and will be my focus on the project.

I believe I’m well on schedule and our team would see completion of the MVP quite soon. Still, a lot of effort is still required for reaching that goal.

ABET:  On design requirements,  we need to make sure the system meets the minimum performance in detecting body part degree differences, the fps of the application, the accuracy of the verbal instructions, and the correctness of the user interface. Shiheng Wu and Hongzhe Cheng have conducted tests on the backend to make sure the first three design requirements are met. For the correctness of the user interface, Jerry Feng and I have checked the interconnection of the pages and made sure that the user are able to easily navigate through each of the pages, and  has ensured that when users uploaded images through the customization pipeline, they are displayed correctly in selection page and can be used as references in the training page. All tests for the subsystems went well.

We will redo all of the above tests on the full system with communication between the frontend and the backend correctly and robustly integrated. We will further conduct tests on user requirements, including testing the posture detection accuracy, verbal instruction latency, and accessibility of the system. In particular, we will use the systems ourselves and see if it meets our expectation for the system’s functionality and smoothness. We will also each invite three of our aquaintances to try the system out and receive feedback on all three user requirements from them. We can then improve the user experiences of our system based on our own evaluations and the testers’ suggestions.

Posted on

Shiheng’s Status Report for 11/11/2023

This week I continued developing the backend part of our software.
There were some back and forth between the group members on project details, but they were soon resolved and we were quickly back on track.

Following Monday’s interim demo session, I worked with Eric on hand posture detection as our demo did not detect them. We oversimplied the model and made the comparision inaccurate.
Fortunately it got resolved on the afternoon of the same day and we are now able to detect user’s hand posture. User will be reminded that their hand is not in the correct posture, e.g doing a fist instead of flat hand.

I also worked on more functionalities this week on the backend, including dividing and identifying body parts into ‘head’, ‘torso’, ‘arms’, ‘legs’, ‘feet’, communicating with Ray and Jerry about feedbacks needed on the frontend. The functionaility is mostly finished for normal picture (i.e user with full body included in frame with clear reference picture).

The plan for next week and following weeks is to implement the cases where unclear pictures are provided, or user is not posing their body inside the frame. I have been working closely with Eric’s test cases to identify potential issues from incoming rendered images and json files. Functions that need to implement are:
Calibration for one’s body and identifying their missing keypoints -> Prompt them to change their posture and include the full body in frame.
Multiple people in frame -> Use similiarity scoring to identify the actual user
Instruction wording -> Clear consise commands to user, and priortize lower body parts in training
Potential Integration issues -> Fix them as the project progresses
I also need to work with members on frontend about progress and implementation.

ABET question:
1. Unit testing on backend code: Lots of pre and postconditions are currently implemented for unit testing and debugging purposes
2. Image testing: Test on the go, using user inputs and reference images handpicked to test the functionality of the code
3. User feedback testing: Voice instructions will be played to user which will be evaluated on their quality and clarity
4. Edge case testing: Cases where improper images are uploaded? User not/partly in frame? User doing a completely different pose?

Posted on

Jerry’s Status Report for 11/11/2023

My deliverables this week changed from what I had planned for last week.  After meeting with my team it was decided that it would be better for me to work on a screen for renaming the custom images users uploaded.  Additionally, I learned from Eric that Openpose needs to be passed a directory of images instead of an image on its own, and so I reworked the image upload to the application as well.  A lot of the challenge in getting the image renaming to work is how awkward it is to pass information between screens in Kivy and the challenge of having screens and widgets be responsive to user input.

A simple demo with animal pictures can be seen below:

https://youtu.be/yKAvS6Dsj_U

 

I believe that in terms of meeting MVP, I am slightly behind schedule as I still have to finish some of the iteration process involved with giving users multiple training cycles per pose so they can correct their pose iteratively and will prevent us from throwing out a tons of verbal instructions and causing confusion for the users.  I also really wanted to get support for pose sequences to work and was unable to, but the team decided this was a stretch goal that was not to be included in the MVP so the multiple training cycles feature has taken priority.  To make up the lost ground, I will try to communicate more with Eric and Ray to be able to quickly understand how the existing code interacts with Openpose and be able to complete this feature in the next week.

The deliverables I hope to complete for next week is to get dynamic training cycles finished.  By dynamic training cycles, the vision is for the user to do a pose, and for the app to tell the user that it cannot to pose comparison if not all of the key body points of the user are in the frame, and then give the user 10 seconds to adjust their pose before repeating the process until it detects all of the user’s key points are in the frame.  When all of the user’s keypoints are in the frame we move on to the pose correction process and show the user the errors they have, give audio output for the highest priority error and then give the user 10 seconds to correct this error, then repeat the process until no errors are detected.

ABET Question:

We have not done many full system tests so far, and I have only performed unit testing on specific features.  However, for full system testing and validation, we plan to each recruit 5 different volunteers to test our system.  I am currently doing dynamic training cycles, so I plan to try and test that the system is able to accurately detect errors on a wide range of body types.  This means getting volunteers who have different body types than those of us on the team (i.e. not asian males).  I will see if users are able to adjust posture fast enough in the time we set for them in the dynamic training cycle loop through user feedback testing.  I will also use user feedback testing to see if the UI for file uploads is easy to use.  In the results for user feedback testing, if 3 or more users mention that there is not enough time between evaluation cycles to adjust their posture, I will lengthen the time necessary for the user to adjust their pose.  I also plan to do some logging of joint angles so if there are issues with the backend processing I can pass them to Shiheng and Eric so they can have more information to debug.

Posted on

Team Status Report for 11/11/2023

Following the interrim demo, our group continued to work on the frontend and backend of our system to further polish the functions and begin the process of integration.

The imminent problem mentioned during the interim demo about palm angles raised by professor Tamal was resolved in this week, as Shiheng and Eric worked through the openpose system and figured detecting keypoints on hands to determine hand postures.

We continued to discuss details for the future weeks of the project and made various important design decisions on certain details of our software.  We synced up our progress in Saturday meeting with Eric explaining integration steps of the software, Shiheng explaining the approach on the backend about pose comparison, Ray and Jerry working on the pipeline and communicating with the needs and ideas that backend needs to pass onto the frontend and UI.

Steps to get the application working:

  • Everybody work on main
  • Git clone from main branch into folderX
  • Copy everything from folderX into openpose root directory (C:/xxx/openpose/)
  • C:/xxx/openpose/ – Python3 main.py
  • Remember to recursively copy all the things, and remember to have bin/openposedemo.exe ready

Things to Do:

Front end:

    • Calibration (Integrated)
    • Pretty UI?
    • Sequence (Integrated in Training Iteration)
    • Skeleton Drawing – Ray (Discuss with Roland(shiheng))
      • Parse JSON to get coordinate/angle
        • Roland: JSON from openpose contains all coordinates of bodypose being captured, will just connecting the dots get an appropriate skeleton?  Normalization choice: Scaling the whole body by a factor (rough match)/tailoring limbs and body parts (exact match)?
      • Redraw the skeleton
      • Mark red for incorrect body parts
        • Roland: Pass back an error_list containing all the incorrect body parts above the threshold. No prioritizing body parts here, all incorrect body parts will be passed back.
    • Custom Upload renaming – Done by Jerry
  • Training Iteration – Jerry (Discuss with Eric for integration)
    • User tries to start training by clicking “Start” button in the training page
    • Camera get user image and send to backend (10s countdown)
    • User image processed by OpenPose to get user coordinates
    • Case 1: user coordinates contains every body part
      • Front end receives correct status code/return value
      • Display skeleton, verbal instructions is reading
      • ***If the user is correct, change to the next pose – Sequence working!!!
    • Case 2: user coordinates does not contain every body part
      • Front end receives bad return value
      • Display a flash warning saying your body is not in camera
      • Verbal speak “Change your position”
    • Iterate back to step 2
    • “Stop” button clicked to stop process or change pose…

Back end:

  • Error Priority – Roland
    • Feet->Leg/Thigh->Upper Body(Torso)->Arms, Instructions should be issued from this sequence (e.g Errors in Arms shouldn’t be mentioned before errors in lower body is fixed)
    • One instruction/error at a time to not confuse users learning Taichi
  • User Selection (Multiple people in frame chose the user in training) – Roland
    • Choosing the most similar person in frame (Similarity scoring)
    • Openpose input formatting for multiple people?
  • Instruction Wording – Roland
    • Dividing body parts into ‘head’, ‘torso’, ‘arms’, ‘legs’, ‘feet’, giving different instructions for corresponding body parts
  • Macro connection – Eric
    • Continue to integrate the frontend and backend, work with rest of the team

Later:

  • Verification & Performance Evaluation, Data collection for the final report (Second Last week)
  • User interview and UI test, report, slide, presentation. (Schedule for those events and start right after the integration is finished and functionalities achieved)
Posted on

Hongzhe’s Status Report on 11/11/2023

For this week, we were able to demonstrate our work in the interim demo to the staff member and show some of our progress. Given that we started late, I believe our team has made great efforts up until now. 

Personally, I achieved the following. Given the precious feedback from Professor Tamal, we found the bug where we were not capturing the angle for wrists. I was able to find the support for OpenPose on hand posture processing and update the backend model with Shiheng together to fix the issue. In order to do that, I also reprocessed all the reference dataset for future usage. At the same time, since OpenPose serves right in between the front and back end, I am now in charge of integration. I was able to integrate the real-time pipeline such that after hitting the capture button, the user image would be saved and the back end comparison procedure is called.

For the future weeks, I will keep integrating changes and parameters that go between the front and back end. Just to mention here that I will mostly work on integration on the real-time pipeline and Jerry would control his own.

ABET: Tests that I have created include Full back end functionality test and OpenPose functionality test. Basically the ones shown in interim demo.

Tests to be implemented particularly for my work would be the following (related to OpenPose and integration – full pipeline):

  1. Full pipeline functionality test: to test that the integration works fine and front and back ends are well connected
  2. OpenPose performance test: to see how much processing time is taken by OpenPose
  3. User Latency Test: to obtain the latency from the second data is captured from camera to when user first get verbal/visual feedback