Category: Team Status Report

The culmination of our project approached with the final presentation week, during which the team strategically allocated its efforts towards the meticulous creation of slides, thorough reviews of the presentation content, and collaborative endeavors. This period saw extensive discussions centering on crucial aspects such as testing plans, real-world applications, and in-depth data analysis, all with a primary focus on assessing the effectiveness of our developed system.

A significant facet of our recent efforts has been the intensive focus on comparative user testing, specifically aimed at measuring the accuracy of limb angles. This involved a dual approach, incorporating video training alongside our team’s system. As part of this process, ongoing data analysis is underway by Jerry and Ray, with the objective of extracting valuable information that will inform further refinement and optimization.

Our team’s productivity remained consistently high throughout the week, particularly in the preparation for the final presentation. Engaging discussions encompassed quantitative results derived from extensive testing, identification of potential areas of improvement, and careful consideration of user feedback. These discussions proved instrumental in shaping the final narrative of our project, ensuring a comprehensive and polished presentation.

Simultaneously, our focus extended to the backend processes of our project. Collaborative efforts, particularly with respect to finalizing backend functionalities, were undertaken in close partnership with Shiheng and Hongzhe. This phase involved meticulous optimization measures, enhancing overall system efficiency and responsiveness.

Noteworthy attention was also given to the refinement of design elements, with a dedicated effort to ensure optimal user experience. Collaborative discussions, particularly with Hongzhe, aimed at fine-tuning these elements, addressing any remaining concerns, and ensuring a seamless integration of design and functionality.

As the team progresses, we find ourselves currently on schedule, actively engaged in the critical phases of testing and evaluation. The imminent completion of the video and final report stands as a testament to our commitment to delivering key project deliverables with the highest standards of quality and precision. The collective efforts of the team position us well for the successful conclusion of this project.

Our team also wishes to express gratitude to all the faculty and TAs of the course for their continued assistance and guidance throughout the project.
Special thanks to our team’s TA Eshita and Professor Byron for meeting with us weekly, checking our progress, and providing constructive feedback throughout the semester.
ABET Question:

Unit Test on Frontend:
Functionality Testing to ensure the custom file upload feature functions as intended. Users are able to upload custom pictures. Widget Order Testing to verify that widgets associated with file upload are ordered correctly, user are able to rearrange and remove pictures.
UI Testing: UI elements and widgets work correctly under full screen resolution, and scales correctly.
Training Loop Testing: The training loop screen correctly showcased user image, skeleton, and reference picture.
Skeleton Drawing Testing: Correct skeletons are posed for both user and reference posture, previous issue mentioned in last week about scaling have been fixed.
File Storage: Local file storage is working correctly, both for prerecorded poses and reference poses.

Unit Test on Backend:
Angle testing: Justified angles captured by the Openpose system through the keypoints being passed into the backend. Used online protractor to measure angles compared to the calculation derived from keypoints.
Voice testing: Ensure voice module is robust and produces correct instructions as intended. Integrated throughout the testing process and accepted advice from users to improve instruction. Clear verbal instructions given with correct angles.
User posture testing: Different poses are being passed into the algorithm for testing, including incomplete postures, wrong postures. Correct angles and instructions are verified from feedback of group members and volunteers. Incorrect and correct limbs were clearly identified and passed to the frontend for drawing.
Correct person testing: When multiple people are involved in one frame, the algorithm is correctly identifying the user practicing the posture (most similar person).

Integration testing:
Testing on windows laptops, applications constructed by Kivy is able to launch and run correctly.
Pose sequencing: Functionality is working as intended, user is able to move on from one pose after doing it correctly. After finishing the sequencing, user will be presented with overall score of the sequence.
Parameter Adjustment Testing: Users are able to customize preparation time, move-on time, and tolerance angle for custom training. The sliders communicate with backend correctly to reflect the changes.
Time Trials: Timer are utilized to measure performance on different laptop setups from group members. User testing has come back with results of the app being slow, main bottleneck is the Openpose utilizing CUDA. Time requirements are within our set standards.
User Testing: Jerry collected data and did analysis on this part mainly. User experiences were measured, and bugs were found about the training loop, fixes were implemented to address the issue. Efficiency for the application increased after implementing the fix.

For our Taichine project this week, we finished all planned functionalities in our MVP and were mostly focused on polishing the final details of the projection prior to the testing and verification phase.

During Monday and Wednesday meetings, the team focused on testing on the front end along with demos of the project to faculty members. In the process, bugs were found throughout the test when group members were trying out our final product.

Shiheng and Ray worked on the backend and integration scripts about cases where users were not correctly positioned in frame and when nobody is detected in the camera. Previous solutions were not comprehensive to users and more for the purpose of debugging, which now have been replaced by voice instructions and redrawing of the skeleton to guide users to improve their current posture. Shiheng also worked on implementing voice instructions and fixed failures where the pygame module will cause unintended termination of the training. Ray improved UI elements on the frontend and changed scaling algorithms for better showcase of skeletons.

Hongzhe (Eric) worked on cropping and regeneration of the openpose data on the Taichi pose pictures to accommodate the frontend changes of picture frame, during which he also helped figuring out the bugs encountered on the frontend on skeleton drawing for the reference pose and user input. Eric also suggested that logic improvements on the backend to Shiheng about pose priority, where fixes were implemented upon.

Jerry worked more on the custom pose implementation side and focused on file storage structures for sequencing and stepping through the images for prerecorded poses. Now both pipelines follow similar naming trends and users are more easily directed to figure out where the custom poses are stored. Also, some button logics were revised on the frontend by Jerry to improve performance of the application and remove redundancy.

We will focus on testing, verification, and validation of our project for the weekend and for the week prior to final demo. In addition, we will work simultaneously on other scheduled matters including slides, posters, and videos to accommodate for more flexibility during the final week.

As we are approaching the end of semester, our team worked on integration and finetuning the infrastructure for the upcoming testing and verfication.

Jerry has been focused on enhancing the user experience by introducing custom pose sequence support. He’s also succeeded in integrating the app with file manager to handle multiple files, providing users with an efficient way to manage their custom pose uploads. Additionally, Jerry has improved user interaction by enabling the removal and reordering of files to be uploaded, directly within the app’s display.

Shiheng has taken on the challenge of allowing pose selection from multiple persons, a crucial feature for users within more complex environments and handling different outputs when multiple bodies are found in frame. In addition, he is concentrating on backend support for skeleton drawing for Ray, implementing a new and different approach for measuring angles that could be handled better on the frontend without the need for processing the raw data from Openpose.

Hongzhe (Eric) is dedicated to refining the visual aspects of the application. His work involves processing reference images to enhance their display quality. He is also focusing on establishing parameter connections, which will contribute to a more cohesive and user-friendly experience. Additionally, Hongzhe is also involved in fine-tuning the logic on the training page to ensure an optimized user journey.

Ray is playing a pivotal role in consolidating the efforts of the team. He is synthesizing key points and reference angles provided by Shiheng into the overall skeleton drawing process. Ray is also leading the redesign of UI components for improved aesthetics and usability, working on cleaning up the code and optimization. Furthermore, he is involved in refactoring reference pictures to enhance user experience with bigger picture components for better feedback experience.

ABET Question (How we have grown as a team?):

Throughout the semester, our team has experienced substantial growth and cohesion. We have established a robust framework for collaboration and decision-making, utilizing Google Drive for weekly reports, design documents, Github repo for codes, and Slack channel for communication and collaboration efforts.

Our mandatory lab sessions on Mondays and Wednesdays have provided consistent opportunities for team members to discuss progress and align our efforts. Additionally, our Friday or Saturday sessions over zoom have been dedicated to refining plans and syncing up progress during the week.

The use of Gantt Charts has proven instrumental in visualizing our project timeline and ensuring that tasks are allocated efficiently. This strategic planning has allowed us to stay on track and adapt to unforeseen challenges effectively. By regularly revisiting and adjusting our goals based on progress and feedback, we have fostered a dynamic and adaptive team culture.

Our commitment to an open and inclusive environment has been foundational to our success. We actively seek input from all team members, ensuring that each voice is heard and valued. Though we faced changes in group composition and change of plan in the beginning of the semester, everyone in the group was equally treated.

Thanks to everyone’s effort in creating an equal and exclusive environment, we have been able to make substantial progress from scratch as a group and advance through the hardships we faced throughout the semester.

Following the interrim demo, our group continued to work on the frontend and backend of our system to further polish the functions and begin the process of integration.

The imminent problem mentioned during the interim demo about palm angles raised by professor Tamal was resolved in this week, as Shiheng and Eric worked through the openpose system and figured detecting keypoints on hands to determine hand postures.

We continued to discuss details for the future weeks of the project and made various important design decisions on certain details of our software.  We synced up our progress in Saturday meeting with Eric explaining integration steps of the software, Shiheng explaining the approach on the backend about pose comparison, Ray and Jerry working on the pipeline and communicating with the needs and ideas that backend needs to pass onto the frontend and UI.

Steps to get the application working:

• Everybody work on main
• Git clone from main branch into folderX
• Copy everything from folderX into openpose root directory (C:/xxx/openpose/)
• C:/xxx/openpose/ – Python3 main.py
• Remember to recursively copy all the things, and remember to have bin/openposedemo.exe ready

Things to Do:

Front end:

• • Calibration (Integrated)
  • Pretty UI?
  • Sequence (Integrated in Training Iteration)
  • Skeleton Drawing – Ray (Discuss with Roland(shiheng))
    • Parse JSON to get coordinate/angle
      • Roland: JSON from openpose contains all coordinates of bodypose being captured, will just connecting the dots get an appropriate skeleton?  Normalization choice: Scaling the whole body by a factor (rough match)/tailoring limbs and body parts (exact match)?
    • Redraw the skeleton
    • Mark red for incorrect body parts
      • Roland: Pass back an error_list containing all the incorrect body parts above the threshold. No prioritizing body parts here, all incorrect body parts will be passed back.
  • Custom Upload renaming – Done by Jerry

• Training Iteration – Jerry (Discuss with Eric for integration)
• • User tries to start training by clicking “Start” button in the training page
  • Camera get user image and send to backend (10s countdown)
  • User image processed by OpenPose to get user coordinates
  • Case 1: user coordinates contains every body part
    • Front end receives correct status code/return value
    • Display skeleton, verbal instructions is reading
    • ***If the user is correct, change to the next pose – Sequence working!!!
  • Case 2: user coordinates does not contain every body part
    • Front end receives bad return value
    • Display a flash warning saying your body is not in camera
    • Verbal speak “Change your position”
  • Iterate back to step 2
  • “Stop” button clicked to stop process or change pose…

Back end:

• Error Priority – Roland
  • Feet->Leg/Thigh->Upper Body(Torso)->Arms, Instructions should be issued from this sequence (e.g Errors in Arms shouldn’t be mentioned before errors in lower body is fixed)
  • One instruction/error at a time to not confuse users learning Taichi
• User Selection (Multiple people in frame chose the user in training) – Roland
  • Choosing the most similar person in frame (Similarity scoring)
  • Openpose input formatting for multiple people?
• Instruction Wording – Roland
  • Dividing body parts into ‘head’, ‘torso’, ‘arms’, ‘legs’, ‘feet’, giving different instructions for corresponding body parts
• Macro connection – Eric
  • Continue to integrate the frontend and backend, work with rest of the team

Later:

• Verification & Performance Evaluation, Data collection for the final report (Second Last week)
• User interview and UI test, report, slide, presentation. (Schedule for those events and start right after the integration is finished and functionalities achieved)

Our group is making progress according to schedule this week and preparing for the upcoming interim demo.

Ray and Jerry are collaborating on the Kivy (UI) design pipeline, there have been various implementation issues including environment switches (Linux to Windows) and learning the different methodology that Kivy provided compared to traditional Python. Kivy also provided various widgets that seemed easy to implement and apply to our program, but they turned out to be a lot more complex than we previously estimated. Fortunately, we collaborated during our usual meeting on Friday to debug and discuss elements of the UI design, which fixed most of the issues we previously had in development. Ray also worked on the file management part in creating a relatively primary file management system for us to store reference and user poses.

In addition to the work previously mentioned, Jerry has been making significant contributions to the development of his pipeline for single file uploads. His efforts have proven to be highly successful, as his code has been integrated into the main program developed by Ray. This integration marks a crucial step forward in the application’s functionality, allowing for the straightforward acceptance and processing of basic single images. With this foundation in place, Jerry is poised to continue his work on expanding the application’s capabilities. His next focus will be on further enhancing the application’s features by enabling it to accept and utilize pose sequences.

Eric (Hongzhe) continues to work on the Openpose system and its integration into our application. Eric is also learning Kivy along with Ray and Jerry to speed up the application development and integration of pose recognition. Continuing his progress from last week, Eric also did extensive testing on various Taichi pose pictures to make the skeleton from Openpose to overlay on the original picture. He is also working on file directory for output json files and with Shiheng for him to accept the inputs for comparison algorithm from our designated directory. Eric also helped with debugging throughout the programing both on Kivy and communicating demands with Shiheng’s comparison algorithm, which he would gladly provide postures for Shiheng to do unit testing for his algorithm.

Shiheng worked more on his own about the comparison algorithm for our design of capturing single image inputs. However, during the Friday meeting we held, the rest of the group did code review on the algorithm and pointed out various issues to be optimized, since some parts of the code were originally being hardcoded for testing purposes. After fixing and improving the script from team member’s advice, it could achieve most of the designs we planned initially, including scoring, producing joint angles as outputs, and invoking our voice engine. Since we have not decided how to implement the voice module into our program, Shiheng will continue to work with the rest of the group about playing generated voice instructions and polishing on the details will be the key of achieving a robust Taichi instructor tool.

During the past week, our team has been diligently working on various aspects of our project and collaborating to work the magic.

 

Ray and Jerry are working on the UI design using Kivy about functions like image uploading and displays. They will continue to cooperate with other members of the team to integrate functions like Openpose and voice engine into the application. This will play a pivotal role in ensuring a user-friendly and visually appealing experience for our application compared to the more naïve implementation of Tkinter.

 

Eric has successfully integrated Openpose into our system, enabling it to accept image inputs and generate JSON outputs for the comparison algorithm for Shiheng. This is a pivotal step in our project, as it provides the foundation for our comparison algorithm and more detailed instructions on body postures. We’ll look at how we want to display Openpose feedback integrated with camera images to the user in the following weeks.

 

Shiheng has implemented a cosine similarity algorithm for comparing key points that represent body posture. This algorithm will allow us to quantify the similarity between different body postures, providing a strong basis for evaluation and giving out instructions. We can analyze and compare the alignment and positioning of individual limbs, offering a more detailed assessment of body posture.

 

We also discussed issues on ethics after we specified more about project details on our Friday meeting, during which we first exchanged opinions about the reading and answers we made on the project, then proceeded to argue more on privacy, autonomy, and potential issues of misusing the project.

 

Our team is making solid progress, and we’re on track to deliver our comprehensive Taichi instructor system. We’ll continue working on these components, work on feedbacks from our design review report, and aim to achieve our project milestones in the coming weeks.

For the week prior to the fall break, our group mainly focused on finishing and polishing the Design Review document. We split up the work evenly, with every team member taking up some part of the writing and filling in the part of the subsystem they are responsible for in the following developments. We were able to find existing journals and research to justify our design requirements, provide details for our system specs, and think of alternative approaches if some parts of the system would go wrong. The writing of the design documentation aided us to explain a lot more details not mentioned in our design review presentation due to time constraints (since we need to fit contents of two presentation into one as we redesigned our project after the proposal presentation), providing faculty a better picture of the project and helped clarifying details among group members.

While enjoying our fall break, we also made some progress in implementing some of the subsystems.

Thanks to Hongzhe (Eric), the Openpose posture detection is now working with imported images (handpicked high resolution from online video source) which we are going to use as reference postures and baseline in evaluating user poses. As we described inside our design documents, these images will be transformed into json files with key points of postures in 2D that will be passed into the comparison algorithm for calculation of the differences.

Shiheng worked on the Text-to-Speech engine, where it could take in text instructions generated from a comparison algorithm and pass them into the voice engine to generate real time instructions. The time required for the voice engine to generate output files is low (< 0.5 seconds for an average length instruction) and the output voices are clear and easy to understand. We will continue to look into voice engine outputs and determine the best way to present these vocal instructions to users.

On the frontend, Jerry looked into the project considering several Python packages for better UI and user considerations. He determined that the Kivy package will provide us with lots of widgets that could be easily implemented and a much better interface than Tkinter which was what we planned to use originally.

Ray focused on learning the kivy language and creating the prototype frontend application. He created a prototype page for the main menu and the pose selection page. He is also working on the screen manager feature of Kivy to support page switch in the application and expeirmenting on openpose data interfacing with Kivy UI.

Week 10/7/2023: Continuing to revise design and initial project development.

On Wednesday, we finished our design review presentation, which we continued our discussion on our design during our Thursday night meeting. We were joined by professor Tamal who gave us valuable feedback on our current project and how we could improve our MVP framework to include more functionality. One of the things we adapted was the idea of evaluating users’ performance over a short period of time to understand the fact that they need to setup and transit into the position, eliminating the chances that they are doing a completely different posture but accidentally got a higher score due to the frames grabbed by our system, and provide an internal evaluation/scoring system for body postures. We still need to polish these ideas before next Friday, when the design report is due, before which we would need to quantify these thresholds and reflect the changes in our report.

On Friday morning, we received detailed feedback thanks to hardworking faculties on our design presentation. We believed that a detailed presentation was given but it seemed we still needed to clarify some issues that were not showcased clearly within our presentation.

The team worked throughout the week to perfect the details and narrowed down how we wanted to approach this project, which we determined that minor changes need to be made to our schedule. We have begun coding and incorporating systems gradually into our project, for which we already have Openpose algorithms working ahead of time thanks to the extra effort Hongzhe (Eric) put in to configure and run the system.

We’ll continue to work with faculty members in the following week on our design report and start data collection through imported poses we handpicked online from Taichi professionals, integrating Openpose API, and start developing the comparison algorithm. We currently have no plans to order any hardware equipment, and please stay tuned for our next update and design documentation.

 

ABET question:

1.       Engineering Ethics: We highly respect the privacy of users due to the necessity of using cameras to capture body posture. It is possible that the camera could collect more personal info than posture, so we planned to make this a local application without the need for exchanging information with a cloud server. All collected data will be stored and evaluated locally without the need for connection to the internet, and users can choose to close the camera during the usage of the application if they feel necessary. (e.g. reviewing posture, attending to other businesses, or feeling not to practice)
2.       Math model in comparing body postures: Using the idea of cosine similarity, we attempt to compare body postures captured from users to our standard preset provided by Taichi professionals. To account for differences in heights and shapes of people, we directly measure the joint angles of a person instead of directly approximating how similar the person is due to the differences of absolute positions of joints. Normalization principle could also be applied to postures to account for magnitude differences in vectors for varied body sizes.
3.   Machine Learning model in OpenPose: In the posture recognition application OpenPose, we are able to use the trained convolutional neural network for recognizing the core body coordinates from the input video/image. Convolutional neural networks are based on simple batch processing, dot product of matrices and many techniques such as pooling and regularization to avoid outliers and overfitting. It utilizes all kinds of principles and formulas from Mathematics as logarithmic calculation to enable convolution and activation function. It also uses differentiation calculus for training the network parameters when back propagation.

Week of 09/30/2023: A week of Research and Findings

We have a new team member Jerry Feng joining us following our new proposal being approved on Monday. Doing research works on different parts of the project and reintroducing the project to group members were the main focuses of the week.

During Monday and Wednesday meetups, Shiheng, Ray and Eric (Hongzhe), made a proper introduction to Jerry about our Taichi project, discussed its background, and explained why we chose this as our project. In addition, Ray and Eric discussed with Jerry in depth about the openpose algorithm  and existing pipeline in our original plan, while Shiheng focused more on the comparison algorithm and how voiceover should be implemented in parallel to the development of the posture pipelines. While meeting up with Professor Bryon and Eshita, we brainstormed about how Jerry could integrate into our existing framework and decided that developing an alternative pipeline allowing customization would be best in Jerry and the rest of the team’s interest in working parallelly.

After transitioning smoothly into our new workflow, we spent most of our time researching our own parts of the project. We discovered various compatibility issues when trying to set up environments on our own laptops, but fortunately they were all resolved by the end of the week through our discussions and research online, which we all had a good understanding of how to implement the project. Everyone on the team has done decent research on narrowing down the ideal programming language, packages, and algorithm, which could be justified from various aspects including but not limited to compatibility, offline support, efficiency, and ease to use.

For the rest of the week, we spend the majority working on design review slides and replanning out our project quantitatively with the new pipeline Jerry owned. We refined our proposal slides to include more quantitative values to measure our performance, specified measures dealing with pipeline failures, and brainstormed various test cases for future verification purposes. Additionally, we worked on creating a new Gantt chart to include current works and reorganizing the work as Ray and Jerry now have some overlapping they could collaborate on during the semester.

Gantt Chart is Attached Below to show our progress:

Our team experienced drastic changes this week as we changed our project theme completely from hospital setting in to the traditional Chinese Taiji posture detection and instructor. After the discussion with course staff in the past few days, we recognized some usability and cost issues not apparent to us initially. Such discrepancies cost us substantial time and effort to justify our work. Thus, by the end of the week, we decided to switch gears and move on to the current topic–Taiji Pose Recognition Project: Taichine.

We adopted the idea of Taiji instructor since it is not a major change from our original idea. Most hardware and software usage will stay the same, but the overall setting of Taiji is much more manageable regarding a semester-long project. We still plan to use the camera-RasPi-backend setup as before, but the hardware will have enough capabilities to detect, classify, and evaluate body posture in real-time since Taiji pose recognition has higher tolerance in recognition efficiency. There will also be more online resources available to us while fewer privacy concerns should arise. We will spend less time monitoring personal data and focus on certain actions/routines so that we can offer clear evaluations in short periods of time.

It has been a week of transitions and chaos, but we successfully identified valuable online resources on Taiji postures and its recognition, which we could apply as training sets into our OpenPose algorithm. For the next week, we will focus on fixiating our hardware/software quantifications and formulate an optimal design for our system. If time allows, we plan to start data collection for our machine learning process. We will make necessary changes to our schedule and make sure we do not fall behind in the upcoming weeks, when working on Design review and training our models.