Posted on

Ray’s Status Report for 11/04/2023

This week I was preparing for the interim demo components for my part on the project.  In particular, I specifically worked on the training page to make sure the camera and the preparation time function for our application works.

The training page is now connected to the seleciton page and its camera is now working. Pressing the Capture button will start a countdown, and after the countdown, the footage of the user will be captured. Now the captured footage will be saved as an image file, but the actual pipeline we want to realize is to send the image to backend for comparisons. The connection with backend is my major task for next week.

Meanwhie, I managed to dynamically add widgets to the selection page. Now the main task is to create custom widgets so that each pose selection button has a preview of what it looks like. This will also be part of my work next week.

I also reached out to professionals for interviews on Taichine Training and searched for papers. This is to support our argument that the user’s feet postions should be used as the root for postures in comparison functions. I managed to get consent to interview for next week, and I also found a few relevant papers supporting our arguments.

Also, I set up a repository for our project to help our team integrate the full application. The UI code has worked successfully on all team members’ laptops, which is good news!

Overall, I am back on schedule and we can finally start our overall integration next week. I’m looking forward to it!

Posted on

Shiheng’ Status Report for 11/04/2023

I dedicated additional time to refining the comparison algorithm in our project, which focuses on capturing single image inputs with local reference posture. During our Friday meeting, the rest of the team conducted a code review of the algorithm and highlighted several areas that needed optimization. This was necessary because some parts of the code had originally been hardcoded for testing purposes, Hongzhe helped me pointed out some of the issues in my code. After addressing and enhancing the script based on our team members’ feedback, we managed to align it with most of the initial design goals. This included functionality for scoring, generating joint angles as outputs, and integrating our voice engine. I worked closely with Hongzhe this week in developing scripts and practiced some unit testing in different environments based on his output from the Openpose script.

 

As for the voice module implementation, we have not yet settled on a specific approach. I will continue collaborating with the rest of the team to determine how to incorporate the voice component into our program. I am currently attempting to write a separate script to call on and test generated voice files, but I would need to do some testing before integrating that part possibly on Sunday or during the next week. If the separate script could not be integrated in time, we will do a manual approach for the interim demo on this part and just showcase that the generated .wav files could be played.

Posted on

Jerry’s Status Report for 11/4/2023

This week I finished the file uploading for the frontend of my pipeline and finished integrating with the rest of Ray’s frontend.  A short demo on the full functionality can be seen here:

https://youtu.be/PB4mf4SlUR4

I also was able to get Openpose to build and work correctly on my laptop thanks to some help from Eric.  A photo can be seen below:

I spent the most time trying to enable support for the upload of image sequences to the app.  This was a lot more challenging to accomplish than I initially anticipated as Kivy’s relative positioning system has so far been very unintuitive to use.  It also seems that older versions of Kivy have different attributes for relative positioning so this confused me quiet a bit in my debugging.

I think with the difficulties I ran into this week, I am actually slightly behind schedule.  I will put more hours into this project next week, especially during the day of break on Democracy Day.  I will also communicate more with Ray as our discussions have been very helpful in increasing my understanding of Kivy’s positioning system.

 

Next week I plan to be able to try and get some basic setup for supporting pose sequences and being able to use python to call Openpose to process images and return JSON files.

Posted on

Hongzhe’s Status Report on 11/04/2023

In the past week, each team member is making substantial progress on building on some visible and usable components in the project and starting to integrate for the interim demo. At the same time, during the lecture on Monday, we learned a lot from the ethics lecture on what we should consider when being an engineer creating technology that makes not only technical impact. On Wednesday, we also got some valuable feedback from Professor Byron to check the status of what we would show on the interim demo.

Personally, in the past week, I accomplished the following things. First of all, utilizing the interface I created last week, I was able to process all the reference data we have on the default 24 Tai Chi Poses and generate the coordinate data. The data will be further used by Roland on testing comparison algorithms and for reference in the final product. Secondly, providing advice to my teammates, now everyone has OpenPose successfully built on their Windows PC. Last but not least, in preparing the interim demo, I was able to create a program that connects OpenPose and the comparison algorithm. The subunit test takes in a user image and reference pose name, processes the user image to get coordinates, compares the user and reference coordinates to generate text feedback. Note that this is the middle portion of the final product, we will further connect the real front-end app and camera interface and the text-to-speech app in the back to perform the functionality in the future. We might be able to integrate more functions tomorrow before the demo, but this is what I have been working with Roland to achieve.

In future weeks, given that the OpenPose infrastructure is mostly done, I will be focusing on integration, and some sub parts of app infrastructure such as skeleton drawing etc. We will also try to reach out to some interviewees to get some further confirmation on project details.

Posted on

Team Status Report for 11/4/2023

Our group is making progress according to schedule this week and preparing for the upcoming interim demo.

Ray and Jerry are collaborating on the Kivy (UI) design pipeline, there have been various implementation issues including environment switches (Linux to Windows) and learning the different methodology that Kivy provided compared to traditional Python. Kivy also provided various widgets that seemed easy to implement and apply to our program, but they turned out to be a lot more complex than we previously estimated. Fortunately, we collaborated during our usual meeting on Friday to debug and discuss elements of the UI design, which fixed most of the issues we previously had in development. Ray also worked on the file management part in creating a relatively primary file management system for us to store reference and user poses.

In addition to the work previously mentioned, Jerry has been making significant contributions to the development of his pipeline for single file uploads. His efforts have proven to be highly successful, as his code has been integrated into the main program developed by Ray. This integration marks a crucial step forward in the application’s functionality, allowing for the straightforward acceptance and processing of basic single images. With this foundation in place, Jerry is poised to continue his work on expanding the application’s capabilities. His next focus will be on further enhancing the application’s features by enabling it to accept and utilize pose sequences.

Eric (Hongzhe) continues to work on the Openpose system and its integration into our application. Eric is also learning Kivy along with Ray and Jerry to speed up the application development and integration of pose recognition. Continuing his progress from last week, Eric also did extensive testing on various Taichi pose pictures to make the skeleton from Openpose to overlay on the original picture. He is also working on file directory for output json files and with Shiheng for him to accept the inputs for comparison algorithm from our designated directory. Eric also helped with debugging throughout the programing both on Kivy and communicating demands with Shiheng’s comparison algorithm, which he would gladly provide postures for Shiheng to do unit testing for his algorithm.

Shiheng worked more on his own about the comparison algorithm for our design of capturing single image inputs. However, during the Friday meeting we held, the rest of the group did code review on the algorithm and pointed out various issues to be optimized, since some parts of the code were originally being hardcoded for testing purposes. After fixing and improving the script from team member’s advice, it could achieve most of the designs we planned initially, including scoring, producing joint angles as outputs, and invoking our voice engine. Since we have not decided how to implement the voice module into our program, Shiheng will continue to work with the rest of the group about playing generated voice instructions and polishing on the details will be the key of achieving a robust Taichi instructor tool.

Posted on

Shiheng’ Status Report for 10/28/2023

My work this week mainly focused on comparison algorithms. Using the json data generated from Openpose thanks to Eric, I was able to craft my python script for comparing two different postures. There was some confusion at the beginning since there were some extra outputs than I previously expected, but it was figured out after communicating with Eric and looking at the documents from Openpose. The keypoints were narrowed down and point representing positions like eyes and ears were eliminated to improve the accuracy of determining overall body posture.

Only point 0-14 are used in judging posture right now for efficiency, I’ll see how certain keypoints (on the feet) impact the correctness of posture in testings following the implementation.

Using packages like numpy and json, I was able to pull files from the locally designated folder and compare the sets of data within. I added a few steps to process the data including the mentioned reduced datapoints, but also reforming vectors from the given limb datapoints. The reformed ‘limbs’ were then passed into the cosine similarity comparison. The response of the algorithm was quick, and results were easy to understand. The way we wanted to display the results and the mechanism is still to be determined, so I will display a raw output below from my algorithm to showcase one example of comparison based on two trials of a same image but which I did some minor changes in the output position of the upper limb of the second image.

The next steps will be to connect the comparison algorithm with the TTS engine to invoke vocal instructions and move on the integration steps accordingly.

Posted on

Ray’s Status Report for 10/28/2023

For the past week, I have been learning and writing KV language as well as adjusting the current prototype framework that we have. I changed a lot of widgets’ configurations so that they are now aligned according to the size of the application window, and I also successfully got the setting page to work.

My next important task is to create the training page, which our group comes to an agreement on the design. We now want to display the user’s footage on the top left corner, the reference posture on the top right corner, and the two skeletons at the bottom of the application window. The skeletons will be drawn with the same body part lengths, but the angles between different body parts will depend on the reference pose and the user’s attempt at imitating it. We believe this will help users know how to correct their mistakes the most intuitively.

Also, with help from Eric, Openpose is finally running on my laptop! While the framerate is very low, it should be enough to let me start testing out the openpose api in the official documents. I plan to try them out next week and see how things work for our overall integration.

Also, my teammates and I also looked into the ethics aspect of our project. We met and discussed this week to see if there are any extra design choices necessary to make our project ethically robust. It turns out the parameter adjustment option that we are offering to our users on the setting page has already resolved some of the most important ethical issues concerning our project.

I believe I am currently a little behind schedule since the integration process is expected to start next week, but I believe I can catch up early next week as I have clearly identified and prepared for the tasks left for the frontend part of our project.

Posted on

Team Status Report for 10/28/2023

During the past week, our team has been diligently working on various aspects of our project and collaborating to work the magic.

 

Ray and Jerry are working on the UI design using Kivy about functions like image uploading and displays. They will continue to cooperate with other members of the team to integrate functions like Openpose and voice engine into the application. This will play a pivotal role in ensuring a user-friendly and visually appealing experience for our application compared to the more naïve implementation of Tkinter.

 

Eric has successfully integrated Openpose into our system, enabling it to accept image inputs and generate JSON outputs for the comparison algorithm for Shiheng. This is a pivotal step in our project, as it provides the foundation for our comparison algorithm and more detailed instructions on body postures. We’ll look at how we want to display Openpose feedback integrated with camera images to the user in the following weeks.

 

Shiheng has implemented a cosine similarity algorithm for comparing key points that represent body posture. This algorithm will allow us to quantify the similarity between different body postures, providing a strong basis for evaluation and giving out instructions. We can analyze and compare the alignment and positioning of individual limbs, offering a more detailed assessment of body posture.

 

We also discussed issues on ethics after we specified more about project details on our Friday meeting, during which we first exchanged opinions about the reading and answers we made on the project, then proceeded to argue more on privacy, autonomy, and potential issues of misusing the project.

 

Our team is making solid progress, and we’re on track to deliver our comprehensive Taichi instructor system. We’ll continue working on these components, work on feedbacks from our design review report, and aim to achieve our project milestones in the coming weeks.

Posted on

Jerry’s Status Report for 10/28/2023

This week, I have managed to make progress on the frontend of my pipeline using Kivy.  Kivy is a new library to me, that also comes with its own CSS-style language (KV lang).  Learning this new language and library and how it interacts with Python was fairly challenging as most of the file processing needs to be done in Python, while a lot of the frontend design is most conveniently done in (KV lang).  Ultimately, I was able to get a basic UI working that is able to pull up a file explorer,  and when a file is clicked by the user the path of that file will be printed out to the VS code terminal which I demonstrate in the video below.

Video Link: https://youtu.be/MLIPRchwNvM

In addition to my technical tasks, I also met with my team and discussed the ethics of our project and how we might make more ethical design choices for our project, the notes of which were sent in our team slack.

 

I believe I am well on schedule, as from here, saving files should not be too difficult, which will finish the functionality for the frontend part of the project.

For the next week, I will focus on getting openpose to build and finishing off the frontend functionality.  If I complete things ahead of schedule, I will also look into making the frontend look nicer, and then pushing my code to the github for integration for demo day.

 

Posted on

Hongzhe’s Status Report on 10/28/2023

In the past week, we as a team pushed the progress of the project to be fully detailed and laid out. Individually we were able to create more usable modules and codes as subparts for the final design. We also had some great discussion with the faculties to finalize and brainstorm some small design choices and learned how to conduct better background research. In the meantime, along with the course schedule, I learned more about thinking ethically as an engineer by reading papers and applying the thinking to our own project.

For the technical portion I own, I was able to create a function that takes a image input directory path, run OpenPose on all the images in the folder and output their corresponding JSON file containing the coordinates. This marks a milestone for the progress on OpenPose, as this function/interface should be able to serve as the interconnection between front end infrastructure and back end comparisons. After the camera is able to capture an image of the user, it would temporarily store it in some folder, then the path would be passed to this function and generate the coordinates, the coordinates would finally be passed to the comparison algorithm and feedback system. Note here that user image would only be temporarily stored in order to protect user’s privacy to the largest extent. The image would be deleted once the user moves on to further rounds of practicing. I also let the function generate a rendered image sample in case we need to use it in the UI as below.

For future days, given that the OpenPose infrastructure is mostly done, I will try to learn Kivy with Ray and Jerry and help on build some portion of the app infrastructure, majorly the portion that connects with OpenPose and about displaying coordinates and skeleton on screen.