Category: Ray’s Status Reports

The past week has been the final presentation week. I mainly focused on working with creating the slides with my team and filling out the final presentation reviews. I had a great time learning about all the useful and fun systems the other teams created.

Besides the final presentation, we had discussions among ourselves about the testing plan and real-world application of the application. We did some analysis on the data Jerry collected and checked how supportive the tests will be for showing the effectiveness of our system.

We also created the poster for next Monday’s public demo, which we believe will be a good opportunity to show case our application and collect some good user testing data. I managed some of the formatting and layout process for the poster.

Overall, we have completed all the functional implementations for the project. We are well on track finalizing our project, and for next week, I’ll work on the public demo and the final report, wrapping up the project as smoothly as possible.

 

My team and I made considerable progress on the project in the past two weeks. Except for some minor bugs, our application is functioning properly in general, and we are also actively optimizing our application to make it more convenient for our users.

To briefly summarize my work over the two weeks:

a. Training page posture sequencing works

Last Saturday and Sunday, I worked on the training page posture sequencing based on the implementation plan I created in advance. Since I finished the posture drawing functionality ahead of the plan, I implemented the posture sequencing functionality as well.

Now, for a Taichi posture sequence, the training page go through all subposetures in the sequence so that the users can actually train themselves on the full Taichi posture. When the user correctly performed a subpose, the training will automatically switch to the next subpose.

I also created a new configurable parameter “move-on time” for the users  in the setting screen to control the interval between subposes. It is different from the “preparation time” parameter, which is time interval between the moment start button is pressed and the evaluation of the first subpose.

Below is a image of the new trianing page (Image provided by Shiheng (Roland)).

b. Result page implemented

When the training of a Taichi posture ends, training screen now switches to a result screen, showing average score, total time, and a random tip on how to use our application correctly.

c. Tutorial page implemented

I created a new page showing a tutorial on how to use our application correctly. It is accessible from the menu page. Below is the tutorial page.

d. Minor fixes for better user experience

Selection screen has much larger pose item in order to make the image preview easier to view. Back buttons in all screens are enlarged. I also made a few small fixes on training screen timer display issues.

In general, my team and I am back on the schedule and we are promptly finalizing our project. Next week, I will fix the bugs and issues I found during testing and also brainstorm on what extra feature I want to add to our application.

This week, our team mainly worked on frontend/backend integration, and I took most of my time working on the skeleton draw functionality.

In order to make our system as intuitive as possible, we will have a screen showing direct comparison between the reference pose and the user pose on the training page of our application. Specifically, we want to draw the skeleton of the reference pose and then overlay the user pose on top of it, so that the user can intuitively see which of their limbs have incorrect angles.

I implemented the algorithm for drawing the reference skeletonthis week. It’s much harder than I expected since I need to conducts a couple of transforms on the data derived from openpose. The coordinate system of openpose is very different from Kivy’s, and I need to find a way to center the drawn skeleton. Also, to process the data outputed from the backend, I read through the entire backend code and pointed out a few places where the naming of the variables does not fully represent what they are. For now, the drawn reference skeleton looks good, and I will attempt the drawing of the user skeleton this weekend.

Also, in order to maximize the portion of screen used so that users can see their poses from far away, I changed the formatting of the training screen, which now has the look shown by the image above.

Meanwhile, I also helped a bit with Jerry’s frontend pose sequence task where we both tried to understand how relative layout works.

I think we are still on schedule with our project and our application is in good shape. Still, we will probably need the thanksgiving break to fully integrate the system and test out the bugs. After finising the skeleton drawing algorithm, I will help Jerry work on the pose sequence functionality.

I had made substantial progress on frontend this week. The first accomplished task is the dynamic insertion of widgets into the selection page is finally up and working. I designed a custom widget to hold both the previews and the names of the postures. Then all the widgets are added into the menu using python code instead of hard coding it with kv language. Below is the result:

Clicking the row that previews the pose will bring you to the training page and begin the training on that pose. This is, in my opinion, optimal in terms of both aesthetics and intuition.

The other thing I implemented is the mode page. Since we have both integrated postures of 24-form Tai Chi and custom postures from users, I created the mode page where the user can choose which set of postures they want to play with.

Clicking the upper block will bring the user to the same page as what the first image in this post shows. Clicking the lower block will bring the user to this page instead:

Here, the custom user poses imported through Jerry’s pipeline are displayed. Implementing this new page took a while since a lot of the page connection logic needs alterations.

I also started the frontend-backend integration with Eric and we have cooperated and agreed on variables and features that needs introduction into the frontend. This will extend into next week and will be my focus on the project.

I believe I’m well on schedule and our team would see completion of the MVP quite soon. Still, a lot of effort is still required for reaching that goal.

ABET:  On design requirements,  we need to make sure the system meets the minimum performance in detecting body part degree differences, the fps of the application, the accuracy of the verbal instructions, and the correctness of the user interface. Shiheng Wu and Hongzhe Cheng have conducted tests on the backend to make sure the first three design requirements are met. For the correctness of the user interface, Jerry Feng and I have checked the interconnection of the pages and made sure that the user are able to easily navigate through each of the pages, and  has ensured that when users uploaded images through the customization pipeline, they are displayed correctly in selection page and can be used as references in the training page. All tests for the subsystems went well.

We will redo all of the above tests on the full system with communication between the frontend and the backend correctly and robustly integrated. We will further conduct tests on user requirements, including testing the posture detection accuracy, verbal instruction latency, and accessibility of the system. In particular, we will use the systems ourselves and see if it meets our expectation for the system’s functionality and smoothness. We will also each invite three of our aquaintances to try the system out and receive feedback on all three user requirements from them. We can then improve the user experiences of our system based on our own evaluations and the testers’ suggestions.

This week I was preparing for the interim demo components for my part on the project.  In particular, I specifically worked on the training page to make sure the camera and the preparation time function for our application works.

The training page is now connected to the seleciton page and its camera is now working. Pressing the Capture button will start a countdown, and after the countdown, the footage of the user will be captured. Now the captured footage will be saved as an image file, but the actual pipeline we want to realize is to send the image to backend for comparisons. The connection with backend is my major task for next week.

Meanwhie, I managed to dynamically add widgets to the selection page. Now the main task is to create custom widgets so that each pose selection button has a preview of what it looks like. This will also be part of my work next week.

I also reached out to professionals for interviews on Taichine Training and searched for papers. This is to support our argument that the user’s feet postions should be used as the root for postures in comparison functions. I managed to get consent to interview for next week, and I also found a few relevant papers supporting our arguments.

Also, I set up a repository for our project to help our team integrate the full application. The UI code has worked successfully on all team members’ laptops, which is good news!

Overall, I am back on schedule and we can finally start our overall integration next week. I’m looking forward to it!

For the past week, I have been learning and writing KV language as well as adjusting the current prototype framework that we have. I changed a lot of widgets’ configurations so that they are now aligned according to the size of the application window, and I also successfully got the setting page to work.

My next important task is to create the training page, which our group comes to an agreement on the design. We now want to display the user’s footage on the top left corner, the reference posture on the top right corner, and the two skeletons at the bottom of the application window. The skeletons will be drawn with the same body part lengths, but the angles between different body parts will depend on the reference pose and the user’s attempt at imitating it. We believe this will help users know how to correct their mistakes the most intuitively.

Also, with help from Eric, Openpose is finally running on my laptop! While the framerate is very low, it should be enough to let me start testing out the openpose api in the official documents. I plan to try them out next week and see how things work for our overall integration.

Also, my teammates and I also looked into the ethics aspect of our project. We met and discussed this week to see if there are any extra design choices necessary to make our project ethically robust. It turns out the parameter adjustment option that we are offering to our users on the setting page has already resolved some of the most important ethical issues concerning our project.

I believe I am currently a little behind schedule since the integration process is expected to start next week, but I believe I can catch up early next week as I have clearly identified and prepared for the tasks left for the frontend part of our project.

For the past two weeks, my team and I worked on the design review of our project to finalize the application structure and implementation design. We worked on the design review document together and dicussed on how each subcomponent of our project should work. I worked specifically on finalizing the user interface section.

To accomplish the above task, I shifted my focus from tkinter to kivy, which is the new package we chose for implementing our application interface. I learned about the basics of initiating and configuring the widgets and wrote kv language files to simplify the design process. Using the skills I learned, I created the prototype pages for the main menu and the pose selection page. Below are the prototypes I created. (Consider this my answer to the ABET question.)

In order to let the application run as expected, I have to get used to kivy’s screen manager functionality to switch between different pages. I am currently working on it and I plan to work out a functional application frame next week. Also, the live camera embedded in the training page is also something I need to look into, and I will start working on it as soon as the previous task is mostly completed.

I am still trying to configure openpose on Windows; if necessary, I will ask Eric about the building process and make sure openpose run on my computer by next week.

I am overall on schedule for this week. Still, starting from next week, I have to put more attention on cooperating with my teammates since the integration process of our application should soon begin.

We start on our implementation process this week and everyone gets to work on their respective sections of work. I started learning and writing UI for our system with TKinter.

I got through many of the UI elements over the week and I plan to planning to approach the image display, video display, and database functionalities tomorrow. Below is the video I will refer to.

I also tried to get openpose to work on my laptop, but I am still getting errors based on CMake versions. My plan is to make Openpose run on my laptop as soon as possible, hopefully in the next 2 days.

I also looked through whole_body_from_image.py and keypoints_from_image.py examples in the openpose official repository. Base on the whole body example, display a sequence of poses in order can be realized by reading each image , setting up the pose datapoints in Shiheng’s cost functions, and then waiting for the user to input a correct pose. The timing could be an issue to implement and I might need to look into the Python functionalities to realize them in our system. Meanwhile, static pose evaluation can be realized with the following code as a reference:

Still, through discussions on the design review we presented this week, we noticed aspects of our project that requires more detailed considerations. In particular, the way we want to convey our pose dectection results need to be intuitive enough for our users. Also, the time interval between each verbal instruction should be different from the interval between each pose detection. I will take these into account when designing the user interface next week and reflect them in the design report that is coming up next week.

My progress for this week is still on schedule, though work for next week might be heavier than regular weeks. I plan to work some time during the fall break if the schedule for next week turns out to be too packed, but I will put priority on planning out how the api and ui should communicate and decide on a design for the ui.

This week, we had a new team member on our team. We had two three meetings together to discuss on labor division as well as technical details of the project and made solid progress on our implementation design. On my part, I researched on the API structure of Openpose and the packages for implementing our user interface.

The API of Openpose supports both Python and C++ for data transfer, and after discussing with my teammates (specifically Jerry), we decided on using Python since it has more functionalities available for user interface implementation. The most time consuming computation our system will do is in openpose, while the api is only responsible for reading outputs from it, so there should not be a huge sacrifice in runtime efficiency.

The API package that I plan to use is TKinter. Upon research, I find it having great potential as the tool for implementing our system. It supports graphics and also has extensive support on GUI elements. (Below is the tutorial I’m watching.)

I was taught briefly about how openpose work in Computer Graphics 15462 and had the chance to read about it in the past. However, I have not integrated the model into a full system before, so for the past week I read about openpose’s API usage on their Github page. Openpose official repository has some helpful tutorials for me to refer to. I read through some of them, but have not had the chance to try running them. I plan to install and build the openpose project on my laptop this weekend.

I am back on schedule this week, and my plan for next week is to get used to the TKinter package and the Openpose API to start creating a prototype UI for our system. Look forward to seeing how it will turn out!

 

This week, I presented our team’s original idea “Patient Monitor System” to the class. After the presentation, my teammates and I received feedback from instructors and, upon reconsideration, we decide to switch to the more applicable idea of designing a Taiji Training monitor system. To build upon this new idea, I searched for relevant papers and researches on pose recognition. In particular, I found and skimmed through the paper “3D Human Pose Estimation on Taiji Sequence” [1], finding it very relevant to our new project idea while targeting a different goal. In particular, the part where the conversion from 2D joints to 3D joints using neural network is intriguing and worth looking into.

My progress, along with our team’s schedule, is a bit behind due to our change in plan. To catch up on our original schedule, we will rework on our abstract to establish a reliable ground for our new idea for the first few days in the next week, and try to get the tasks originally planned for next week finished after we have a clear plan for the system we want to design.

In particular, I want to lay out how our systems will recognize and evaluate Taiji Poses. The Taiji Poses we plan to work with is the 24-form-Taiji sequence, there are clear ground-truth poses in this sequence for machine learning. We also have the idea of letting our user input the Taiji videos of their choices. We would also want to find a good way for the users to know how good their Taiji Poses are; some possible solutions to this might be an LED screen or a programmable LED unit. I also hope to make plenty of progress in our data collection next week, since data collection for our current idea is considerably simpler than for our previous idea.

[1]https://etda.libraries.psu.edu/files/final_submissions/17625