Category: Jerry’s Status Reports

This week I mostly focused on comparative user testing.  In our final demo, the professors expressed the desire to see user testing that measures the difference in accuracy of limb angles to the reference pose, with training with a video, and training with our system.

Users would test using a side by side of the video Ray gathered the reference poses from, and a webcam on the side to mimic a mirror, so they could look at their own pose and adjust.  I specified the users to train on the pose sequence of “repulse the monkey” in the video.  I had them train with the video for 5 minutes and then took a picture of the pose they felt they did the best at.  I would then have the users train on our system with a tolerance of 5 degrees for 5 minutes and use the latest training image and keypoints coordinates json file the system took for evaluation.

I fed the picture of the pose the users did with the video into openpose to get a json file of the coordinates, and then fed the json file into the comparison algorithm with the reference pose’s json file to get a list of similarities.  I did a similar thing with the json file of the users’ keypoints when training with our system.  I am still doing data analysis on the files and as such do not have an update yet to share.  We found that the users did better with the video than with our system when looking at an average of all of our similarities, however when we look at the similarities split up by joint angle type, we see that this is due to one main outlier: the angle between the right calf/shin and the right foot.  There are other joint angles that show large differences in favor of the similarity measurements of the user training without our app, but they are not nearly as dramatic as the angle between the right calf/shin and the right foot.

The full plot of the similarity differences can be found below.  The y axis is the similarities of the users joint angles training without our app minus the similarities of the users joint angles training with our app.  Thus a negative value, means the users showed better similarity to the reference pose when training without our app and a positive value means the users showed better similarity to the reference pose when training with our app.

Link to graph: https://docs.google.com/presentation/d/1GJe1FTYBJIslWOp5YyHfYAI1yyZRDDB5vdt_yO-MKDY/edit?usp=sharing 

I also helped make the poster and final presentation this week as well, including making a short gif of the custom upload pipeline in action.

We are well on schedule as we are doing testing right now and evaluating our system.  Nothing remains to be done with our project in terms of functionality.

In terms of deliverables we plan to finish making the video and writing the final report to turn in on time.

ABET Question 6:

In terms of tests we did, I did numerous unit tests on my custom file upload in terms of testing the functionality of the system and making sure widgets were getting ordered correctly, and that the custom upload feature was robust under continuous use.  In terms of full system testing, our team did many full system tests during the integration phase of our project, and user testing and comparative user testing also doubled as a full system test.  In terms of our tests, we found that users overall felt the app was a bit too slow, I noticed one user who mentioned it was very slow, ran into a bug in our system where openpose was continuously running in the training loop, even when the user exited the training screen.  I fixed this bug and our user scores for the speed of our app went up.

 

 

This week I took the lead on user testing for our app.  I created a google form to test for user’s satisfaction with various aspects of our app such as speed and usefulness of voice generated instructions.  In total I had 5 users try out our application.  I had users try the first pose, and had users select the “repulse the monkey” pose.  I had the first user test on an angle tolerance of 4, 10, and 15.  For the other 4 users I had them test on an angle tolerance of 15 and 20 degrees.  After the first user tested, I found a bug where the training loop would continue running even after leaving the training page.  I fixed this bug before testing with the other users, so the first user’s review is biased due to it being on a slower version of the system.  Some commonalities from the feedback we were given is that users overwhelmingly preferred an angle tolerance of 20 degrees, they all found the mirroring of the camera unintuitive, but our app on the whole was fairly usable.   Below is a picture of the responses we received to 2 of the questions in our google form:

I also helped to complete the final presentation slides with my teammates so we can be ready for our final presentation next week.

 

I believe I am on schedule as we have completed some of the verification and user testing needed for our project that we wanted to complete this week.

In terms of deliverables next week, I plan on continuing user testing as well as working on the design report and poster.  I also anticipate working on some of the issues mentioned in our user testing if my partners need help completing some of the feature improvements users would like to see.

This week I am excited to announce that I have finished putting together pose sequence support for the custom images upload page.  I was able to have dynamic image removal, and reordering from the user.  The user is also able to name the pose they would like to use, with the default name of the pose being the name of the first image in the sequence.  Users upload sequences by selecting multiple files from the computers native file manager that displays as a popup.  Full functionality can be seen in the video below:

https://youtu.be/ZXbXMUYbDSE  (In the video I accidentally say custom image upload support, I meant custom pose sequence upload support.)

I believe I am right on track with an adjusted schedule that my team has put together, where we wanted to get custom pose sequence support finished before Thanksgiving.  Since the meat of custom pose sequence support (functionality of dynamic changes to poses in upload phase and upload in correct ordering) is done, I believe I am well on track with our adjusted schedule.

In terms of deliverables, I hope to be able to finish tweaking the button sizes so that the buttons do not overlap with images when only 1 image is being uploaded.

ABET Question:

Many of the ways we have grown as a team is that as we have worked together more, our communication as a team has gotten better and better.  Originally, before I joined, the team communicated predominately through WeChat but they have been very gracious in switching to Slack to better accommodate me.  We have also gotten better at adjusting to using Slacks functionalities to organize questions we ask each other to coordinate the integration of our project.  We have also gotten better at actively maintaining an open atmosphere in our team meetings so everyone can voice their opinions and have a say in the direction we want to take our project as initially I think we were a little reticent to voice our opinions, but now everyone is comfortable sharing their opinions and discussing differences in view points to ensure a better project.

 

My deliverables this week changed from what I had planned for last week.  After meeting with my team it was decided that it would be better for me to work on a screen for renaming the custom images users uploaded.  Additionally, I learned from Eric that Openpose needs to be passed a directory of images instead of an image on its own, and so I reworked the image upload to the application as well.  A lot of the challenge in getting the image renaming to work is how awkward it is to pass information between screens in Kivy and the challenge of having screens and widgets be responsive to user input.

A simple demo with animal pictures can be seen below:

https://youtu.be/yKAvS6Dsj_U

 

I believe that in terms of meeting MVP, I am slightly behind schedule as I still have to finish some of the iteration process involved with giving users multiple training cycles per pose so they can correct their pose iteratively and will prevent us from throwing out a tons of verbal instructions and causing confusion for the users.  I also really wanted to get support for pose sequences to work and was unable to, but the team decided this was a stretch goal that was not to be included in the MVP so the multiple training cycles feature has taken priority.  To make up the lost ground, I will try to communicate more with Eric and Ray to be able to quickly understand how the existing code interacts with Openpose and be able to complete this feature in the next week.

The deliverables I hope to complete for next week is to get dynamic training cycles finished.  By dynamic training cycles, the vision is for the user to do a pose, and for the app to tell the user that it cannot to pose comparison if not all of the key body points of the user are in the frame, and then give the user 10 seconds to adjust their pose before repeating the process until it detects all of the user’s key points are in the frame.  When all of the user’s keypoints are in the frame we move on to the pose correction process and show the user the errors they have, give audio output for the highest priority error and then give the user 10 seconds to correct this error, then repeat the process until no errors are detected.

ABET Question:

We have not done many full system tests so far, and I have only performed unit testing on specific features.  However, for full system testing and validation, we plan to each recruit 5 different volunteers to test our system.  I am currently doing dynamic training cycles, so I plan to try and test that the system is able to accurately detect errors on a wide range of body types.  This means getting volunteers who have different body types than those of us on the team (i.e. not asian males).  I will see if users are able to adjust posture fast enough in the time we set for them in the dynamic training cycle loop through user feedback testing.  I will also use user feedback testing to see if the UI for file uploads is easy to use.  In the results for user feedback testing, if 3 or more users mention that there is not enough time between evaluation cycles to adjust their posture, I will lengthen the time necessary for the user to adjust their pose.  I also plan to do some logging of joint angles so if there are issues with the backend processing I can pass them to Shiheng and Eric so they can have more information to debug.

This week, I have managed to make progress on the frontend of my pipeline using Kivy.  Kivy is a new library to me, that also comes with its own CSS-style language (KV lang).  Learning this new language and library and how it interacts with Python was fairly challenging as most of the file processing needs to be done in Python, while a lot of the frontend design is most conveniently done in (KV lang).  Ultimately, I was able to get a basic UI working that is able to pull up a file explorer,  and when a file is clicked by the user the path of that file will be printed out to the VS code terminal which I demonstrate in the video below.

Video Link: https://youtu.be/MLIPRchwNvM

In addition to my technical tasks, I also met with my team and discussed the ethics of our project and how we might make more ethical design choices for our project, the notes of which were sent in our team slack.

 

I believe I am well on schedule, as from here, saving files should not be too difficult, which will finish the functionality for the frontend part of the project.

For the next week, I will focus on getting openpose to build and finishing off the frontend functionality.  If I complete things ahead of schedule, I will also look into making the frontend look nicer, and then pushing my code to the github for integration for demo day.

 

Over the past two weeks, I contributed to writing the “Introduction” and “Use-Case Requirements” sections of the design report, as well as  my custom pose/pose sequence pipeline.  I did some designing on what I wanted the UI to look like and a basic flowchart for how I want the image upload to proceed, and added the pictures to the design report.

I also looked into researching academic papers for how accurate we can expect our joint angle based system to be for pose detection and whether it would translate well to different body types.  I did not find definitive evidence that joint angle based pose detection would translate well to different body types, just that it is a commonly used method for pose detection.  I did find one study by De Gama et al. That showed a 100% accuracy for pose detection among a small sample size using a joint angle based pose detection system.  Based on my findings, we can have some confidence in the accuracy of our joint angle based approach to lose detection.

I also looked into several options for packages other than TKinter to use for the frontend of our app.  Ultimately, I brought forward the wxPython and Kivy packages forward for the team to consider.  We decided to switch from TKinter to Kivy for our frontend development after some debate.  What won us over to Kivy was its continuous support from professional developers and that it was compatible with a Python3+ versions.

For the next week, I plan to create the frontend of my app to be able to allow users to upload custom images to the app.  I will also discuss more with Ray about how we plan to integrate our code together.  I will also work with Hongzhe to get Openpose up and running on my machine as well.

ABET question #7:

The new tool I am looking into learning is the “Kivy” Python package that will allow us to make a much more sophisticated frontend for our application.  In particular I am looking at the various Kivy widgets pertaining to my pipeline, such as the FileChooserListView widget.

1.  This week I spent time trying to get openpose to build.  I was unsuccessful due to files to install dependencies being missing from the release of openpose I tried to install.  To be more specific Caffe is a necessary dependency and I had to manually install Caffe into the openpose directory structure.  The build_win.cmd file fails to run as I do not have “ninja” installed.  I tried to set the “WITH_NINJA” flag to 0 so CMake would build caffe using the Visual Studio C++ compiler instead of ninja.  When I do this and try to run the file the flag seems to somehow get set back to 1.  This is a very mystifying issue and I plan to ask Eric about it tomorrow and see if he ran into the same issue.  In addition to this the weeks ABET question prompted me to think more about user privacy which led me to look into options and techniques for doing sanitization of user inputs in python and analysis tools to find potential security issues.  The main vulnerability I am worried about is if User A has images stored on the app, and User B logs into the app and is able to inject malicious code through fields of user input (how long to record user for).  I think this is a reasonable worry as we have a decent amount of user input into our system so I think user sanitization is a very sensible idea.  So far I think the bandit tool looks somewhat promising as it is distributed in PyPI so it can be downloaded through pip.  There are customization options for us to ignore certain levels of security issues, allowing us to focus on the most critical issues.  In terms of user sanitization I am still looking at options and need to discuss them with Sirui tomorrow, but as of right now “regular expressions” seems promising due to its ability to filter out certain items in user input and it is a standard package in python so there is no need to install anything to use it.  I was also involved in team meetings where we clarified parts of how our design would work, specifically the specifics of how we were going to evaluate users poses.
2. My progress is on time with the group schedule.
3. Deliverables I hope to complete by the end of next week is to help Ray get a basic UI design out for the design report and get openpose to build.  I also plan to maybe look into and discuss more advanced tools for UI instead of just using TKinter.
4. In terms of engineering principles that our team will use to develop solutions, I personally focused on the principle of ethics as I wanted to make sure this app would protect users privacy.  Especially with our app where there would be pictures taken of people in incorrect poses, this might be embarrassing for some people and something they wish to keep private.  Towards this end I looked into incorporating user sanitization into our app to prevent “injection” attacks as we will be taking in input from the user.  Additionally, I looked into tools the team might want to incorporate into the testing process.