Following the interrim demo, our group continued to work on the frontend and backend of our system to further polish the functions and begin the process of integration.
The imminent problem mentioned during the interim demo about palm angles raised by professor Tamal was resolved in this week, as Shiheng and Eric worked through the openpose system and figured detecting keypoints on hands to determine hand postures.
We continued to discuss details for the future weeks of the project and made various important design decisions on certain details of our software. We synced up our progress in Saturday meeting with Eric explaining integration steps of the software, Shiheng explaining the approach on the backend about pose comparison, Ray and Jerry working on the pipeline and communicating with the needs and ideas that backend needs to pass onto the frontend and UI.
Steps to get the application working:
- Everybody work on main
- Git clone from main branch into folderX
- Copy everything from folderX into openpose root directory (C:/xxx/openpose/)
- C:/xxx/openpose/ – Python3 main.py
- Remember to recursively copy all the things, and remember to have bin/openposedemo.exe ready
Things to Do:
Front end:
-
- Calibration (Integrated)
- Pretty UI?
- Sequence (Integrated in Training Iteration)
- Skeleton Drawing – Ray (Discuss with Roland(shiheng))
- Parse JSON to get coordinate/angle
- Roland: JSON from openpose contains all coordinates of bodypose being captured, will just connecting the dots get an appropriate skeleton? Normalization choice: Scaling the whole body by a factor (rough match)/tailoring limbs and body parts (exact match)?
- Redraw the skeleton
- Mark red for incorrect body parts
- Roland: Pass back an error_list containing all the incorrect body parts above the threshold. No prioritizing body parts here, all incorrect body parts will be passed back.
- Parse JSON to get coordinate/angle
- Custom Upload renaming – Done by Jerry
- Training Iteration – Jerry (Discuss with Eric for integration)
-
- User tries to start training by clicking “Start” button in the training page
- Camera get user image and send to backend (10s countdown)
- User image processed by OpenPose to get user coordinates
- Case 1: user coordinates contains every body part
- Front end receives correct status code/return value
- Display skeleton, verbal instructions is reading
- ***If the user is correct, change to the next pose – Sequence working!!!
- Case 2: user coordinates does not contain every body part
- Front end receives bad return value
- Display a flash warning saying your body is not in camera
- Verbal speak “Change your position”
- Iterate back to step 2
- “Stop” button clicked to stop process or change pose…
Back end:
- Error Priority – Roland
- Feet->Leg/Thigh->Upper Body(Torso)->Arms, Instructions should be issued from this sequence (e.g Errors in Arms shouldn’t be mentioned before errors in lower body is fixed)
- One instruction/error at a time to not confuse users learning Taichi
- User Selection (Multiple people in frame chose the user in training) – Roland
- Choosing the most similar person in frame (Similarity scoring)
- Openpose input formatting for multiple people?
- Instruction Wording – Roland
- Dividing body parts into ‘head’, ‘torso’, ‘arms’, ‘legs’, ‘feet’, giving different instructions for corresponding body parts
- Macro connection – Eric
- Continue to integrate the frontend and backend, work with rest of the team
Later:
- Verification & Performance Evaluation, Data collection for the final report (Second Last week)
- User interview and UI test, report, slide, presentation. (Schedule for those events and start right after the integration is finished and functionalities achieved)