Author: stevenl4

What did you personally accomplish this week on the project?

I worked on testing my pose recognition UI, and making it more robust to noise generated from inaccuracy from OpenPose. I also worked on adding more UI components, for all the features that we want in the final product. I also worked on serial input to the arduino (part of camera rig), and mapped inputs from my gesture/location control to serial inputs to Anna’s camera rig.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Behind schedule. I’ve integrated my part of the project with Anna’s, so now the application can control camera rig. However, I am behind trying to build the program on the Jetson.

What deliverables do you hope to complete in the next week?

Continue refining the UI, and attempt to integrate Catherine’s part of the project.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

One major risk is the unreliability of gesture recognition, as OpenPose struggles with noise and time consistency. To address this, the team pivoted to a location-based input model, where users interact with virtual buttons by holding their hands in place. This approach improves reliability and user feedback, with potential refinements like additional smoothing filters if needed.

System integration is also behind schedule due to incomplete subsystems. While slack time allows for adjustments, delays in dependent components remain a risk. To mitigate this, the team is refining individual modules and may use mock data for parallel development if necessary.

The camera rig needs a stable stand and motion measurement features. A second version is in progress, and if stability remains an issue, alternative mounting solutions will be explored.

Finally, GPU performance issues could affect real-time AR overlays. Ongoing shader optimizations prioritize stability and responsiveness, with fallback rendering techniques as a contingency if improvements are insufficient.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Gesture-based input has been replaced with a location-based system due to unreliable pose recognition. While this requires UI redesign and new logic for button-based interactions, it improves usability and consistency. The team is expediting this transition to ensure thorough testing before integration.

Another key change is a focus on GPU optimization after identifying shader inefficiencies. This delays secondary features like dynamic resolution scaling but ensures smooth AR performance. Efforts will continue to balance visual quality and efficiency.

Provide an updated schedule if changes have occurred.

This week, the team is refining motion tracking, improving GPU performance, stabilizing the camera rig, and finalizing the new input system. Next week, focus will shift to full system integration, finalizing input event handling, and testing eye-tracking once the camera rig is ready. While integration is slightly behind, a clear plan is in place to stay on track.

 

What did you personally accomplish this week on the project?

Gesture recognition seems too unintuitive for the application, plus the inputs we received from the pose recognition model were too noisy for reliable velocity estimation for gestures. So, I pivoted to a location based input model, where the user instead of making gestures, moves their hand to a virtual button on the screen, which registers input if the user “holds” their hand location over that button for a period of time. This is a better solution, since estimating position was a lot more reliable than estimating velocities, since (I don’t think) OpenPose is time consistent. Also, visual buttons on the screen provide better feedback to the user.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Behind. Currently we’re supposed to do system integration. But I’m waiting on my partners subsystems, and currently finishing up my own. We have a little slack time for this, so I am not too worried.

What deliverables do you hope to complete in the next week?

I hope to write input events/buttons for all the input events we have planned as features for our project. I also hope to get started on testing the eye-tracking system (i.e. correcting eye level) once Anna finishes her camera rig, through serial command inputs from the program to Arduino via usb.

 

What did you personally accomplish this week on the project?

https://github.com/kevidgel/usar-mirror/commit/5f6e604137110f6559df3144245f885c7efa9c0f — the pose tracking doesn’t suck anymore. Also worked on the gesture recognition algorithm to be more robust to missing/implausible data.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Behind. But I will spend some extra time this week to catch up.

What deliverables do you hope to complete in the next week?

Maybe I get to compile it on the Jetson if I have time. But mainly testing gesture control.

 

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Most significant risk currently is the system integration failing. So far everyone has been working on their tasks pretty separately (software for gesture recognition/eye tracking, software for AR overlay, hardware for camera rig + UI). It looks like everyone has made significant progress on their tasks, and are close to the testing stage for the individual parts. However, not much testing/design has gone into how these subprojects will interface. We will discuss this in the further weeks. Moreover, we have made some time in the schedule for the integration, which gives us ample time for making sure everything works.

Another risk is performance, the compute requirement for the software is a lot and the Jetson may not be able to handle it. But this has already been mentioned in our last team status report, and we are currently working on it.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

No changes currently.

Updates & Schedule change

The team seems to be roughly on schedule.

 

What did you personally accomplish this week on the project?

Worked on the gesture control algorithm. I implemented a left/right swipe recognition input based on keypoint velocity, but due to OpenPose being fairly slow an the actual keypoints being tracked being fairly noisy, the accuracy isn’t that good. Potential solutions include switching from a velocity-based algorithm to something more positional, such as detecting whether the hand is to the right or left of shoulder, or writing a filter for more stable keypoints –> more stable velocities. Regarding eye-tracking, I managed to extract eye keypoints in screen-space coordinates. Since the camera rig isn’t finished, and the webcam I currently have is fixed, I simply wrote a function to output the signed distance between the average y-position of the eyes and a certain height (in (-1,1) viewport coordinates). This will be input to a feedback algorithm to adjust our camera rig.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I’m supposed to be doing gesture recognition algorithm testing right now (re: schedule), but currently I’ve been strapped for time and not doing that. More refinement needs to be done to the algorithm to make it robust enough for our application. To catch up, I will allocate more time in my schedule for this project.

What deliverables do you hope to complete in the next week?

I do plan to refine the algorithm to be more robust against inaccurate readings from OpenPose, as well as doing some performance fine tuning so that keypoint collection happens faster and the algorithm could be more accurate. The inputs it detects are also pretty limited, will have to discuss more about which gestures we want however. I also don’t have the Jetson currently, and I know I’ve been putting this off literally every week since the second week, but I have yet to build OpenPose on the Jetson.

What did you personally accomplish this week on the project?

I finally got the Jetson and I spent some time flashing the board with new firmware so we can use it for our project. However, I was a little busy this week, so I mainly worked on figuring out the design details of my portion of the project. Here is the planned system diagram of how I’m going to implement gesture control + eye  tracking.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am a little behind, and I plan to be even more behind due to a few midterms next week. Because OpenPose can do keypoint tracking for both the eyes, facial features, joints, I think I can work on both gesture + eye tracking at the same time to speed up development. The schedule was planned with a lot of slack time, so I don’t feel pressured.

What deliverables do you hope to complete in the next week?

Get OpenPose running on the Jetson, and start software foundation for our application.