Team A4: UsAR Mirror

What did you personally accomplish this week on the project?
This week, I worked on integrating the 3D face tracking pipeline with real-time AR rendering, including syncing 6DoF head pose estimation using depth-based landmarks and OpenGL model placement. I also optimized overlay performance and began evaluating pixel-level drift across different detectors.

Is your progress on schedule or behind?
I’m slightly behind schedule. Integration is taking longer than expected, but I plan to finish it within the next few days.

What actions will be taken to catch up?
I will prioritize completing the C++ pipeline and debugging edge cases to ensure model rendering correctly follows facial motion.

What deliverables do you hope to complete in the next week?
We aim to complete full system integration and deliver a minimally functional final product. I will also focus on testing and quantifying system latency, drift, and pose accuracy to support our performance analysis.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

One major risk is the unreliability of gesture recognition, as OpenPose struggles with noise and time consistency. To address this, the team pivoted to a location-based input model, where users interact with virtual buttons by holding their hands in place. This approach improves reliability and user feedback, with potential refinements like additional smoothing filters if needed.

Finally, GPU performance issues could affect real-time AR overlays. Ongoing shader optimizations prioritize stability and responsiveness, with fallback rendering techniques as a contingency if improvements are insufficient.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

No changes have been made.

Provide an updated schedule if changes have occurred.

This week, the team is doing full system integration, finalizing input event handling, and testing eye-tracking.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I finished building the 2nd camera rig. I also finished setting up the UI so that I can now work on it.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am on track. I will finish the UI this week.

What deliverables do you hope to complete in the next week?

I hope to finish my UI.

What did you personally accomplish this week on the project?

I continued worked on testing my pose recognition UI, and generating testing metrics to see if we meet the design requirements. I also worked on the final presentation.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Behind schedule. We still need to integrate everything together. Catherine’s code should in theory integrate with my code base. I’ve also verified that Anna’s code works with mine. So, pairwise, the systems have been integrated. But we still need to integrate everything as a final product.

What deliverables do you hope to complete in the next week?

Integration. We hope to have a (barely) working final product by the end of this week.

Testing:

To test my subsystem, I’ve created tests for each of the design requirements we are trying to target, (1) latency (2) input accuracy and (3) pose estimation accuracy.

1.  To test latency, I’ve simply measured the time it takes for the gesture recognition pipeline to receive the image and then provide pose estimates (essentially model evaluation time). This was simple as inserting “stopwatches” into the codebase, and averaging measurements. The end-to-end latency was about 55 ms, which meets our target.
2. For input accuracy, I made “test inputs” for each of the inputs that we wanted to test. Since the inputs are now location based, these tests essentially consisted of holding up my hand over the button I wanted to click on the screen. Overall, this test went very well, since the buttons were fairly large and OpenPose wasn’t noisy enough to “miss” these buttons. Out of the 20 trials I did for a button, 19-20 of them would “pass” (>= 95% acc)  — which means registering an input. However, the input accuracy would change under different lighting conditions, meaning that if the room was too bright or too dark, the accuracy would drop to around 18-20 / 20 successes (>= 90% acc), which is still good enough for our purposes.
3. For pose estimation, I held my hand relatively stationary, and measured the standard deviation, and max deviation from the mean. For a 1080p camera, this deviation was about 35 px, which is a little more than we’d like (<20px), but is still good enough for our purposes. Note that this metric is bound by the model we choose (in this case OpenPose).

What did you personally accomplish this week on the project?
This week, I worked on integrating and rendering 3D objects—such as glasses—on top of a human face using OpenGL. I aligned the 3D model with facial landmarks and refined the rendering pipeline to ensure the virtual object appears naturally overlaid in real time. This required deeper integration with face tracking data and improvements to pose stability and object placement.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?
I’m on schedule. However, GPU performance and integration continue to be a challenge—especially on the Jetson platform, where driver and shader issues have impacted rendering stability. Given that this is a prototype, I’ll be making compromises on some graphical fidelity and fallback features to prioritize real-time responsiveness and ensure the demo remains functional and stable.

What deliverables do you hope to complete in the next week?
Next week, I plan to finalize the alignment and positioning of the glasses overlay under varying head poses. I will also test the robustness of the pipeline under different motion and lighting conditions. If time permits, begin generalizing the overlay system to support additional 3D elements (e.g., hats or earrings).

Verification and Validation Plan
Verification (individual, subsystem-level):
Pose accuracy: Comparing the rendered glasses position against known synthetic inputs or manually annotated frames to ensure consistent alignment with the eyes and nose bridge.

Latency: Measuring rendering latency from input frame to final render output using timestamped profiling to ensure the pipeline meets real-time constraints (<100ms end-to-end).

Robustness: Testing the system under varied conditions—such as rapid head movement, occlusions, and lighting changes—to ensure the overlay does not jitter or drift significantly.

What did you personally accomplish this week on the project?

I worked on testing my pose recognition UI, and making it more robust to noise generated from inaccuracy from OpenPose. I also worked on adding more UI components, for all the features that we want in the final product. I also worked on serial input to the arduino (part of camera rig), and mapped inputs from my gesture/location control to serial inputs to Anna’s camera rig.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Behind schedule. I’ve integrated my part of the project with Anna’s, so now the application can control camera rig. However, I am behind trying to build the program on the Jetson.

What deliverables do you hope to complete in the next week?

Continue refining the UI, and attempt to integrate Catherine’s part of the project.

Verification

I verify my system by (1) making sure that my location-based input system reaches the desired accuracy by making a series of desired inputs (for example, hovering my hand over the left swipe button) and recording how many of these test inputs fail/succeed. I will also make sure that the eye-tracking system (with the camera rig controller) corrects the eye level by first setting different eye levels on the screen to “calibrate to”, and making sure the camera rig corrects for it within the desired percentage away from the desired screen eye leel.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

One major risk is the unreliability of gesture recognition, as OpenPose struggles with noise and time consistency. To address this, the team pivoted to a location-based input model, where users interact with virtual buttons by holding their hands in place. This approach improves reliability and user feedback, with potential refinements like additional smoothing filters if needed.

System integration is also behind schedule due to incomplete subsystems. While slack time allows for adjustments, delays in dependent components remain a risk. To mitigate this, the team is refining individual modules and may use mock data for parallel development if necessary.

Finally, GPU performance issues could affect real-time AR overlays. Ongoing shader optimizations prioritize stability and responsiveness, with fallback rendering techniques as a contingency if improvements are insufficient.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Gesture-based input has been replaced with a location-based system due to unreliable pose recognition. While this requires UI redesign and new logic for button-based interactions, it improves usability and consistency. The team is expediting this transition to ensure thorough testing before integration.

Another key change is a focus on GPU optimization after identifying shader inefficiencies. This delays secondary features like dynamic resolution scaling but ensures smooth AR performance. Efforts will continue to balance visual quality and efficiency.

The PCB didn’t exactly match the electrical components, especially the stepper motor driver being used. We had ordered a different kind of stepper motor to match our needs (being run for long periods of time), but it required an alternative design. So, we made new wire connections to be able to use the stepper motor.

Provide an updated schedule if changes have occurred.

This week, the team is refining motion tracking, improving GPU performance, and finalizing the new input system. Next week, focus will shift to full system integration, finalizing input event handling, and testing eye-tracking once the camera rig is ready. While integration is slightly behind, a clear plan is in place to stay on track. We will begin integrating the camera rig that is ready while the second one is being built.

In particular, how will you analyze the anticipated measured results to verify your contribution to the project meets the engineering design requirements or the use case requirements?

We will use logs to determine whether there is more than 200 ms latency between the gesture recognition and camera movement and a 1 s delay for generating face models.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I found a way to mount the camera rig securely. I also got started on building the 2nd camera rig.

I also got all four stepper motors to work.

Steven and I also integrated gesture recognition with the camera rig for two stepper motors. However, it shouldn’t be a problem for four stepper motors since the command is just a little different.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am on track. I will finish building the 2nd camera rig this week so that I can get started on the UI.

What deliverables do you hope to complete in the next week?

I hope to finish my 2nd camera rig and work on the UI.

How will you analyze the anticipated measured results to verify your contribution to the project meets the engineering design requirements or the use case requirements?

I will test that my camera rig doesn’t move linearly for no more than 11.8 inches (the width of the display) by including a stop feature when it reaches the distance limit. The same thing goes for the angle. I will include a stop feature to make sure that it doesn’t turn beyond 90 degrees relative to the center. As for the 5 degree deviation, the way the camera rig works is that it increments the distance or rotation little by little so that users can control how much camera the user wants to move by, so there will be no deviation.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

One major risk is the unreliability of gesture recognition, as OpenPose struggles with noise and time consistency. To address this, the team pivoted to a location-based input model, where users interact with virtual buttons by holding their hands in place. This approach improves reliability and user feedback, with potential refinements like additional smoothing filters if needed.

System integration is also behind schedule due to incomplete subsystems. While slack time allows for adjustments, delays in dependent components remain a risk. To mitigate this, the team is refining individual modules and may use mock data for parallel development if necessary.

The camera rig needs a stable stand and motion measurement features. A second version is in progress, and if stability remains an issue, alternative mounting solutions will be explored.

Finally, GPU performance issues could affect real-time AR overlays. Ongoing shader optimizations prioritize stability and responsiveness, with fallback rendering techniques as a contingency if improvements are insufficient.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Gesture-based input has been replaced with a location-based system due to unreliable pose recognition. While this requires UI redesign and new logic for button-based interactions, it improves usability and consistency. The team is expediting this transition to ensure thorough testing before integration.

Another key change is a focus on GPU optimization after identifying shader inefficiencies. This delays secondary features like dynamic resolution scaling but ensures smooth AR performance. Efforts will continue to balance visual quality and efficiency.

Provide an updated schedule if changes have occurred.

This week, the team is refining motion tracking, improving GPU performance, stabilizing the camera rig, and finalizing the new input system. Next week, focus will shift to full system integration, finalizing input event handling, and testing eye-tracking once the camera rig is ready. While integration is slightly behind, a clear plan is in place to stay on track.

What did you personally accomplish this week on the project?
This week, I focused on preparing a working prototype for the interim demo. I finalized the integration of motion tracking data with the OpenGL rendering pipeline on the Jetson Orin Nano, which now supports stable AR overlays in real time. I implemented basic camera motion smoothing to reduce jitter and improve alignment between the virtual and real-world content. On the performance side, I began profiling GPU usage under different scene conditions, identifying a few bottlenecks in the current shader code.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?
I’m on schedule for the interim demo, though there’s still polishing left. Some experimental features like dynamic resolution scaling are still pending, but the core functionality for the demo is working. To stay on track, I’m prioritizing stability and responsiveness, and will continue optimizing shader code and system performance after the demo.

What deliverables do you hope to complete in the next week?
Next week, I plan to polish the demo for the presentation, focusing on smoother motion tracking, more consistent AR overlays, and better GPU performance. I also want to finish porting over the refined blending strategies from macOS to the Jetson, and begin experimenting with fallback rendering techniques to handle lower-performance scenarios gracefully