Month: February 2025

Accomplishments This Week:

• Planning and Research:

The flow diagram illustrates the high-level process for a 3D reconstruction pipeline using Intel RealSense and side webcams. The process begins with a one-time synchronization and calibration of the RealSense camera with the webcams, so we would be able to know the relative position of webcams in relation to RealSense to better render resultant images from their perspectives. This step involves ensuring that all cameras are spatially and temporally aligned using calibration techniques such as checkerboard or AprilTag patterns. The goal is to establish a unified coordinate system across all devices to facilitate accurate data capture and reconstruction.

Following calibration, the depth image capture phase begins, where the RealSense camera captures depth. The captured data is then processed using Dynamic Fusion, a technique that performs non-rigid surface reconstruction. This step updates and refines the 3D face model in real time, accommodating facial deformations and changes.

Once the face model is processed, the pipeline evaluates whether the 3D face model contains holes or incomplete areas. If gaps are detected, the system loops back to Dynamic Fusion processing to refine the model further and fill in the missing parts. If the model is complete, the pipeline proceeds to facial movement tracking, where the system shifts focus to monitoring the user’s facial movements and angles across frames. This is not too changing since the face would move rigidly so only one 6DoF transform is needed per frame.

Finally, the reconstructed model is aligned with the side webcams during the 3D model alignment step using the 6DoF transform we computed. This ensures the resultant AR overlay would accurately stick on the face of the users.

• Dynamic Fusion Investigation:
  • Discovered an OpenCV package for DynamicFusion, which can be utilized for non-rigid surface tracking in our system.
  • Began reviewing relevant documentation and testing standalone implementations.
• Calibration Code Development:
  • Wrote the AprilTag and checkerboard-based calibration code.
  • Cannot fully test yet, as RealSense hardware is required for debugging and system implementation. (still waiting for RealSense camera)

Project Progress:

• • Slightly behind due to missing hardware required for implementation (I need the RealSence in order to know the input format of the depth information and to integrate Dynamic Fusion), but was able to complete the design and most part of implementation.
  • Currently I have tried using Blender to create scenes with depth map and RGB image to simulate RealSence data to use in my program, currently have a working version that is slower than the requirement (takes around few seconds per frame).
  • Completed the camera calibration code but also cannot test if it works due to lack of hardwares.
  • Another concern I have is opencv’s package is able to work locally, but the environment setup is pretty challenging and might be hard to migrate to jetson.
• Actions to Catch Up:
  • Continue refining non-rigid transform calculation process using DynamicFusion.
  • RealSense was ready for pickup, will integrate RealSense measurements with my code.
  • Also needs to check how necessary is using DynamicFusion since it does pose a pretty big delay to the 3D reconstruction processing pipeline.
• By next week, I hope to be able to run the RealSense on my code to validate initial results. By next status report, I should be able to attach images of 3D reconstructed model of my face using the pipeline I wrote

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).  

     This week, I worked on designing the control system for the camera’s rotation, tilt, and vertical movement. At first, I considered building a custom camera rig from scratch using servos for rotation and tilt and linear actuators for vertical movement. However, I realized that integrating these components with the Arduino could be tricky, and that coming up with my own design that isn’t polished enough might lead to wasted time and money. To make the process more efficient, I decided to research existing designs that could be adapted for our project to ensure that we have a working design.

     I came across several options, but most were either too complex or lacked clear instructions. One design stood out because it allowed for pan, tilt, and vertical movement—exactly what we need for our augmented reality mirror. I would have three of these set up on the mirror; one horizontally at the top of the mirror for the realsense camera and two vertically on the sides of the mirror for the webcams. However, the design required a lot of material prep, had minimal step-by-step guidance, and involved a more complicated assembly.
Reference: https://www.youtube.com/watch?v=hEBjbSTLytk 

     I also spent time researching different motor options for controlling the cameras, focusing on cost and ease of implementation. After looking at multiple designs, I chose one that looked similar, except that it included 3D-printable parts, making it much easier to put together. This design also provides a full list of required parts, estimated costs, and dimensions, which helped me confirm that it would work with our webcams and budget. I would have to adjust the dimensions of the parts to fit the realsense camera (which is longer). I also made sure it could be smoothly integrated into the overall project.
Reference: https://www.instructables.com/Automatic-Arduino-Powered-Camera-Slider-With-Pan-a/

     I decided to use an Arduino as the main controller because it’s easy to program, has strong support for motor control, and works well with both servo motors and linear actuators. Its built-in libraries make it simple to create precise movements, allowing for smooth camera adjustments in all directions. Plus, it leaves room for future improvements, like adding user controls or automating movement based on environmental data.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

     My progress is a little behind. This week, we had our proposal presentation, and I started on designing the camera control system when I was supposed to start assembling the camera control system. I worked on researching all the necessary materials and assessed whether the design is feasible for our project, given the fixed budget and time frame. I will go to Techspark on the 9th to see which materials I can borrow and fill out the purchase form so that I can get the materials as soon as possible. Once I get all the materials, I can start building this week so that I can work on programming the control system and integrating it into the mirror later on.  

What deliverables do you hope to complete in the next week?

     Next week, I hope to have my digitally fabricated parts ready for me to assemble the week after next week. I would have to adjust the dimensions in the STL files so that our cameras can fit in the designs. I hope to have my other materials ready and delivered so that I can have all the materials ready to build the week after next week.