Category: Tahaseen Status Reports

This week, I worked on integrating the homography logic into the overall Flutter UI with Caroline. This required me to make some changes to the process of calibration and the input of images. Furthermore, I added more error handling methods to make sure the program does not just crash on usage (when more than 4 corners are detected). A problem I have been working on is the light gradient in various light settings. As the light disperses with the angle of the projector, the light is considerably dimmer and faded out. To fix this, I may have to change the overall projection to be closer to the projector. Furthermore, we standardized the size of the projection space to ensure that we can work perfectly in a set space – but also remain flexible to tuning to other spaces.

This coming week, I will be spending a lot of time completing the final report and filling out the poster. Additionally, I will be adding wire management and aesthetic factors to the setup (specifically the camera mount). As needed, I will continue tuning the calibration and overall system display.

This week, I finalized a camera mount that is set to an appropriate height for our testing environment in HH 1303. Additionally, I decided to use red markers in order to guide the user in outlining the table in the case of projector overflow. Processing these sped up calibration significantly and allowed it to be a more consistent determination of the table outline. The rainbow streaking is a problem we are circumventing by adjusting the selected colors for projection and reducing the overall amount of projection. After these updates and the new camera setup, I have been working on mapping the projection appropriately onto the table so that the homography retains a standard level of brightness. This means playing with the height and angle of the projector further to find an optimal setup.

This coming week, I will finalize my process and complete the integration of calibration into the UI. This will enable us to run everything from one device and one package. In this project, I learned a lot more about calculating homographies and the limitations of a projector system. I also learned how to use Flutter, which I’ve never had to use previously. My main strategy was to rely on forums and data sheets in order to gain a full understanding of the components I was using. For Flutter, I really relied on their online documentation and example apps other people had made using this tool.

This week, I focused on putting together the entire pipeline on the actual projector and camera setup. There were a lot of lessons learned during this portion of integration because we learned we needed more stable mounting mechanism and clearer guidance toward the user on setup. Furthermore, when testing the camera with the projector, I realized that the camera saw the projection in the RGB states because of the faster fps. This will likely not be a problem however by grayscaling and thresholding. This is necessary for the auto-calibration process. The USB camera we were testing with has a fisheye effect that our actual camera does not have.  None of these factors will delay our timeline significantly, but this does mean I need to put more bandwidth on manufacturing a more secure mounting for the camera and projector.

Next week, I will manufacture the mounts for the projector and camera. I will also work on speeding up the autocalibration by running some computer vision on the images taken in.

Verification:

• Projector Calibration
  • Ensure the bleed over onto the floor is within the specified range from the requirements. If not, update table corner detection to identify table bounds.
  • Ensure that the spacing is measured to be the distances specified in the requirements. If not, change table bounds.
• Hardware
  • Ensure that the hardware mounting for the projector is secure and not susceptible to falling from minor bumps. Verified via checking camera detection of projection difference.
  • Ensure that the hardware mounting for the camera is secure and not susceptible to falling from minor bumps. Verified via checking camera detection of projection difference.
• Recipe Processing
  • Ensure that the recipe is split into appropriate steps. This will be analyzed from the user acceptance process.

This week, I worked on translating the warp homography and code to Flutter and tried it on the actual planned projector setup. Previously, I had been using a smaller projector for testing purposes. There have been several snags in the process of designing the calibration for the initial warping. Since the current method requires user input for the homography, it makes the process a little harder and less user friendly than intended. The checkerboard is meant to make this easier but is still under quality testing. Integrating with Flutter has been more difficult than anticipated given the different hierarchical structure. Futhermore, I think it may be necessary to design a lip for the new projector stand to ensure the projector is comfortably situated on the stand while optimizing the projection on the table.

I am on track this week, but plan to spend significant time on streamlining the calibration procedure.

This week, I spent some time working with the Xavier and tried to connect the camera to it so that we would be able to do some preliminary testing. However, there were a few roadblocks here that should be resolved this week as we work to onboard the Xavier. On the other hand, I revisited the projection method such that it utilizes the camera and prevents the user from having to mess with the projector setup. This would involve another step in the calibration process that would create the homography based on the existing camera and projector setup. Furthermore, I have designed the first iteration of the hardware mount that will be 3D printed this week unless an existing metal/wood solution is found. There is some concern that the filament will not hold up.

I am on track and this week I plan to work on completing the calibration procedure, mount and sorting out our central compute.

This week, I talked to Marios and acquired a much larger tripod that is perfect for mounting the projector. Since the new one is sturdier than the first, the first will be used to mount the camera + AGX and the second for the projector. This is a positive change from our previous system and will require new brainstorming on mounting designs. Additionally, I outlined the projector calibration and have begun implementing the script for it. Currently the procedure is (1) inform user calibration is taking place, (2) detect table outline through user markers, (3) verify that the projected calibration grid matches an unwarped version and either reimage in software or prompt the user to raise the projector to preset height/angle changes. Caroline and I also discussed the pros and cons of using the homography that came with the Flutter UI vs a Python script that would require a frame by frame input.

I am on track and this week I plan to schedule out large portions of time to implement a working calibration script and design a hardware mount with our new materials. Our team is on track for integration before the interim demo

This week, I placed the order for a stronger projector and received it. I had been testing with a smaller Panasonic projector, but after testing in different light modes, I found that the brightness was insufficient to meet the requirements of the project. Afterwards, I helped my team setup our coding workspace in Bitbucket so that we could track our progress by issue using Jira and regularly merge with each other to avoid conflicting code. This way, integrating everything at the end will be more seamless and we can avoid any downstream issues. There was a shuffle in task assignment so I will no longer be focusing on setting up the backend of the user site. Instead, I will be focusing more time in making sure the projector is correctly and intelligently calibrating to the workspace table. Therefore, I focused on researching potential steps for the projector calibration mode through existing calibration methods for projectors. After reviewing various resources (these 2 were particularly helpful: https://us.seenebula.com/blogs/buying-guides/projector-calibration and https://simplehomecinema.com/2022/07/06/what-you-need-to-calibrate-your-tv-or-projector/), I was able to gain more insight into the types of things I needed to ensure were calibrated for a satisfactory user experience.

I am currently on track with my tasks. Next week, I will have a barebones setup for a calibrator that I can display through the projector. Additionally, I need to think of an alternative mount for the larger projector on the tripod.

This week, I collected a projector and a tripod from Prof. Sankaranarayan.  We discussed different mounting solutions – specifically to avoid negative interactions with the cooking process (steam, smoke, food splatter, etc.). After some discussion, I decided to switch to having a projector angled from a side view that would project onto the table. We would calculate the appropriate planar homographies in order to warp our image. I have begun the initial calculations and now need to test them. Additionally, the brightness of the projector is satisfactory for its small size, but will require the lights in the testing space to be slightly dimmer than an average kitchen. Finally, I helped Sumayya troubleshoot flashing the AGX from my laptop. I was able to install the required SDK but had some trouble connecting the peripheral. However, we resolved this via using a PC.

I am on currently on track, but by next week I want to run a test with my computed homographies. I anticipate this to be a larger task because it will require a lot of fine testing. I also want to do an initial mockup of the web app.

This week, I researched specs needed for the projector portion of the project and began outlining the specifications for the mounting of the product. This was the most challenging because I had to determine what was an appropriate fit for our project. I reached out to Prof. Aswin Sankaranarayan for advice on this and camera placement. I also worked with my team to outline the individual technical requirements and testing & requirements. Additionally, I reviewed the HTTPServer documentation and found it pretty easy to read. Setting up one was pretty intuitive and should be easy for the final product. I am on schedule.

For this upcoming week, I want to have a design for the hardware mount. Additionally, I need to acquire a projector with an appropriate lumens and do some distance testing to verify brightness.