Posted on

Tahaseen’s Status Report 04/27

This week, I worked on integrating the homography logic into the overall Flutter UI with Caroline. This required me to make some changes to the process of calibration and the input of images. Furthermore, I added more error handling methods to make sure the program does not just crash on usage (when more than 4 corners are detected). A problem I have been working on is the light gradient in various light settings. As the light disperses with the angle of the projector, the light is considerably dimmer and faded out. To fix this, I may have to change the overall projection to be closer to the projector. Furthermore, we standardized the size of the projection space to ensure that we can work perfectly in a set space – but also remain flexible to tuning to other spaces.

This coming week, I will be spending a lot of time completing the final report and filling out the poster. Additionally, I will be adding wire management and aesthetic factors to the setup (specifically the camera mount). As needed, I will continue tuning the calibration and overall system display.

Posted on

Team Status Report for 4/27

Projection warp may not work for demo environment. We need to make sure to complete calibration before the actual demo time.

Additionally, we found that the projection calibration slightly changed once we deployed the calibration code on flutter. We did not account for the fact that Flutter does transformations differently than python/numpy which is why much of this week was spent on adjusting the homography to work on Flutter. Regardless, we were able to make the necessary changes and complete integration.

No changes made to system.

No schedule changes.

Unit Tests:

  • Button Press / Swipe Test
    • tested how well and how long it is taking for system to recognize and execute button gestures.
    • Found that the timing is reasonable and the accuracy is about 90% ( out of 10 trials, accurately recognized the gesture 9 times)
    • Realized that video frame needs to be cropped/zoomed into button region to detect gestures. Otherwise the hand is too small for gestures to be recognized
  • Voice command Tests
    • tested how well and how long it is taking for system to recognize and execute voice commands
    • found that volume and pitch of one’s voice impacts recognition. The full system had difficultly recognizing commands from Caroline’s voice but was easily able to recognize Tahaseen’s voice commands when she spoke loudly and with a low pitch.
  • Recipe execution tests
    • we tested how the cooking experience is impacting the overall system. For example, when boiling water, we wanted to see if the steam was blocking the camera view.
    • Found that the steam doesn’t greatly impact the view as the burner is located to the side of the camera
Posted on

Sumayya Syeda’s Report for 4/27

Progress Update:

This week I attempted a new method for how I will be detecting gesture recognition regions. Previously, I was using SIFT to look for the template button image in the projection image captured by the camera. But poor lighting and projection warp proved this method to be difficult.

I realized that instead of using the image of the projection to detect the button, I can instead use the already calculated homography to map coordinates in the UI to the coordinates in the projection. In this way, I can remove the need for the camera to detect the button and instead use the camera only to compute the homography. I decided to go with this method for the final demo as it works better at recognizing the button region.

While testing gestures with the usb camera, I realized that I need to zoom in to the frame / button region as the model can not recognize gestures if the hand does not take up more of the frame. This is why it became even more important to be able to detect the button region so I can properly crop the area.

Schedule: On Track!

Next Steps:

  • Test all features with both recipes
  • Continue fine tuning gesture recognition
  • Practice for demo day
  • Work on Report & Poster
Posted on

Caroline’s Status Report for 04/27

This week, I continued to work on integration. We switched to using my laptop for the project, so I was mostly working on system integration. The calibration script has been added to the pipeline so that the program waits for when the user interacts with the table calibration part. Before, it was assumed that the calibration would happen without user interaction, but now, when the user performs a new calibration, they web app shows a loading screen on the web app -> tells the user to move the red dots on the table -> the user confirms on the web app that it is done -> web app shows calibration loading screen until calibration is complete. I also have been working on fine-tuning the voice commands to work more consistently when everything is running together. It works very well independently, but it still does not recognize enough when running all together.

The voice commands are a bit behind schedule, as I hoped to be finished by now, but I don’t think this will take too much longer.

To Do:
– improve voice command accuracy
– integration and recipe processing
– polish web app (just appearance wise)

Posted on

Sumayya Syeda’s Report for 4/20

Progress Update:

This week I went back to work on object tracking. I tried multiple methods including SURF, ORB and SIFT, but found that SIFT worked the best. The biggest difference between my SIFT algorithm from a few weeks ago and this recent change is the use of a perspective transform to get a better boundary around the target object.

I am also working on region detection with the projector. Since we are using a light projector,  the camera will only see the color that is absorbed by the region in the projection at a given time. The projector scans each single channel color one by one and the corresponding color in the projection screen will show in the camera frame. For example, if the image I am looking for in the projection is red and the projector is scanning for red, the camera will see the red image. But if the projector is scanning for green or cyan, the camera will not see the image.

Since the frame rate is consistent and I know when the projector will be scanning for each color, I am able to factor these properties into determining the correct time to capture the frame and see the target image.

Projection as seen by the camera

Image displayed on the table

Schedule: on track!

Next Week:

  • continue to test region detection
  • continue to improve gesture tracking/recognition latency
  • continue to test computer vision with full system

 

As you’ve designed, implemented and debugged your project, what new tools or new knowledge did you find it necessary to learn to be able to accomplish these tasks? What learning strategies did you use to acquire this new knowledge?

 

With this project, I was able to explore new methods in computer vision to detect and track both objects and gestures. It was important to be able to throughly read the documentation from the source libraries, but also be able to follow advice and guidance from public forums. I learned to use various online resources such as YouTube videos, Chat GPT and articles to debug my issues and learn of new techniques.

Posted on

Team Status Report for 4/20

Right now, our risk is meeting our user requirement of allowing this device to work on any table. We are tuning the configuration to the table and lighting set up of the conference room we are using, so it is unknown how well our system will run on other tables. Right now, we are more focused on getting our initial configuration to work, and then we will focus more on generalizing to other tables. If we do not have enough time to do this, we will just reduce the scope of the system, possibly still working on different tables but maybe tables more similar to the one in the conference room. We will also have to tune to the room and table where we are doing the final showcase, which would be the next step.

No changes made to system.

Implementing the software on the AGX was pushed back due to hardware constraints. No other schedule changes.

Posted on

Tahaseen’s Status Report for 04/20

This week, I finalized a camera mount that is set to an appropriate height for our testing environment in HH 1303. Additionally, I decided to use red markers in order to guide the user in outlining the table in the case of projector overflow. Processing these sped up calibration significantly and allowed it to be a more consistent determination of the table outline. The rainbow streaking is a problem we are circumventing by adjusting the selected colors for projection and reducing the overall amount of projection. After these updates and the new camera setup, I have been working on mapping the projection appropriately onto the table so that the homography retains a standard level of brightness. This means playing with the height and angle of the projector further to find an optimal setup.

This coming week, I will finalize my process and complete the integration of calibration into the UI. This will enable us to run everything from one device and one package. In this project, I learned a lot more about calculating homographies and the limitations of a projector system. I also learned how to use Flutter, which I’ve never had to use previously. My main strategy was to rely on forums and data sheets in order to gain a full understanding of the components I was using. For Flutter, I really relied on their online documentation and example apps other people had made using this tool.

Posted on

Caroline’s Status Report for 04/20

Last week, I worked on changing the voice module to use the GPU instead of the CPU. It involved me configuring CUDA on my device and changing function calls in python. I did see better results when the system was running, but it still needs to be tested with a fully integrated system. I focused on moving our design to the AGX. I had many issues this week while installing it but I have been making progress. First, I realized that there was only 32 GB available on the device given to us, so we had to buy an SSD to continue using it. I had attempted to make progress with just the 32GB, but I used so much storage that the device would no longer boot, and I had to reflash. Then, I got the SSD on Friday and started moving our design to it. I had a lot of trouble making Cuda work, but it currently works as of Saturday morning. Cuda worked easily on my own laptop when I tested this week, but it was very difficult for me on the Jetson. I am almost done working through compatibility issues with our code on the AGX as well.

I am a bit behind schedule. I was hoping to have finished installing things on the AGX earlier in the week, but it was pushed back due to the hardware issue. By the end of Saturday, I expect for our code to fully run on the AGX.

The is coming week, I plan to tweak my UI flutter apps to work smoothly with the overall system. It is already mostly implemented, but it will change slightly when we integrate further.

Posted on

Team Status Report for 4/6

Our biggest risk this week is integrating all of our code into the AGX Xavier. We don’t anticipate this to be a terribly big risk , but it is something we have yet to do. It is a high priority this week and we hope to finish integrating all of the current code onto that. This will also give us the opportunity to test the camera with the AGX which we anticipate will give us a significantly higher performance. We’ve already begun working on this process from Friday. We have not had any other changes to our timeline and we anticipate being on track.

Validation:

  • User Acceptance Study: Our primary method of validation will be via the user acceptance study because it will allow us to assess how the design flow suits the user experiences. This study will encompass factors such as ease of UI, taste of product, training time, intuitiveness, etc. This will ensure that we can meet our basic use case requirements.
  • Total Error Rates: Total error rates on the number of hangups, crashes, and general errors that occur throughout the user’s experience are key to understanding how we can process errors better. Part of this is also accounting for cases where error is detected and where it’s not. This will ensure that we can meet our engineering design requirements.
  • Multi-Recipe Functionality: Our system should be compatible with multiple recipes in our library. This will prevent the danger of hard coding to a specific recipe which is not at all what we want. This will ensure we can meet our engineering design requirements and our general use case requirements.
  • Latency: Our latency validation is important to test at the higher level with all of the subsystems working together. Since several of our subsystems are computationally expensive, running them in parallel should not result in a notable latency while the user is interacting with the experience. This is key to our engineering design requirements but definitely impacts our use case requirements.
  • Expected vs. Recorded time: This validation is with regards to the expected time a recipe is marked to take and how much time it actually takes with the user’s ability to interact with the device. This is a major use case requirement because it conveys the actual effectiveness of our product.
Posted on

Tahaseen’s Status Report for 04/06

This week, I focused on putting together the entire pipeline on the actual projector and camera setup. There were a lot of lessons learned during this portion of integration because we learned we needed more stable mounting mechanism and clearer guidance toward the user on setup. Furthermore, when testing the camera with the projector, I realized that the camera saw the projection in the RGB states because of the faster fps. This will likely not be a problem however by grayscaling and thresholding. This is necessary for the auto-calibration process. The USB camera we were testing with has a fisheye effect that our actual camera does not have.  None of these factors will delay our timeline significantly, but this does mean I need to put more bandwidth on manufacturing a more secure mounting for the camera and projector.

Next week, I will manufacture the mounts for the projector and camera. I will also work on speeding up the autocalibration by running some computer vision on the images taken in.

Verification:

  • Projector Calibration
    • Ensure the bleed over onto the floor is within the specified range from the requirements. If not, update table corner detection to identify table bounds.
    • Ensure that the spacing is measured to be the distances specified in the requirements. If not, change table bounds.
  • Hardware
    • Ensure that the hardware mounting for the projector is secure and not susceptible to falling from minor bumps. Verified via checking camera detection of projection difference.
    • Ensure that the hardware mounting for the camera is secure and not susceptible to falling from minor bumps. Verified via checking camera detection of projection difference.
  • Recipe Processing
    • Ensure that the recipe is split into appropriate steps. This will be analyzed from the user acceptance process.