Posted on

Liam’s Status Report 4/19/25

Personal Accomplishments

This week I enhanced our gaze estimation accuracy by identifying a 3cm off-center error in our calculations. This discrepancy stems from the stereo camera setup – since we’re using the rectified image from the left camera, we can’t assume the image originated from between both cameras. I also began developing a screen implementation to facilitate integration testing. Additionally, I installed Parsec on my home computer in Missouri to conduct ML training, as I couldn’t find a suitable pre-trained model online. This particular model delivers superior accuracy for screen-based gaze tracking compared to alternatives, making it a low-risk decision that allowed meo to simultaneously progress on other aspects of the project.

If you are interested in how calibration works opencv has a good tutorial:

Here is a picture of me calibrating my webcam:

 

Progress:

I am still working on the gaze estimation onto the chessboard at the same time as doing the screen so I do not impede progress on my team

 

Future Deliverables

Switch to ETHx-Gaze dataset (still)

Switch over to Jetson (still)

Get Screen Estimation working for now

Posted on

Liam’s Status Report 4/12/25

Personal Accomplishments

I was able to accurately determine where a user was located in space by calculating the angle from the camera and knowing the depth at a certain point. I also quickly made a small web app to visually debug the gaze estimation. While testing this out I realized that where the camera was located in comparison to the user played a big role in the estimation. For some depths it becomes hard to calculate gaze direction when looking right or the error seems to very big compared to the true gaze direction. I spent the last couple days reading a lot of research papers and found that there was a better dataset called ETHx-Gaze that has around 5 degrees of angular error compared to around 10 that Gaze360 has. The positional error could decrease from around 5.51 to 2.47 inches. I already have the new dataset working using my webcam so I would just have to integrate it with the stereo camera software.

Progress

If I can’t get the gaze estimation to work more accurately in the next day or two, we will have to consider switching to something like a screen to simplify the problem.

Verification:

Since putting LEDs on the board remains uncertain, I anticipate that verification for my subsystem will require significant manual testing. I’ll need to recruit users or volunteers to simulate gameplay scenarios. The process would involve instructing participants to look at specific squares, then using the previously mentioned web app to confirm their gaze is correctly detected. With LEDs implemented, this verification process could be more automated.

Future Deliverables

Switch to ETHx-Gaze dataset

Switch over to Jetson

Posted on

Liam’s Status Report 3/29/25

Personal Accomplishments

This week, I had a midterm and presentation, so I couldn’t afford to spend as much time as I wanted to on the capstone. There seemed to be a weird issue with the images. I think since this a point cloud image, I have to use the depth numpy array to properly create a normal looking image.

 

Progress

Booth and other classes have put me slightly behind. I will put in a lot of work later tonight and tomorrow to get a proper working MVP for the interim demo.

Future Deliverables

MVP gaze demo

UART for arduino

Posted on

Liam’s Status Report 3/22/25

Personal Accomplishments

I used the zero MQ python library to have the gaze estimation script and the camera script communicate using a queue. Later tonight I’m going to find a screw for the 3d printed stand that we now have. Doing some research on available gaze models I came across some research on github showing how 9 calibration samples could improve accuracy by 5%. This might be something we explore in the future.

Progress

I think I’ve brought myself back to schedule by having the two parts of our software communicate with each other.

Future Deliverables

Cleaning up code and gaze model

Maybe add some training data?

 

Posted on

Liam’s Status Report 3/15/25

Personal Accomplishments

I was able to save the depth and color images as a np array and as an image locally to my computer. Since the stereo camera software only runs on python 3.7 this creates issues with using any type of models since python 3.7 has been EOL for a while. I might explore using it through a HTTP port and having two python programs running at the same time.

 

Progress

I am slightly behind schedule. Being bottlenecked by 3d stand not being printed by techspark.

Future Deliverables

Get the stereo camera working properly with two python programs.

 

Posted on

Liam’s Status Report 3/8

Personal Accomplishments

While working on my laptop instead of the Jetson I was able to modify the sample program given by the manufacturer for using callbacks. I was also able to use a gaze estimation model online to get values for a given frame. I am still not very certain of how the depth is given by the camera but will figure out tonight or tomorrow. Its hard to do proper gaze detection since I have nothing to mount the camera on.

Progress

Currently on track.

Future Deliverables

Proper stand for camera to start doing depth detection when playing the game

Posted on

Liam’s Status Report 2/22

Personal Accomplishments

I helped Trey this week with testing the electromagnets. The new cable also came in, and I was able to properly test it now. I will now be working on creating the POC for gaze estimation. Once we get the gaze working, I will have to 3D print a stand for the camera. The Jetson should be fully configured at this point we just need to write code for the estimation.

Progress

If I don’t finish the proof of concept this weekend I will be worried about staying on track. Besides that I am on track.

Future Deliverables

  • Proof of Concept
  • Stand for camera
Posted on

Liam’s Weekly Report 2/15

 

Personal Accomplishments

This week I worked on setting up the Jetson Xavier as well as researching the proper way to do the gaze estimation. Setting up the Jetson proved to not be as easy as I thought since it can only be flashed from Ubuntu. WSL lets me flash the jetson but not install the toolset. I had to find an ethernet cable to connect the Jetson to the local network to install packages. I discovered a Nvidia toolkit called deepstream on the Jetson that comes with a preconfigured pipeline to use GazeNet for gaze estimation. I am considering buying a wifi m2 expansion card.

I also helped collectively to work on the design presentation.

I migrated our Gantt chart to using Github which allows us to group all the code In the project in one organization

Progress

I am currently on time. We are currently in the process of moving when we wanted an MVP so I will have to make sure I am ton op of my deadlines in the following weeks.

Future Deliverables

  • MVP Pipeline
  • Working Camera
Posted on

Team’s Status Report for 2/8

Risks

One of the most significant risks that could jeopardize the project is getting gaze detection working properly while the user looks at a chessboard. If at any point in the project, we realize that we can’t continue with on-board gaze detection we will switch over to looking at a simulated chess board on a screen. We are developing a proof of concept of this subsystem so that this change can happen as early as possible if this is the case.

Changes

No changes so far since we have not started the process of building the chess assistant. We will implement changes as needed once our actual testing begins.

Gantt Chart

We have created an initial schedule that breaks down our tasks to complete in the first couple of weeks. These include proof of concept work, research, and initial prototyping. No updates have been made to our schedule as of now. The Gantt chart can be found below.

C8 Initial Gantt Chart

Posted on

Liam’s Weekly Report for 2/8

 

Personal AccomplishmentsEarly in the week, I worked on doing preparation for our proposal slides. This week I did research on how to properly do gaze estimation. While trying to do a proof of concept for gaze estimation I realized I would need to know the depth to be able to properly calculate where a user is looking at. Doing research made me realize that I had to get some type of stereo camera that would allow me to calculate the distance from the camera. While doing research, I found that using an Nvidia Jetsen can help simplify our software stack since Nvidia has some models that can only be run on Nvidia hardware. Example of GazeNet(Nvidia)

gazenet.jpg

Progress

I am currently on track. Depending on shipping and whether or not I can get a stereo camera will impact my timeline for creating the POC, which is due next week.

Future DeliverablesCreate proof of concept for gaze estimation

Setup Jetsen