Month: November 2023

This week my team and I worked on our interim demo. Our demo consists of showcasing our eye-tracking, audio signal capturing and slight-preprocessing, and a test version of the front-end.

I worked on setting up the tobii eye-tracking camera and writing and testing a simple eye-tracking script. Setting up the camera required: installing the Tobii Experience application, setting up the display, and setting up calibration. After executing these steps, the application allowed a pre-gaze feature, which is a spotlight circle that moves around the screen according to where the user is looking at the screen. The test script displays the user’s current eye-gaze position on a windows terminal. This can be clearly seen in this video we shot.

Here, the coordinate system of the screen is (0,0) for top left corner, (1,0) for top right corner, (1,1) for bottom right corner, (0,1) is the bottom left corner.

So far I’ve made decent progress in terms of the eye-tracking, and I am on schedule currently. For next week, I plan to write more scripts to integrate the front-end with the eye-tracking, such as looking at the bottom right section of the homepage redirects to a different html view. I will also look into finding a way to continuously send eye-gaze position to the google board in real time.

This week I spent time working with the google board to have it take in input from the microphone and parse it into chunks that can be processed. This proved to be a bit more difficult because the google board OS did not seem to recognize the lapel microphone as an input. After working with it in the terminal, the microphone is now an unnamed input object that acts as the default microphone. On top of this, because the google board does not have a speaker, I spent time making a way to play the audio files from the google board. This process involves having the file pushed to a git repository and downloaded on a machine with a speaker (i.e my computer). I also spent time trying to adjust the settings for the recording stream to optimize the quality of audio we can get as well as testing different recording conditions to see what kind of noise is also picked up by the microphone. The last thing to do with the microphone is to see what format gives the easiest to work with audio files. For example, a 1MB audio file, although is high quality, will take longer to process and may cause us to not meet the latency requirement.

This upcoming week, I will be looking at different libraries to see which runs faster on the google board. The current library we are using is librosa but might turn out to run slower than PyTorch. In the worst case scenario, we are able to code up the dynamic time warping algorithm from first principles, but it is unlikely to run faster than the prewritten ones.

We are currently still on track and will continue to work hard to stay on track.