Author: cmarcet

This week I focused on getting the CV working with the web app. I’m able to get it working no longer using OpenCV but tensorflow, since there was a convenient npm package that I was able to download and use that had the functionality that we were looking for. Currently, the web app is ran locally and when ran counts the number of blinks the user performs in a 1 minute interval, which I think is a good demonstration of the functionality that we will need for our final product.

Next week, I would like to get the web app communicating with the browser extension and be able to display the blink count on the extension instead of just printing it to the console. I would also like to create and implement the screen dimming mechanism of the extension, which will automatically dim the tab that the user is currently on if they are in a dark environment. The primary thing I’m concerned with here is how I will get the ambient brightness data from the webcam and how accurate that data will be. I’m on time this week since I was able to figure out the CV and I anticipate the screen dimming will not take long so looking forward if I hope to get that done next week which leaves ample time for the integration with the Ras Pi server which I anticipate will take the most time.

This week I attempted to do the blink detection algorithm and browser extension integration. Last week I was able to get webcam access in the browser extension with the intention of just integrating the blink detection script into the JavaScript of the browser extension. I ran into some issues trying to make this work, specifically Chrome no longer allows remote script to be executed with the release of Manifest V3. I proceeded to install opencv.js locally and uploaded it with the browser extension files but still ran into issues since opencv.js uses eval which Chrome prevents from running even when calling local code. So I’ve abandoned the idea of running the blink detection algorithm via the extension and instead decided to get it up and working with a simple web app then once that is working properly have this web app store the data from the blink detection algorithm in a cookie in the user’s browser and then have the extension read the cookie and display the data.

I definitely wanted to be able to get the blink detection algorithm done this week, but don’t think this makes me behind quite yet. I think if I can’t figure it out next week I will consider myself behind. Ideally I also get the automated screen brightness adjustment feature of the extension done next week too since I should not run into security issues with this because it does not require installing a library like opencv.

This week I worked on making a prototype browser extension. Specifically, I made a Chrome extension which I’m running in Chrome’s developer studio (this is where I’ll be testing it until we are finished then I will upload it to the Chrome Web Store so that others can download it). Currently the extension consists simply of a button which when pressed creates an alert in the user’s system tray. In the finished extension, the alert will occur when the data from the sensors indicate poor posture, but right now I just have the user press the button to demonstrate the system tray notification functionality. In addition, I created a script which allows the user to grant camera permissions in the options menu of the extension. This is important because we need the camera for the CV eye strain algorithm. This unfortunately must occur in the options of the extension and not in the extension itself because this is not an innate function of Chrome browser extensions.

My progress is on schedule. Next week I hope to add the CV script and see if we can run it in the extension (I know that Lilly has the CV script working locally but we have not yet tested it via the extension). I also want to set up the local server on the raspberry pi since we recently got possession of the pi we will be using.

This week I focused on researching the browser extension. The browser extension is going to be used in order to allow the user to easily interact with our project and give them an intuitive UI to view their data and make changes such as toggle options for their camera being on/off and whether sound notifications are on/off.

I researched which browser would be best to make our extension for. I came to the conclusion that Google Chrome would be the best because of the thorough documentation that’s available, the APIs that Google provides, and its straight-forward development process. I researched what type of permissions we will need to ask the user for since the user being aware of what they’re installing is important to us. I wanted to establish the most minimally invasive permissions necessary so we’re not asking for more than what we need and compromise the user’s security. I established we will need at least the following Chrome permissions: audio (for audio notifications), notifications (for on-screen alerts), activeTab, and scripting (in order to adjust brightness of screen). These permissions will be listed in the manifest.json file which is part of the family of files that make up the extension. Note that the user must accept to all these permissions at installation time even if they later choose to disable related features. Otherwise, some permissions can be declared “optional” which is something I would like to discuss with my team to see if this makes more sense for our project.

My progress is on schedule for this week. Next week I hope to have a prototype extension ready in order to test out the basic infrastructure of our extension. I also want to begin with setting up the RasPi hosted server.