Category: Cora’s Status Reports

Last week I worked on getting the blink rate integrated with the server and the browser extension. I’m running the python script locally which calls the OpenCV library which enables us to collect blink rate. This data is shared via a HTTP request to the server which shares that data with another HTTP request with the browser extension.

This is a picture of the python script running (note that the user will not see this in our final product but it will run in the background). Note the blink rate is at 2.

This is a picture of the browser extension. After requesting to get the blink rate from the server via the “Get Updated Blink Count Value” button, the HTTP of the extension was updated and “2” was displayed.

This week I worked on further integration with Lilly on the browser extension and the neck angle. Additionally, I am currently working on getting the graph display for the blink rate working. I’m running into some issues with using third party code which is necessary in order to make graphs. My solution which I’m debugging at the moment is to sandbox the JavaScript which uses the third party code in order that we can use this code without breaking Google Chrome’s strict security policies regarding external code.

I am on track this week. I think after getting the graph UI working the browser extension will be near finished as far as basic functionality and the rest will be small tweaks and making it look pretty. I’m hoping to reuse the graph code to display neck angle as well so once I get it figured out for the blink rate this shouldn’t be an issue.

This week I worked on the browser extension’s UI, continued integration with Kaitlyn (pressure sensors), and started integration with Lilly (neck angle).

For the browser’s extension UI, I implemented the CSS for displaying the pressure sensor data. The four ovals correspond to the four pressure sensors that we will have on the seat. The server sends the browser extension four binary values, when the binary value is 1, this indicates that the user is leaning too much in this area.

This is what is displayed when the user is sitting in the correct posture (their baseline they set which is their goal posture).

This is just an example of what the circles looked like filled in. When a circle turns red, this means that the user is leaning too much in that direction.

Additionally this week I continued working on integration with Kaitlyn and started integration with Lilly. With the neck sensor data, we were able to get the angle collected via the python code which collects the sensor data -> server -> browser extension.

I’m on track for this week. For the interim demo next week, I’d like to make sure that everything works seamlessly between the CV and the browser extension and make sure that the extension can demonstrate that it can receive data from the pressure sensors and the neck angle sensor.

This week I focused on doing more integration between the Rasp Pi server and the browser extension as well as worked on the screen dimming part of the browser extension.

Kaitlyn and I worked on sending actual pressure sensor data across the network instead of just a simple request. We wanted to see that the data the browser extension was receiving actually changed when changes were made on the pressure sensor and that the latency wasn’t too extreme. We did this by sending the data to the server through the script that collects the data from the Ras Pi, then the server sends this data to the browser extension once it receives a request to do so. The browser extension currently makes this request when I click a button but this is for testing purposes right now and will be done behind the scenes for our actual finished product.

This image shows the pressure sensor data that the browser extension received back after I clicked a button to send a request. Note this is just raw data for now.

Additionally, this week I worked on the screen dimming part of the browser extension. Specifically, I worked on getting the dimming working for multiple tabs since this will be necessary for the final product. Previously, I’d have to open the extension in all tabs if I wanted to dim them, but now I can just do it in one tab and the effect will happen over all open tabs.

  

This is the before.

  

This is the after. Note that I’m adjusting the brightness via a slider in the extension right now but once again this is just for testing purposes and will be done behind the scenes later.

I’m on track this week. Next week I intend on getting the dimming to respond to the ambient brightness of the room (dimming when the room is dark) instead of having to manually do it.

This week I worked on integration between the browser extension and the server hosted by the Ras Pi. I was successfully able to send a simple HTTP request and receive a response. The request is triggered by clicking a button that I added to the extension for now for demonstration purposes, but later this will be behind the scenes.

This picture is before clicking the button and sending the request. I’m sshed into the Ras Pi in the command prompt on the right.

This picture is after clicking the button. The server prints to the console when it receives the request and I inserted a “Hello!” into the http of the extension when it receives a response back from the server.

Additionally, this week I worked on the screen dimming mechanism of the extension. This will be incorporated when the camera detects a dark environment and this will trigger the screen to automatically dim to reduce eye strain. I am on track this week. Next week I’d like to finish the screen dimming and finish incorporating cookies into the CV app so that the web app can communicate with the extension.

This week I focused on getting the CV working with the web app. I’m able to get it working no longer using OpenCV but tensorflow, since there was a convenient npm package that I was able to download and use that had the functionality that we were looking for. Currently, the web app is ran locally and when ran counts the number of blinks the user performs in a 1 minute interval, which I think is a good demonstration of the functionality that we will need for our final product.

Next week, I would like to get the web app communicating with the browser extension and be able to display the blink count on the extension instead of just printing it to the console. I would also like to create and implement the screen dimming mechanism of the extension, which will automatically dim the tab that the user is currently on if they are in a dark environment. The primary thing I’m concerned with here is how I will get the ambient brightness data from the webcam and how accurate that data will be. I’m on time this week since I was able to figure out the CV and I anticipate the screen dimming will not take long so looking forward if I hope to get that done next week which leaves ample time for the integration with the Ras Pi server which I anticipate will take the most time.

This week I attempted to do the blink detection algorithm and browser extension integration. Last week I was able to get webcam access in the browser extension with the intention of just integrating the blink detection script into the JavaScript of the browser extension. I ran into some issues trying to make this work, specifically Chrome no longer allows remote script to be executed with the release of Manifest V3. I proceeded to install opencv.js locally and uploaded it with the browser extension files but still ran into issues since opencv.js uses eval which Chrome prevents from running even when calling local code. So I’ve abandoned the idea of running the blink detection algorithm via the extension and instead decided to get it up and working with a simple web app then once that is working properly have this web app store the data from the blink detection algorithm in a cookie in the user’s browser and then have the extension read the cookie and display the data.

I definitely wanted to be able to get the blink detection algorithm done this week, but don’t think this makes me behind quite yet. I think if I can’t figure it out next week I will consider myself behind. Ideally I also get the automated screen brightness adjustment feature of the extension done next week too since I should not run into security issues with this because it does not require installing a library like opencv.

This week I worked on making a prototype browser extension. Specifically, I made a Chrome extension which I’m running in Chrome’s developer studio (this is where I’ll be testing it until we are finished then I will upload it to the Chrome Web Store so that others can download it). Currently the extension consists simply of a button which when pressed creates an alert in the user’s system tray. In the finished extension, the alert will occur when the data from the sensors indicate poor posture, but right now I just have the user press the button to demonstrate the system tray notification functionality. In addition, I created a script which allows the user to grant camera permissions in the options menu of the extension. This is important because we need the camera for the CV eye strain algorithm. This unfortunately must occur in the options of the extension and not in the extension itself because this is not an innate function of Chrome browser extensions.

My progress is on schedule. Next week I hope to add the CV script and see if we can run it in the extension (I know that Lilly has the CV script working locally but we have not yet tested it via the extension). I also want to set up the local server on the raspberry pi since we recently got possession of the pi we will be using.

This week I focused on researching the browser extension. The browser extension is going to be used in order to allow the user to easily interact with our project and give them an intuitive UI to view their data and make changes such as toggle options for their camera being on/off and whether sound notifications are on/off.

I researched which browser would be best to make our extension for. I came to the conclusion that Google Chrome would be the best because of the thorough documentation that’s available, the APIs that Google provides, and its straight-forward development process. I researched what type of permissions we will need to ask the user for since the user being aware of what they’re installing is important to us. I wanted to establish the most minimally invasive permissions necessary so we’re not asking for more than what we need and compromise the user’s security. I established we will need at least the following Chrome permissions: audio (for audio notifications), notifications (for on-screen alerts), activeTab, and scripting (in order to adjust brightness of screen). These permissions will be listed in the manifest.json file which is part of the family of files that make up the extension. Note that the user must accept to all these permissions at installation time even if they later choose to disable related features. Otherwise, some permissions can be declared “optional” which is something I would like to discuss with my team to see if this makes more sense for our project.

My progress is on schedule for this week. Next week I hope to have a prototype extension ready in order to test out the basic infrastructure of our extension. I also want to begin with setting up the RasPi hosted server.