Author: aaronl2

This week I worked on preparing for the final presentation, and working on integrating the item recognition component into the web app. I also finalized the distance pruning algorithm and started testing the distance at which items are detected.

For the final presentation, I created a new and updated hardware block diagram to illustrate the changes that have been made since the start of our project.

The main differences are that we are no longer using the built-in Bluetooth adapter, and are instead using 3 external USB bluetooth adapters to measure RSSI values. Additionally, the task of reporting the the item list is now delegated to a separate Python GATT server running separately from the distance pruning algorithm.

As part of integrating the item recognition system, I ran into several issues deploying to AWS. Although our previous commits deployed successfully, the deployment failed as soon as we added Tensorflow to the list of required packages. Initially, I tried to rewrite the Django migration command, as I thought that was the issue from reading the error logs. However, as it turned out, I was reading the old logs from a previous failed deployment. After finding the correct error log, I realized that the AWS server was running into a memory error when installing Tensorflow. After installing Tensorflow manually with the –no-cache-dir directory set, the AWS deployment succeeded.

For this final week, I will be working on editing the final video, and preparing for the final demo. I have picked specific items to use for the final demo, and I will  verify that they are detected in a practice run. Additionally, I am going to create a visualization for the distance data to use in the final report.

 

 

 

This week I worked on deploying the Django web app to AWS, as well as making some changes to the app to streamline transferring item information betweent the Javascript and Python portions of our app. After examining the various available AWS services, including Lightsail and Lambda, we decided to use AWS Elastic Beanstalk (EB) to host the web app. However, getting EB to work with our web app took a lot of experimenting as well as searching online for solutions to our problems.

The first problem I encountered was that EB did not have access to some of the Python packages which our Django app was using. In fact, EB actually does not support the most recent release of Python (3.9), so I had to set up a Python 3.8 environment to test run the web app on. I slowly determined which packages were problematic by looking at the EB environment initialization logs after each failed deployment. I downgraded or removed these packages until the EB environment succesfully deployed. One tip I wish I had known earlier is that the AWS command-line interface only deploys your most recent git commit, not just any changes that have been made to the local files. Thus, any changes have to be committed before the AWS CLI will upload them.

The second problem with our server was that the static files were not being served for the webpage. After reading various blogs about deploying Django apps on EB, I learned that the Nginx server running on the EB instance needs to be configured to serve the static files from a directory. Also, the Django app needs to be told to prepare the static files with the collectstatic command. Using the option-settings feature of EB, I created a configuration file which directed EB to run the necessary commands to prepare and serve the static files using Django and Nginx respectively.

We also learned that the Web Bluetooth API requires https in order to function, for security reasons. In order to make our site secure, I also had to create a self-signed development SSL certificate and upload it to the AWS Certificate Manager. My first few attempts failed due to the certificate being invalid, but after following this blog:

https://medium.com/@chamilad/adding-a-self-signed-ssl-certificate-to-aws-acm-88a123a04301

I managed to succesfully upload a certificate. I then added an HTTPS listener to our AWS EB instance using the certificate, and I also installed the certificate locally on my computer. However, the https still did not work, as modern browsers such as Chrome require that SSL certificates specify the Subject Alternative Name (SAN) field in order to be accepted. After following this Stack Exchange post:

https://security.stackexchange.com/questions/74345/provide-subjectaltname-to-openssl-directly-on-the-command-line

I created a new certificate with the SAN field specified and the certificate was accepted.

After installing the certificate on my Android phone, I was also able to test and verify that our system works on a smartphone. Previously, we were unable to test this functionality because the Web Bluetooth API does not run unless https is enabled.

Schedule-wise, we are falling a bit behind on getting persistent Bluetooth to work, however we should be able to complete that with the extra slack time we have.

For next week, I will be working with Janet on making the Bluetooth connection persistent, so the user does not need to reconnect to check their items. I will also be performing distance testing to determine how well our system gates items inside versus outside the backpack.

This week I worked on improving the distance pruning algorithm, physically mounting the system to the backpack, and testing the system.

After discussing with Professor Sullivan, I decided to test an even longer averaging period of 10 samples to see if that would improve the stability of our system. While it did result in less “wobble” (where the system is uncertain about whether a tag is in the backpack and constantly switches it into and out of the item list), it did increase how long it takes for the item list to be updated after a tag is removed to about 5 seconds. This means that we likely won’t meet our original requirement of 1 second for tag update time, however we feel this is a worthwhile tradeoff for the better stability.

In addition to testing a longer averaging period I also tried using a Kalman filter, based on Wouter Bulten’s webpage at https://www.wouterbulten.nl/blog/tech/kalman-filters-explained-removing-noise-from-rssi-signals/. I tested different values for the process noise and gain, however I found that the Kalman filter was not as stable as the direct averaging was. Thus, I decided to stick with averaging instead of a Kalman filter.

For the mounting, I used the backpack’s existing pockets to hold the battery and the RPi Zero, and sewed the Bluetooth adapters to different points in the backpack spaced 20cm apart. I then carried the device on my backpack and walked around the neighborhood to test if the system could withstand movement, which it did. I also tested shaking the backpack to see if any of the tags inside would be undetected, but the system still detected all 10 tags inside of the backpack.

In addition to testing the system physically, I also ran battery tests throughout the week. I ran the system at it’s maximum power draw level, by having it track 10 tags while it was also reporting the tag list with the Bluetooth GATT server. Over 5 tests, the battery lasted for an average of 19.5 hours, with no test going under 18 hours. Thus the battery life meets our requirement of 18 hours of continuous use.

 

This week I worked on adding support for multiple Bluetooth adapter sensors to the Raspberry Pi Zero’s script. To accomplish this, I had to modify the script to support multi-threading, with each thread registering a Bluetooth scanner with each of the Bluetooth USB adapters. In addition to adding these threads, I had to modify the distance pruning script to record and utilize multiple RSSI readings from each of the Bluetooth adapters, as opposed to using just a single RSSI value. To do so, I created a new Python class for handling items which stores the RSSI values and automatically adds itself to the item list according to the distance pruning algorithm. Additionally, the class handles dropout as well in case the beacon signal is lost.

As part of the distance pruning algorithm, we tested a smoothing coefficient to average out the reported RSSI values over time for better accuracy. However, from testing on the borderline distance where the beacons would be considered outside of the backpack (50cm), the smoothing did not appear to help prevent the beacon from “stuttering” (jumping in and out of the list of items considered to be inside the backpack).

A new idea we have for helping with accurately identifying which objects are in the backpack is to use a light-level based power switching circuit on each of the beacon tags, so that the beacons will only be active under dark lighting conditions. This would make it so that only the tags inside of the closed backpack would begin advertising, while items outside of the backpack would be exposed to the ambient room lighting and therefore remain dormant. To test the feasibility of such a solution, I soldered wires to the power traces of the beacon tags, and found that the tags automatically began advertising again after receiving power. Thus, it would be possible to control the beacon tags with an additional light-sensing circuit on the outside. However, a new enclosure would have to be produced for the tags, as such a light-sensing circuit would not fit inside the original casings of the tags.

In addition to working on the RPi-Tag interface, I also worked on the RPi-Web app interface. Although we were able to send data to the web app last week, we had not been able to send a full item list. This week, I modified the GATT server to report the item list in string (character-by character) format, with each item’s mac address added to the string, separated by a semicolon. This also required changing the GATT server’s characteristic type from integer to string. On the web app side, I modified the Javascript script to be able to continuously handle update events from the RPi, as opposed to the one-off data transfer the original script from last week did. This also required modifying the script to request notifications from the RPi GATT server, and also modifying the GATT server to periodically report the item list to the web app upon receiving the nofications request.

For next week I plan on exploring the light-sensing switch circuit concept further, by ordering photoresistors and transistors and testing how well the tags can be switched on and off. I also plan on mounting the RPi system inside a backpack, and testing whether the system can withstand the physical movement associated with normal use of a backpack.

 

This week I worked on interfacing the Raspberry Pi with the web app over Bluetooth. To accomplish this, I configured the Raspberry Pi to run a Bluetooth GATT Server to which the web app could connect to by Bluetooth pairing. After connecting, the GATT service on the Raspberry Pi presents the list of items ID’s in string format for the web app to read.

The code for the GATT server is written in Python, and it uses the Bluez package’s BLE advertisement example code to run. I used the CPU temperature reporting example written by Douglas Otwell as inspiration for the code. The GATT server works by presenting a reporting “service,” which the web app can connect to via the service’s UUID. The service can then send data to the web app by exposing “characteristics” which can also be read via the characteristic UUID. We haved tested this functionality and have successfully transferred data between the Raspberry Pi and web app.

In addition to working on the app-RPi Bluetooth interface, I also continued to work on improving the RPi-tag interface. With the arrival of a dedicated USB Bluetooth dongle, I began additional testing to determine it’s sensitivity and to compare it with the built-in Bluetooth antenna. Although the reported RSSI values were lower, the USB Bluetooth adapter was still able to detect the iBeacon tags within the necessary range for our purposes.

One issue I encountered with the USB Bluetooth adapter is that the onboard Bluetooth controller is actually disabled if the OS detects a separate USB adapter. Despite various attempts at changing this behavior, including modifying configuration files and disabling certain services, I was unable to stop the built-in adapter from disabling itself. As a result, we have decided not to use the built-in adapter to determine tag distance. Instead, we are opting to use a total of 4 external USB Bluetooth adapters space apart in the backpack to determine the tag location. Having 4 RSSI readings will help improve the accuracy of the distance pruning algorithm, and make it less likely that an item outside of the backpack will be detected as being inside.

For next week I plan on finalizing the RSSI calibration values for the new USB Bluetooth adapter, as the previous values I had for the built-in adapter no longer apply to our system. I also plan on adding time-out functionality so that if a tag is not detected for long enough it is removed from the item list, even if its distance is unknown.

This week I worked on setting up and configuring the H1 iBeacon tags, connecting to them using the Bluez stack, and doing testing to determine the calibration values needed for our distance pruning algorithm. The manufacturer of the beacons, Mokosmart, provided a smartphone app to connect to the iBeacons and configure their transmission power and interval.

Although our initial plan was to set the tranmission power to the lowest setting available, this power setting actually made the Bluetooth signal too weak to be detected by the Raspberry Pi. Additionally, the lower power transmission levels also had higher variance in the RSSI values reported at 50cm. After testing different power transmission levels, we found that level 4 (-12dBm) was a suitable balance between power consumption and detection reliability.

For testing, I used the aioblescan Python package created by François Wautier. We initially tried using the bluepy package created by Ian Harvey, however we discovered that the bluepy package could not achieve the 1 second update interval required for our project. We switched to the aioblescan package, as it is specifically designed for reading BLE advertisement packets, and it therefore has a sufficiently fast update interval for our needs. Using the package, I wrote a simple script to read in 1000 advertisement packets, and calculate the mean and standard deviation of the RSSI values reported. I then placed a beacon exactly 50cm from the Raspberry Pi, and ran the script. Below is a sample screenshot of the testing loop results.

I also tested 5 different beacons to measure the variation across different beacons. All of the mean RSSI values reported were within +/- 3 dBm of each other, indicating that the different transmission powers of the beacons were within tolerance of each other.

To increase the robustness of our system, we decided to order an additional Bluetooth receiver for the Raspberry Pi. This would allow us to obtain two distinct RSSI values, which would further increase the accuracy of our distance pruning algorithm. By placing the Raspberry Pi and additional receiver at opposite ends of the backpack, we can reduce the effective volume where a tag could be and still be within 50cm of both receivers. This helps mitigate the issue of items near the backpack being mistaken for being inside the backpack.

For next week, I plan on testing the ability of the system to work in a backpack, by attaching the tags to items and placing the Raspberry Pi in an actual backpack. I will also be working on an improved algorithm using two RSSI values instead of one, in preparation for the arrival of the additional Bluetooth Receiver.

This week I worked on the design review presentation and design report. I completed both an overall system diagram and a software-specific block diagram for our project.

I also began creating a bill of materials from our parts list. This budget list includes everything we currently need for our project. We also included a Tile tracker tile in order to compare our system to existing products.

Following feedback given after the design review presentation, I have also begun rethinking our distance-based gating requirement. Rather than having a fixed algorithm, I am working on a calibration protocol which would allow the system to more accurately determine tag distance by having the user place the tags at a specific distance from the Raspberry Pi and measuring the signal strength.

For next week I plan on getting the parts ordered and finishing the design report. I plan on adding variable names to the software specification diagram as well, to clarify what data is being sent between the different components of the phone app.