Category: Team Status Reports

Currently, the most significant risk to our project is that our integration with the image recognition portion of our project is still not completed, as the endpoint for the model must first be finished. This is something that will likely be completed in the following days, but perhaps not in time for the poster or video. To manage this risk, Joon is working ASAP on setting up an endpoint for the model, and Janet will help with integration into the web app once the endpoint is set up. However, even without the integration of the image recognition portion into our project, our product is still able to achieve our intended use case. The only consequence of failing to integrate the image recognition portion is that users will need to manually enter the name of items when registering them, rather than our model suggesting three names.

No changes were necessarily made to the existing design of our system (in our requirements, block diagram, or system specs), but we will no longer be pursuing the creation of a thin Android app client for Bluetooth persistence for our project. This change was necessary simply because we ran out of remaining time for our project to implement this, and extensive research would have been required before we could begin to implement it. The cost that this change will incur is that users would spend more time needing to connect to the Raspberry Pi every time they open the app. As we are currently at the end of the semester, unfortunately, we have no time to mitigate this cost.

Our schedule can be viewed here.

 

This week we succesfully managed to deploy the web app to AWS as well as test that our app works on an Android phone. Getting the app to work on a smartphone versus a computer was one of the risks we had, so this risk has been eliminated now that we know the app works on a smartphone.

The most significant risk we still face is getting persistent Bluetooth to work, as this would reduce the time the user spends needing to connect to the Raspberry Pi every time they open the app. Currently, we are working on creating an Android applet, which will maintain the Bluetooth connection while the web app is closed. In addition to this risk, our other risks include integrating the item recognition into our app, as currently we are running the web app and the ML item recognition on two different servers, and getting the servers to communicate may be difficult.

As for design changes, we have decided not to implement the sleeping protocol for the Raspberry Pi, as we have determined that even under the most demanding conditions the battery life is sufficient to run the system for at least two 8-hour work days, or one 18-hour day. Additionally, even if there are no events scheduled, the user may want to check their items arbitrarily, and they would not be able to do so quickly if the system is asleep given the long bootup time of the Raspberry Pi.

The only major change to our schedule is that we are increasing the time for the persistent Bluetooth problem. As always, our schedule can be viewed here.

The most significant risk that could jeopardize the success of our project is the Bluetooth persistence issue. Currently, our web application utilizes the Web Bluetooth API to connect with our RPi Zero and receive the list of items inside the backpack. However, we first need to deploy our web app (we are using AWS Elastic Beanstalk) before testing on a phone, as the Web Bluetooth API only works in secure contexts. Overall, this risk is being managed by our current contingency plan, which is to create a thin client on Android which handles the Bluetooth functionality, while including our Web App in a <WebView> in the Android application to handle the interfacing. However, we expect to run into many issues, particularly in the communication between the thin client itself and the web application. We’ve dedicated the upcoming week to implementing this transition and debugging any issues we run into.

No changes have been made to the existing design of the system. As always, our schedule can be viewed here.

Currently, the most significant risk to our project is our ability to accurately determine which items are inside versus near the backpack. To mitigate this risk, we are exploring a new idea of light-based switching of the beacon tags. The concept is that tags inside of the backpack will not be exposed to light (assuming the backpack zipper is closed), whereas a tag next to the backpack would still be exposed to the ambient lighting within the room. Another mitigation we are using is a notification to the user to check that there are no items next to or near the backpack when they are about to leave for an event. By having the user check for such items, the system will not need to be able to distinguish between close to versus inside the backpack.

Another risk we have discovered is that the usage of 4 bluetooth adapters for the scanning process has caused the Raspberry Pi Zero to slow down due to processor overload. To mitigate this risk, we have decided to use only 3 out of the 4 adapters for our scanning process, which will lighten the burden on the RPi Zero. Additionally, we are increasing the timeout update intervals from 500ms to 1s to further reduce the load. Although this increases the time between when a beacon’s signal is lost to when it is removed from the item list, this tradeoff is necessary to ensure that the RPi Zero can still operate without slowing down.

The only changes we’ve made to the existing design of the system involve using 3 rather than 4 adapters for our scanning process, for reasons discussed above.

Our updated schedule can be found here.

A video demo of our interface update from BLE tags can be viewed here.

Currently, the most significant risk that could jeopardize the success of the project is the integration between the web application that visualizes the list of user items inside the backpack and the Bluetooth reader that determines whether the tagged item is inside the backpack. Moreover, because we have the interim demo coming in two weeks, we need to demonstrate that this functionality is fully working to ensure our success for the project. To manage these risks, we are doing extensive research on the Web Bluetooth API and communicating with Bluetooth devices so that we can quickly resolve problems that that arise during integration, both from solutions found by other users published online or alternative implementations that are available, such as having an additional Bluetooth reader. In preparation for integration, each of us have been testing each component incrementally and individually. For the web application, Janet has been testing the web application using dummy items and information so that communication with the Bluetooth scanner is not needed. For the Bluetooth scanner part, Aaron has been testing the Bluetooth scanner independently to check the robustness of the tagged item detection. For the item recognition part, Joon has been testing the CNN model on his machine so when the test is correctly done, the integration is done with the item registration part of the web application.

There is a change made to the existing design of the system. As stated in Aaron’s Status Report for 4/3, we have found that the onboard Bluetooth controller is actually disabled if the OS detects a separate USB adapter. Therefore, we have decided not to use the built-in adapter for the distance pruning of the tagged items. Instead, we have decided to use an array of 4 external USB Bluetooth adapters to determine the tag location. While this will require more of our budget to be used and require more power consumption, we believe that this will enhance the accuracy of our distance pruning algorithm and is worth the tradeoff.

The schedule for the item recognition part has been changed to account for the training of the CNN model for the item recognition and the testing plan for the CNN model is pushed back (but before the interim demo) to account for the integration with the web application and the interim demo. Additionally, we have added more time for Aaron to incorporate the additional Bluetooth adapters, and have pushed back implementing the power-saving sleep feature. The updated schedule is found here.

Currently, the most significant risks that could jeopardize the success of the project include losing any native functionality of what we had planned for the phone app. Originally, we planned to build our user-facing app interface as a native Android app in Android Studio with Kotlin. However, after running into many difficulties with collaboration and build errors, we agreed to switch to developing a web application instead, with the same functionality as planned. However, the largest risk is losing any persistent functionality (including receiving notifications or receiving Bluetooth data when the web app is not active). To manage these risks, we are doing extensive research on the Web Bluetooth API, which allows us to communicate over Bluetooth using Javascript.  Additionally, Android mobile devices do support mobile web push notifications. As a mitigation, should our team still run into any issues, we plan on creating a bare native Android application in Android Studio + Kotlin that handles any persistent native functionality such as notifications and Bluetooth communication and placing our web application as a WebView into the phone app so that the user may still interact with the interface we’ve built.

Changes were made to the existing design of the system, as discussed above. We’ve changed our user-facing application from a native Android app to a web application. This change was necessary due to persistent build (Gradle) errors and issues with collaboration in the Android Studio IDE.  Costs this change incurs may involve a possible loss of native functionality (as discussed above), but these costs will be mitigated by the possibility of handling any bare native functions like notifications and Bluetooth through either the Web Bluetooth API + mobile web push notifications or a bare Kotlin app that handles these functionalities but displays our web application interface through a WebView.

No changes have been made to our schedule, but it can be viewed here.

 

One of the current risks to this project is the ability to accurately determine the distance of the BLE tags to the Raspberry Pi. Different tags have different power emission levels, and the scanner on the Raspberry Pi also might be weaker than the scanners used in iPhones for which the iBeacon protocol was designed. To mitigate this risk, we are creating a calibration protocol for the tags, whereby the user will place the tag next to the Raspberry Pi at a fixed distance to allow the system to correlate signal power level with distance.

Another significant risk to this project involves the difficulty of Android Studio, particularly when it comes to Git Integration and common build errors. This may make integrating our modules of the project together much more difficult. One way we are managing this risk is to consistently test version control changes between members on Android Studio. Another possible mitigation for this risk is defaulting to a mobile-accessible web app, which may prove more useful to our project and its users.

No major changes have currently been made to the existing design of the system.

Our current schedule can be found here, though Joon has made changes in his schedule on the item recognition part due to more refined series of tasks developed during the design presentation and design report; our team members are currently focusing mostly on the upcoming design report.

Currently, the most significant risk for this project is finding a reasonable and robust testing plan for the learning system where the system will learn the user’s habits (which items they bring at which times of day on which days), as we can’t afford to wait 2 weeks for our system to learn habits each time we’d like to test the feature. One possible solution is to immediately begin collecting data from one of the team members as soon as the tags and integration with the device is done (i.e. one team member will tag and register all their items and bring their items with them from event to event for a period of time). We can also come up with carefully constructed synthetic data for testing purposes. Contingency plans we have ready include allowing the user to see their schedule as it is being developed and learned by the system so that the user can correct any mistakes our learning system makes.

Some changes were made to the existing design of the system based on feedback we received from others after our proposal presentation and more team discussion. The biggest change to our project includes the learning system, where we can learn a user’s habits after a period of time to craft their schedule and item notifications. This change was necessary because, as Professor Kim suggested, the user burden might be too large to require them to create events or sync their Google Calendar and assign items to each event. However, this change imposes additional strain on the user, as users may have to “wait” for their habits to be learned. For example, during the two weeks where our product learns about the user’s item habits, they could have already been able to use our system to help track and manage their assets. As a mitigation, the learning system will only be optional for users, in case they prefer to sync with Google Calendar or create events themselves.

We’ve currently made some tentative changes to our schedule to accommodate the learning system.

User flow diagrams and wireframes have been made for our phone app, which can be seen below. Additionally, our team has also decided on the most important items for item recognition as well as the hardware we need to purchase.

 

Currently, the most significant risk that could jeopardize the success of the project is the selection of which BLE tags we choose to purchase. Certain tags require more work to be programmed, and we need to keep in mind both our budget and the scope and requirements of our project when purchasing tags. We’re currently managing this risk by creating a pro/con list of possible tag purchases and keeping room in our budget for multiple tag purchases, should some of them not meet our needs.

One change we made to the existing design of the system was choosing to include the phone’s camera to streamline easier item registration using computer vision. This change was necessary because the user burden of initial item registration was brought up as a concern by our project advisor, Professor Kim, and we agreed that having to manually register each tagged item might prove too big a burden for most users (particularly if they have many items). The costs that this change would incur include placing a larger performance burden on the user’s phone and adding much more complexity and risk to our project. We hope to mitigate these costs by finding relevant work related to item recognition and are prepared to default to manual item registration, should this aspect of our project not succeed.

Our current schedule can be found here (slack time and tasks are currently not finalized).