Posted on

Aaron’s Status Report for 5/8

This week I worked on preparing for the final presentation, and working on integrating the item recognition component into the web app. I also finalized the distance pruning algorithm and started testing the distance at which items are detected.

For the final presentation, I created a new and updated hardware block diagram to illustrate the changes that have been made since the start of our project.

The main differences are that we are no longer using the built-in Bluetooth adapter, and are instead using 3 external USB bluetooth adapters to measure RSSI values. Additionally, the task of reporting the the item list is now delegated to a separate Python GATT server running separately from the distance pruning algorithm.

As part of integrating the item recognition system, I ran into several issues deploying to AWS. Although our previous commits deployed successfully, the deployment failed as soon as we added Tensorflow to the list of required packages. Initially, I tried to rewrite the Django migration command, as I thought that was the issue from reading the error logs. However, as it turned out, I was reading the old logs from a previous failed deployment. After finding the correct error log, I realized that the AWS server was running into a memory error when installing Tensorflow. After installing Tensorflow manually with the –no-cache-dir directory set, the AWS deployment succeeded.

For this final week, I will be working on editing the final video, and preparing for the final demo. I have picked specific items to use for the final demo, and I will  verify that they are detected in a practice run. Additionally, I am going to create a visualization for the distance data to use in the final report.

 

 

 

Posted on

Joon’s Status Report for 5/8

This week, which was the last week of classes, we had a final presentation and presented what we have done during the semester and our final plans before the final demo. We also worked on the final poster and final video for next week’s deliverables.

For the item recognition part, last week I decided to have as many epochs (50 epochs) to train. However, training as many epochs to train was a bad idea because the model started to get overfit. In other words, the model learns the details of the outliers and the noise in the training dataset, therefore resulting in lower classification accuracy. I went back with the range of 15-22 epochs, as that was the range of the epochs where the validation accuracy started to converge. After the test, I was able to achieve the increased accuracy of 88.08% with 18 epochs.

I also have tested the model with the images of Aaron’s student items because for the final demo, Aaron is the one who will be physically demonstrating the overall system, including the item registration. The most noticeable result was that among 21 student item labels, it was able to classify the water bottle with 100.0% accuracy. We can also see that the notebook gets correctly classified.

    

I also have tested the picture of Aaron’s pencil case with his hand taken and without his hand taken, which is a possibility when a user takes a photo when registering an item. The picture with the hand was correctly identified as an image to be a pencil case, but the picture without the hand was incorrectly labeled as a wallet. But, identifying the pencil case as a wallet is fine because both items are “pouch-like objects” which makes sense to classify a pencil case as a wallet and vice versa. To take account of this issue (which also applies to rectangular objects like a notebook, laptop, etc), we have decided previously to show the top 3 classification results, and this result is shown at the top of the image as a Python list. As we see, the result has [‘wallet’, ‘sports equipment’, ‘pencil case’], which also contains the correct result, ‘pencil case’. Thus, even though the image was incorrectly identified (if we only consider the top 1 result), the user can benefit from simply choosing from the top 3 classification results.

 

Although I have increased the accuracy and made sure that my model works with Aaron’s student item images, my progress on integrating this item recognition part to the web application is slightly behind because of the Tensorflow and Keras modules that throw an error to the deployment of the web application. Thus, to have enough time to integrate and test the deployed web application, I am more leaning towards the backup plan, which is to train the model in the local machine (of a team member doing the final demo) and have the model to communicate with the web application locally. To do so, I am working extensively with Aaron, who is responsible for the deployment, and the web application, and asked Janet to enable the image form feature, where the user can input the image for the item recognition.

For next week, I plan to integrate this model into a web application. Then, we will test the deployed web application and see how it is working with the overall system. Finally, our group will prepare for the final demo and work on the final report.

Posted on

Team Status Report for 5/8

Currently, the most significant risk to our project is that our integration with the image recognition portion of our project is still not completed, as the endpoint for the model must first be finished. This is something that will likely be completed in the following days, but perhaps not in time for the poster or video. To manage this risk, Joon is working ASAP on setting up an endpoint for the model, and Janet will help with integration into the web app once the endpoint is set up. However, even without the integration of the image recognition portion into our project, our product is still able to achieve our intended use case. The only consequence of failing to integrate the image recognition portion is that users will need to manually enter the name of items when registering them, rather than our model suggesting three names.

No changes were necessarily made to the existing design of our system (in our requirements, block diagram, or system specs), but we will no longer be pursuing the creation of a thin Android app client for Bluetooth persistence for our project. This change was necessary simply because we ran out of remaining time for our project to implement this, and extensive research would have been required before we could begin to implement it. The cost that this change will incur is that users would spend more time needing to connect to the Raspberry Pi every time they open the app. As we are currently at the end of the semester, unfortunately, we have no time to mitigate this cost.

Our schedule can be viewed here.

 

Posted on

Janet’s Status Report for 5/8

This week, I completed writing and implementing the Schedule Learning feature for our web application. The schedule learning feature makes use of two new models: a TimeItemListPair and an ItemCounter.

Multiple TimeItemListPairs essentially function as a dictionary, with each pair tied to a user and a timestamp with a list of items currently inside the backpack at that specified timestamp. An ItemCounter exists for each item tied to a specific event (e.g. there could be multiple ItemCounters for the same item, “Folder,” if Folder is assigned to more than one event) and keeps track of how many times this item, for this event, has been found in a TimeItemListPair.

For the assignment logic, our system runs through all the events for a user that are marked as “Schedule Learning” events. If a TimeItemListPair exists such that its timestamp falls within the window of an event (i.e. between the start time and end time of an event), the counter for each item in that Pair will increment by one. Finally, assignment continuously happens, assigning items with counts higher than 85% of all counts to the corresponding event.

Although this feature is not completely fleshed out, schedule learning is not included as part of our MVP, and is simply meant to help decrease the user burden of manually assigning items to each event.

Additionally, I completed an extra feature to add a “Register” button to unnamed items inside a user’s backpack for streamlined registration. This feature can be viewed here.

This past week, I also worked with my teammates on the Final Presentation slides.

Our progress is currently on schedule.

In the next week, our remaining focus will be on completing deliverables for the end of this project course, including the final video, poster, and report.

Posted on

Joon’s Status Report for 5/1

This week, we had the last mandatory lab meeting with Professor Kim and Ryan. We have discussed the updates on our progress since the last meeting and our goals for the final presentation and the report.

For the item recognition part, I was able to increase the accuracy of the model by training for longer epochs. Instead of 15 epochs, where the validation accuracy started to converge, I trained for 50 epochs and it increased the classification accuracy to 84.36%. I believe that increasing the number of epochs is fine because the model is eventually trained onto the AWS server and in the web application, the user can simply input the image to the server and get the classification results from the model in the AWS server. For the reporting purposes of the testing component of the CNN model, I’m planning to not only present the accuracy percentage but also show the confusion matrix, which shows an overview of the classification of given labels to the correct labels.

In order to integrate the image recognition component into Janet’s web application, my main goal for the remainder of the semester is to provide an API endpoint for the model. I have been setting up the model (along with the datasets and the trained feature onto the AWS server using Amazon Sagemaker.

Although the development and the testing for the item recognition part are done, my progress on integrating this item recognition part is slightly behind because I need to provide the API endpoint for the web application and the item recognition module. Since the web app and the ML item recognition are running on two different servers and getting the servers to communicate to each other may be difficult, my backup plan is to train the model in the local machine (of a team member doing the final demo) and have the model to communicate with the web application locally. Then the top 3 recommendations can be transferred and displayed to the web application. To catch up on this, I have to work concurrently with providing an API endpoint between two servers (ML server and web application server) and working on the backup plans.

For next week, I plan to integrate this model into Janet’s web application and provide an API endpoint for the server. Finally, I have to check whether the recommendation is displayed correctly on the web application. Our group will also work on the deliverables for the finals week of this course.

Posted on

Team Status Report for 5/1

This week we succesfully managed to deploy the web app to AWS as well as test that our app works on an Android phone. Getting the app to work on a smartphone versus a computer was one of the risks we had, so this risk has been eliminated now that we know the app works on a smartphone.

The most significant risk we still face is getting persistent Bluetooth to work, as this would reduce the time the user spends needing to connect to the Raspberry Pi every time they open the app. Currently, we are working on creating an Android applet, which will maintain the Bluetooth connection while the web app is closed. In addition to this risk, our other risks include integrating the item recognition into our app, as currently we are running the web app and the ML item recognition on two different servers, and getting the servers to communicate may be difficult.

As for design changes, we have decided not to implement the sleeping protocol for the Raspberry Pi, as we have determined that even under the most demanding conditions the battery life is sufficient to run the system for at least two 8-hour work days, or one 18-hour day. Additionally, even if there are no events scheduled, the user may want to check their items arbitrarily, and they would not be able to do so quickly if the system is asleep given the long bootup time of the Raspberry Pi.

The only major change to our schedule is that we are increasing the time for the persistent Bluetooth problem. As always, our schedule can be viewed here.

Posted on

Aaron’s Status Report for 5/1

This week I worked on deploying the Django web app to AWS, as well as making some changes to the app to streamline transferring item information betweent the Javascript and Python portions of our app. After examining the various available AWS services, including Lightsail and Lambda, we decided to use AWS Elastic Beanstalk (EB) to host the web app. However, getting EB to work with our web app took a lot of experimenting as well as searching online for solutions to our problems.

The first problem I encountered was that EB did not have access to some of the Python packages which our Django app was using. In fact, EB actually does not support the most recent release of Python (3.9), so I had to set up a Python 3.8 environment to test run the web app on. I slowly determined which packages were problematic by looking at the EB environment initialization logs after each failed deployment. I downgraded or removed these packages until the EB environment succesfully deployed. One tip I wish I had known earlier is that the AWS command-line interface only deploys your most recent git commit, not just any changes that have been made to the local files. Thus, any changes have to be committed before the AWS CLI will upload them.

The second problem with our server was that the static files were not being served for the webpage. After reading various blogs about deploying Django apps on EB, I learned that the Nginx server running on the EB instance needs to be configured to serve the static files from a directory. Also, the Django app needs to be told to prepare the static files with the collectstatic command. Using the option-settings feature of EB, I created a configuration file which directed EB to run the necessary commands to prepare and serve the static files using Django and Nginx respectively.

We also learned that the Web Bluetooth API requires https in order to function, for security reasons. In order to make our site secure, I also had to create a self-signed development SSL certificate and upload it to the AWS Certificate Manager. My first few attempts failed due to the certificate being invalid, but after following this blog:

https://medium.com/@chamilad/adding-a-self-signed-ssl-certificate-to-aws-acm-88a123a04301

I managed to succesfully upload a certificate. I then added an HTTPS listener to our AWS EB instance using the certificate, and I also installed the certificate locally on my computer. However, the https still did not work, as modern browsers such as Chrome require that SSL certificates specify the Subject Alternative Name (SAN) field in order to be accepted. After following this Stack Exchange post:

https://security.stackexchange.com/questions/74345/provide-subjectaltname-to-openssl-directly-on-the-command-line

I created a new certificate with the SAN field specified and the certificate was accepted.

After installing the certificate on my Android phone, I was also able to test and verify that our system works on a smartphone. Previously, we were unable to test this functionality because the Web Bluetooth API does not run unless https is enabled.

Schedule-wise, we are falling a bit behind on getting persistent Bluetooth to work, however we should be able to complete that with the extra slack time we have.

For next week, I will be working with Janet on making the Bluetooth connection persistent, so the user does not need to reconnect to check their items. I will also be performing distance testing to determine how well our system gates items inside versus outside the backpack.

Posted on

Janet’s Status Report for 5/1

This week, I completed the features for missing item notifications (i.e., an event is coming up but at least one of the required items for the event is not currently in the backpack) as well as lost item notifications (i.e., the item list at the beginning of an event is not consistent with the item list at the end of an event). A demo of this feature can be found here [with no missing items] and [with missing items]. Two new functions were written for these new features, which can be viewed below:

Additionally, since our team members are all working to finish the remainder of our project, I picked up the schedule learning feature task for our web application. Currently, I’ve collected sample data for what our web application may need for this feature, which consists of large dictionaries with timestamps as keys and an item list snapshot of what items are inside the user’s backpack at that timestamp. This data would then be continually compared against events stored in the database and items would be automatically assigned to the user’s events based on this data. However, this feature is not a requirement of our MVP, and a fully fleshed-out version of this feature would require persistence (thin native client + webview) to be completed first. I’ve also started working with my teammates on our final presentation.

Currently, our schedule is a little bit behind. The components left involve:

  1. Persistence (implementation of thin native client in Android + WebView)
  2. Schedule Learning (not part of our MVP, but dependent on persistence)
  3. Integration with image recognition component (currently depending on Joon to set up an API endpoint for the model)

Save for the integration with the image recognition component, our MVP is already completed. However, implementing features like persistence and schedule learning would greatly increase the cases in which our product can be most useful as well.

To catch up to the project schedule, we’ll be using this final week of development (week of 5/2) to complete the remaining components. Aaron and I will work on Persistence, I will work on Schedule Learning, and Joon will work on setting up an API endpoint for the model. These are the deliverables we hope to complete in the next week.

Posted on

Joon’s Status Report for 4/24

This week, we had an additional meeting with Professor Kim because there weren’t any mandatory lab meetings due to the ethics discussion, and we also wanted to inform him about our progress after the interim demo.

For the item recognition part, I had to increase the item classification accuracy. For the interim demo, the previous model I implemented had 58% item classification accuracy. However, after the discussion during the interim demo, I have realized that I needed to increase the recognition accuracy by a significant amount. While I initially thought that the 60% accuracy was acceptable (and 50% accuracy for the MVP), as the user can manually type the item information whenever the wrong item suggestion is given, I also agreed with the fact that higher accuracy is desirable to decrease the user burden. I had to completely change the model and the better model that I was able to find was the VGG16 CNN model (For more information of VGG16 model: VGG16 Paper and Blog post). This model was provided by the Python Tensorflow and Keras, so I coded and trained the newly implemented VGG16 model. I also had to change the dimension of the images from 256 x 256 to 224 x 224 because the initial convolution layer takes in the 224 x 224 image and the 224 x 224 dimension is widely used in machine learning models for image classification. Another step I also took to increase the accuracy was to have a validation set. With the validation set, I could find the best hyperparameter, which was the number of epochs to train the model. I stopped the number of epochs to train when the validation accuracy and validation loss gets converged. Among 50 epochs, which took way long time, I was able to find the accuracy got converged around 15 epochs, so I stopped training there.

After implementing the model, I was able to increase the accuracy to 80.21%, which was much better than the planned accuracy. For instance, whenever a user inputs an image of a laptop, it correctly identifies that the image is a laptop, with 99.52% accuracy. For visualization and demonstration purposes, I have printed out all 21 labels and the classification likelihood for each label in percentage. I also took a picture of my wallet, and it correctly identifies that the item is a wallet, with 78.09% accuracy. This was good to see because my model works well for the images taken from a user’s smartphone. Also, among 21 labels, there are many rectangular objects such as laptop, notebook, tablet, and textbook, but it can correctly classify that the image is a wallet. Moreover, the item classification returns a top 3 item classification suggestion so that the user can simply choose the item classification from these 3 suggestions. These suggestions are returned in a form of a Python list so that they can be easily transferred to the Django web application.

My progress on the item recognition part is slightly behind because I need to integrate this item recognition module into the web application. To catch up on this, since Janet’s done with her individual work on the web application, I need to work extensively and collaborate with her during the lab session to fully integrate and test the smooth integration of the item recognition module.

For next week, I plan to integrate this model into Janet’s web application. To do so, I have to train my model to the AWS server and look into the methods to integrate the CNN model into the Django framework. In the later part of next week, our group will also work on the final presentation slides.

Posted on

Team Status Report for 4/24

The most significant risk that could jeopardize the success of our project is the Bluetooth persistence issue. Currently, our web application utilizes the Web Bluetooth API to connect with our RPi Zero and receive the list of items inside the backpack. However, we first need to deploy our web app (we are using AWS Elastic Beanstalk) before testing on a phone, as the Web Bluetooth API only works in secure contexts. Overall, this risk is being managed by our current contingency plan, which is to create a thin client on Android which handles the Bluetooth functionality, while including our Web App in a <WebView> in the Android application to handle the interfacing. However, we expect to run into many issues, particularly in the communication between the thin client itself and the web application. We’ve dedicated the upcoming week to implementing this transition and debugging any issues we run into.

No changes have been made to the existing design of the system. As always, our schedule can be viewed here.