This week, we filmed for our final project video on campus. Jiamin had to bring parts of the project back home, so we wanted to make sure we have all the video clips before she left.
We also worked on our final presentation slides together. We collected some feedback from the TA, so we will work on finalizing the slides and preparing the script. Iris and I are also currently working on fully integrating the MySQL database to our web app, so it is capable of storing the log data we collect. We are also aiming to finish the details page before the final presentation if time permits.
We cut out the enclosure using the laser cutter and assembled it to see what our final product looks like. Here are the images of the model and the actual printed product.
Fixed buggy RFID scanner behavior:
We used to have a bug where a byte gets lost in transmission when the Nucleo board sends the unique ID to the Nano. We were able to fix this on Nucleo’s side by always sending two sets of ID in case one of the bytes get lost. The nano is now able to detect if some bytes were lost and recover those bytes by reading the duplicated string.
Connected the red/green LED for RFID:
We wired the red and green LED to blink when invalid and valid ID’s are scanned, respectively. We also had to adjust the resistor values to make sure the green LED wasn’t blinking when the system is off, as the green LED has a lower turn-on threshold voltage.
Optimized Facial Detection FrameRate:
Initially, we were obtaining an fps of ~4. Now, we are achieving an fps of ~7.
Integration:
We integrated the logic for scanning RFID, flashing LED, measuring temperature, detecting a face, and sending the information to the cloud. We had to handle some edge cases, such as when a user moves away during the temperature measurement. Now our system has a fully integrated functionality.
To-Do:
Create a mock database functionality for users to be able to import their own RFID database and check against the scanned ID.
Add code to display temperature results on the screen.
IoT app development (front-end & back-end).
Organize the content inside our enclosure and paint it!
This week, I worked on the IoT application to safely establish a full gateway to from the Nano to the backend of our web application. We used a demo application that used a Node.js backend (provided by Cindy!). We chose to deploy it on Azure App Service because it allows us to use the existing SDK to easily write programs to move data between the our IoT device and the web application.
Now that the demo app is safely deployed, next week I will work on actually deploying our own web application. I already have designed the wifeframes for the pages so I have work on coding up the interface and connect it to the backend now.
This week we had a demo to show our project’s progress so far. We were able to showcase the basic functionalities of the RFID scanner, face/mask detection, temperature collection working separately. Moving along, we are going to finish up integrating the systems and switching the gear to work on the IoT solution.
We received some guidance from Cindy regarding the IoT side. We are using an existing IoT application (from popular tutorials) to first have a demo app up and running. This allows us to first configure the pipeline between the Nano and our application’s backend, so we can work on the actual application interface afterward. Right now, we are successfully sending the temperature and RFID data to the hub and is able to use Azure’s SDK’s to write a script to forward the data to a specific endpoint. We’re still in the process of configuring the application back-end’s endpoint to receive the data. We’ll continue to work on that and hopefully finish up setting up the pipeline by the middle of the next week.
We also had a meeting with TechSpark to discuss possible encasement designs. We weren’t able to have an actual meeting to discuss the specifics since TechSpark was closed for a bit due to a positive Covid case. 🙁
We have brainstormed some ideas and also ordered an existing enclosure for the RPi camera. We are planning on setting up another meeting to see if we can have a prototype ready in a week or two.
This week we spent a lot of time debugging our system in preparation for the demo. During the process of getting the IR sensor to work with the Nano, we noticed that the Nano’s I2C bus has some issues detecting connections from the sensor and the Nucleo board as well. We were able to confirm that the Nano is able to read bytes from the sensor’s I2C connection. However, we’re still in the process of debugging the I2C connection between the Nano and the Nucleo board. If we are unable to resolve this issue, we will consider using an alternative communication protocol that is supported by the Nano and the board such as UART or SPI.
After figuring out the I2C connection between the Nano and the IR sensor, we were able to execute a program that reads and outputs temperature data from the IR sensor every 0.5 seconds. We can simply integrate this script with the other script that was sending hard-coded data to the cloud gateway to finish up the temperature sensing functionality. Here’s a quick video of the temperature sensor working:
We also modified our work schedule. We noticed that the complexity of deploying the face/mask detection algorithm on the Nano is a lot more than we expected. So Iris and Minji will work on Nano’s face/mask detection algorithm and temperature sensing process together. After those functionalities are up and running, we will implement the web application together toward the second half of the remaining weeks. We came to this conclusion because we decided that our priority should be on fully implementing the system’s functionalities.
This week, Iris and I worked together to debug the face/mask detection program and add the IR sensor module to the Jetson Nano. The original program that we were trying to run for face/mask detection uses OpenCV for the analysis. However, we found out that the Jetson Nano doesn’t support the OpenCV libraries that are required for the program. As a result, the program wasn’t being optimized when run on a Nano and the frame rate was extremely slow.
We had to look for a different library to use for our face/mask detection and decided to try out YOLOV3. This is a more lightweight library that can often be used to run computer vision programs on a Nano with a decent frame rate (10~13 fps). We downloaded all the necessary packages and are currently waiting for the training step to finish (it usually takes ~90 minutes).
We ran into many problems when using the MLX90614 IR sensor. It requires an I2C connection with the Nano, so we hooked up the I2C SCL and SDA pins to the Nano’s corresponding GPIO I2C pins. However, we found out that the Nano wasn’t able to detect the I2C connection to the sensor when we tried running the i2cdetect command. We decided to go through several steps of debugging: (1) check if the sensor works with other boards (Arduino Uno, RPi), (2) check if the sensor is getting powered properly by the Nano’s 3.3V output pin, (3) check if the sensor has enough current flow. We went through all of the steps above and concluded that there were no issues with the sensor or the sensor’s power connection/current.
After some research, we found out that it might be an issue with the i2cdetect program not recognizing our specific sensor’s I2C connection. We decided to write up a quick script that tries to read bytes from the I2C bus at a specific address. This was possible because we knew that the sensor’s I2C address is at 0x5A. After executing the script, we discovered that we were actually reading valid data from the sensor! So we were able to produce valid temperature measurements from the MLX90614 sensor and output it from the Nano. The demo video of this short script can be found in this week’s team status report.
This week, I was able to open up the cloud gateway to allow cloud communication between the Jetson and the Azure IoT system. I used the IoT Hub’s built-in endpoints to send sample strings from the Nano and checked if those messages were being received by the hub. I successfully ran the test script and checked that all of the messages were being delivered in real-time. I also worked on configuring the Jetson and have it registered on the Hub so I can remotely monitor the status of the device.
Next week, I am going to finish building the skeleton of the app and draw out some sketches of the wireframes. This will allow me to get a headstart in the app development process.
This week, I focused on thinking about the database and designing the model for storing the temperature data. I will basically need two core tables — one for a community/organization member’s profile and one for recorded temperature. These two tables will have a one-to-many relationship. The profile record will consist of the RFID tag identification code and extracted profile information such as their first and last name. The temperature record will have the temperature, time stamp, and the location of the temperature measurement. The location is necessary for our core use case, so multiple devices can be installed throughout the building to keep track of who was in which room/area of the building. I started building out the model on the back-end side of the application using Django.
I am also preparing to start opening up the cloud gateway for receiving messages from the Nano. Next week, I’m going to work with Iris to connect the Nano to the internet using the wifi module that we already installed. We’ll test out the message transmission by sending simple test messages and see if we actually receive them on the IoT hub’s end. We aim to be able to send and receive messages between the Nano and the cloud by the end of next week.
After submitting the design review presentation, we got back to working on our project this week. Jiamin found out that the current NFC02 board we purchased for processing RFID tags doesn’t actually achieve the functionalities necessary for our project. This has posed a little bit of delay in our timeline, but we are hoping to be back on schedule once the new board, NFC05, arrives.
The rest of the group members continued to work on interfacing with the Jetson Nano to receive a video stream from the Raspberry Pi camera module and setting up the IoT application. On the Nano and software side, our team is on schedule and will continue to work more on it next week. In the upcoming weeks, the two of us are planning on working together to make work on the Nano to expedite the process of receiving and processing data from the two input sources, the IR sensor and camera. This will help us better overcome some of the bottlenecks in our progress which mostly consists of learning how to interact with the Nano and send data to the cloud from the Nano.
This week we spent most of our times working on the design review document. We received many feedbacks after the design review presentation and took them into consideration when constructing our design review document. It helped us to think more about the metrics and validation, especially regarding the accuracy of the facial and mask detection.
This week, I worked on installing Azure IoT Edge on our nano device and configuring it to be registered on our IoT Hub. For now, it seems like we don’t need extra credit for Azure IoT for testing and validating. However, if we need more quota for future testing purposes, we should ask Vyas or Cindy for credit. I also started going through tutorials for building a full-stack web application using React.js and Django. Next week, I will start working on the skeleton of the application and also design the database model for storing the temperature records.
