This past week, I have been working on further integration of BLE (Bluetooth Low Energy) into our existing system. Last Monday, I had produced a fully functional system after resolving a few Python dependency issues. The Jetson was able to establish an ongoing BLE connection with my iPhone, and I had integrated the dirt detection Python script into the Bluetooth pipeline as well. There were a couple issues that I had encountered when trying to perform this integration:
- Dependency issues: The Python version which is required to run the dirt detection algorithm is different from the version which supports the necessary packages for BLE. In particular, the package I have been using (and the package which most Nvidia forums suggest) has a dependency on Python3.6. I was unable to get the same package working with Python3.8 as the active Python version. Unfortunately, the bless package which spearheads the BLE connection from the Jetson to the iPhone is not compatible with Python3.6—I had resolved this issue by running Python3.8 instead. Since we need one end-to-end script running which queries results from the dirt detection module and send data via BLE using the bless Bluetooth script, I needed to keep two conflicting Python versions active at the same time, and run one script which is able to utilize both of them.
- Speed constraints: While I was able to get the entire dirt detection system running, the granularity at which the iPhone is able to receive data is incredibly low. Currently, I have only been able to get the Jetson to send one message about every 25-30 seconds, which is not nearly enough with respect to what our use-case requirements have declared. This issue is further exaggerated by the constraints of BLE. The system is incredibly limited on the amount of data it is able to send at once. In other words, I am unable to batch multiple runs of the dirt detection algorithm and send them to the iPhone as a single message, as that message length would exceed the capacity that the BLE connection is able to uphold. One thought that I had was to send a singular timestamp, and then send over a number of binary values, denoting whether the floor was clean or dirty within next couple seconds (or milliseconds) following the original timestamp. This implementation has not yet been tested, but I plan to create and run this script within the week.
- Jetson compatibility: On Monday, the script was fully functional, as shown by the images I have attached below. However, after refactoring the BLE script and running tests for about three hours, the systemd-networkd system daemon on the Jetson errored out, and the bluetooth connection began failing consistently upon reruns of the BLE script. After spending an entire day trying to patch this system problem, I had no choice but to completely reflash the Jetson’s operating system. This resulted in a number of blockers which have taken me multiple days to fix, and some are still currently being patched. The Jetson does not come with built in WiFi support, and lacks the necessary drivers for native support of our WiFi dongle. Upon reflashing the Jetson’s OS, I was unable to install, patch, download, or upgrade anything on the Jetson, as I was unable to connect the machine to the internet. Eventually, I was able to get ahold of an Ethernet dongle, and I have since fixed the WiFi issue. The Jetson now no longer relies on a WiFi dongle to operate, although now I am able to easily set that up, if necessary. In addition, while trying to restore the Jetson to its previous state (before the system daemon crashed), I ran into many issues while trying to install the correct OpenCV version. It turns out that the built-in Linux package for OpenCV does not contain the necessary packages for the Jetson’s CSI camera. As such, the dirt detection script that I had was rendered useless due to a version compatibility issue. I was unable to install the gstreamer package; I had tried everything from Linux command line installation to a manual build of OpenCV with the necessary build flags. After being blocked for a day, I ended up reflashing the OS, yet again.
This week has honestly felt rather unproductive even though I have put in many hours; I have not dealt with so many dependency conflicts before. Moving forward, my main goal is to restore the dirt detection and BLE system to its previous state. I have the old, fully function code committed to Github, and the only thing left to do is to set up the environment on the Jetson, which has proven to be anything but trivial. Once I get this set up, I am hoping to significantly speed up the BLE transmission. If this is not possible, I will look into further ways to ensure that our AR system is not missing data from the Jetson.
Furthermore, I have devised tests for the BLE and dirt detection subsystems. They are as follows:
- BLE Connection: Try to send dirty/clean messages thirty times. Find delay between each test. Consider the summary statistics of this data which we collect. Assuming the standard deviation is not large, we can focus more on the mean value. We hope for the mean value to be around 0.1s, which gives us fps=10. This may not be achievable given the constraints of BLE, but with these tests, we will be able to determine what our system can achieve.
- Dirt Detection
- Take ten photos each of:
- Clean floor
- Unclean floor, sparse dirt
- Unclean floor, heavy dirt
- Assert that the dirt detection module classifies all ten images of the clean floor as “clean”, and all twenty images of the unclean floor as “dirty”. If this is not met, we will need to further tune the thresholds of the dirt detection algorithm.
- Take ten photos each of: