Author: kushaanm

This week I was focused on setting up the pi for integration. This involved porting the model to it, setting up all the libraries, getting an idea of mounting, and getting some benchmarks.

I wanted to get some performance benchmarks of our CV model on the pi. Previously, I found that the latency on the model was about 4 seconds on the initial pass, which was too slow for our uses. However, when running it in a loop, it seems to be at around .8-1s per frame, which is exactly what we were looking for. Seeing the anomaly on a cold start is good, because we don’t have to switch to a jetson.

 

The libraries were being weird, and I ended up needing to reflash the Pi OS, after which all the image taking software worked well. I also hooked up some decent debugging tools, so we can see the (mis)predictions correctly.

For mounting, we discovered some of the issues with the Pi and batteries, particularly I found that we need the camera to be pretty high up and the ribbon connector means the pi will probably be near the top of the housing.

This week I was focused on the controls for the robot. We first had to hook up the motor drivers for the chassis kit before we could do any testing. Essentially, the motor drivers would give us an abstraction between driving the motors via digital writes and PWM, allowing us to ignore some of the nasty details. For the software, I created a mapping of GPIO pins to easy to understand wheel inputs. We have 8, forward and backward and for each wheel. I experimented with the chassis until I had the mappings and controls donw for basic inputs (F/B/R/L).

 

The testing software was essentially an RC car, where it took keyboard inputs nad outputted the high/low signals. I also started writing the logic for centering the bounding boxes, and deciding the appropriate thresholds. Next week, I want to focus on creating a unified robot chassis framework for the  compute, so I can start running tests using the actual/experimental setup. This should be a big step to finishing the driving automation.

This week, I was primarily focused on the ethics assignment. I came up with some ethical concerns for our project that I hadn’t thought of before. Mainly, I realized that the most dangerous parts of our project were the collimator and circuitry. The circuitry is high power and could shock someone and the collimator focuses the sound waves in a potentially dangerous manner if you held your head to it. I looked at methods to indicate this danger, and came up with a few ideas, such as the color of the cone (i.e. red for danger) and stickers. Using visual markers can help the most at-risk (children) from using this improperly.

 

I also did some research into a controls loop, particularly with Python multithreading and thread communication. I made some playground tasks to undersatnd how it works. I did some testing of inference, and found it to be a little slower than I had hoped. This led me to believe that I would need to either

a) multithread (so we don’t pin the CPU on inference, and can do controls in the meantime)

b) offload via network sockets to a laptop (or similar device), that has stronger dedicated hardware for inference (this can cut down the inference time to <80ms, vs. the 2-3 seconds of the RPI)

c) Look into model quantization/parallel frameworks to accelerate on-device.

d) Upgrade the hardware to a Jetson

This week, I will be assisting the rest of the team with integration (powering via LiPos and robot) and then looking further into sorting the small inference issue.

 

This week I worked on fine tuning the model. I found a dataset that had fire from various point of views and perspectives. My main concerns during this process were: generalization of the perspective (can the model find our use case), quantity of sata (can this actually fine tune), and generalization to picam quality.

 

I fine tuned 11n, 11s, 11m. 11n was the smallest and took half an hour to fine tune 50 epochs. The results were promising. I also took some test images of fire in pans (our use case) on my phone and it generated a clean bounding box. 11s and 11m gave pretty bad results in comparison. Much noisier bounding boxes and under detection. We decided to use 11n going forward.

 

Once the picam arrived I did some testing using that image quality. However, it performed really well and detected 95% of our testing images. I wrote a script that took images every 2 seconds and then inferred on them.

 

One limitation of the current model is it’s night time performance. For the sake of MVP, this is not a pressing concern but is something I want to focus on after MVP. This will likely involve me making custom data.

 

I also helped out on the circuit testing. On that front, we found how to get a clear wave to drive the speaker with. We saw promising results on candle testing but need the collimater to get better results.

 

Next week I plan to work on the controls loop and interfacing since we didn’t have all the necessary hardware before break. Then I will integrate this with the vision code for a rudimentary loop.

This week, I finished setting up the PI. I was able to ssh into it and started setting up the python environment. I installed the necessary packages and troubleshooted some installations. I started running some benchmarks of the base yolo model we want to get an idea of any inference time issues. I also set up the github for the fine-tuning, proof-of-concept, and final code. I will be working on the fine tuning code on Sunday.  I set up sshing and internet on the Pi as well. Once I finish the benchmarking, I will try different serving frameworks to see if the inference time can be further reduced. While doing fine-tuning, I will be writing boilerplate code for the logging, so we can see how the ML model influences the pipeline. I also looked into manipulating GPIO on the Pi.

In addition, I worked on the circuit to test the amplifier. We wanted to make sure the amplifier worked at lower wattage, because we don’t believe we can power it at full spec. However, if it runs at lower spec, we can hit the wattage requirements of the speaker, which is the important part. I helped diagram the circuit, and use the signal generator, DC-power supply, and oscilliscope. Ultimately, we were able to boost a 60Hz wave from 100mV to 600mV. We used a very conservative amount of amps, 200mA instead of the ~2-3A we expect to be running at full power.  We need to perform further testing once we are more confident about this circuit, but it should be fine because our ESP outputs 3V, so we should hit our required ~80W into the speaker.

This week I looked into the Raspberry Pi more in-depth, and also into thermal camera imaging. I looked into benchmarks regarding different inference frameworks, and their benefits. I settled on using PyTorch to begin with, as it is the easiest to proof-of-concept with. Then, I will try using more advanced parallelism frameworks. I also looked into some depth-estimation models, since this will be an important part of our product. I found a few models that look promising, but want to do some benchmarking.

 

We got the RPI this week, so I will be running benchmarks over the weekend on these models, to verify the numbers as well. I want to generate intuition for the accuracy and limitations of these models.

 

I also worked on the design presentation, primarily focusing on my sections, and updating the theme and visuals, since I will be presenting

This week we were primarily focused on creating the presentation. We took a look at parts and determined which would be suitable for our use-case requirements. Particularly, I looked at the computer vision section and what hardware would be necessary. I looked at benchmarks on various compute platforms (RPI vs Jetson) of object detection algorithms. I found that an RPI can infer in ~500ms if we tweak the parameters correctly, which we found suitable for this use case. We also looked at camera options, and I believe that a regular pi camera will provide suitable resolution at a reasonable price. 

 

In our presentation, someone brought up concerns about the effect of smoke on image detection. I considered this problem, and while I don’t have a definitive answer, one workaround I thought about was sensor-fusing with a thermal camera. We could use the combination of inputs to more accurately detect fires. 

 

I am currently on track with the schedule. Next week, I will be looking into creating some notebooks and doing proof-of-concept testing to make sure my intuition is correct about my sensing components.