Jerry:
This time, I’ve gotten the DPU properly instantiated and working on our custom boot image. It turns out that debugging these pieces of firmware can be pretty tricky. The first few times I re-compiled the Petalinux image, I excluded the DNNDK files, DPU driver, and other libraries like OpenCV, and added them in after compilation. After shuffling about to find where to put the files, I managed to make everything except the DPU driver work. The DPU driver, on the other hand, seemed eerily silent. It had a debug print in the first line of its initialization, but I never saw that line. I thought it was because the debug prints weren’t enabled in the kernel, but I tried several kernel configuration and still saw no prints. I then thought it was because the driver was compiled in release mode, removing all printk’s, so I recompiled the DPU drivere (this time, including it and all the other libraries directly in the Petalinux boot image at compilation), but it still didn’t make a noise. Finally after some Google searching for what others found when their (totally unrelated) drivers didn’t print, I found out that I needed to add something to the Linux device tree. It was great seeing all my redundant print statements show up once I recompiled the DPU driver with the device tree update.
There was also a little moment of dependency hell trying to make our Yolo-v3 run. At first I had an error message that the architecture of our DPU wasn’t supported by DNNDK (but why would you include this architecture as an option in the IP??). One painful Vivado recompilation later, I found out that the convolutional neural network IP instantiated an old version of the DPU. Luckily, DNNDK offered a (non-default) compiler for the older version of the DPU. Guess what, that DPU supported the architecture I started with earlier… Another painful Vivado recompilation, and I finally got the DPU to correctly predict a bounding box around a test image.
After that we focused on the focus / zoom equipment. I used the training data from Karthik’s calibration to fit a curve, where it turned out that a 4th degree polynomial had a very close fit to our observed points.
Karthik Natarajan:
Throughout this week, I mostly worked on calibrating the zoom lens and finding the proper value of focus. Firstly, after integrating Jerry’s new stepper library into our featherboard, we were able to move multiple steps at a faster rate than before. And, we have been able to decrease the heat coming from the servos that was mentioned before by releasing the stepper motors once they have finished moving. After integrating Jerry’s modified stepper.py library, I worked on getting the zoom lens to focus properly at different levels of zoom. To reduce both the complexity and the time for each adjustment, I decided to use a function to model the relationship between the focus and the zoom of a lens. This idea was actually initially prompted by finding an analytical curve for the relationship online. However, because this curve was lens specific, I manually looked for the best focus step values, for a fixed set of 11 zoom values by looking at how sharp a person appeared at a distance proportional to the zoom.
After doing this for a while, I saw that there was a small difference due to hysteresis in the proper focus value when moving the stepper motor backward vs forward. To fix this problem, we introduced a slack variable which changed the number of steps based on which way the stepper motor was moving. After getting all of these points, Jerry graphed the data in excel and we came up with the curve below
Nathan:
I spent this week primarily on integrating the Logitech C920 with the Kurokesu enclosure we received last week. This was substantially more difficult than expected, and involved a fair amount of, shall we say, “brute force” in the disassembly of the existing C920 enclosure. The screws in particular had an almost malign reluctance to budge, and so needed to be goaded in a process involving some pliers and a vice clamp. The soldering itself was also rather persnickety, involving delicate manipulation of fine wires, and even scraping away a layer of the PCB in a process that has no doubt measurably shortened my lifespan. In any case, after much turmoil, the C920 PCB was successfully transplanted into its new home in the Kurokesu case, and it works! This thankfully removed one of the major risks our project faced, and paves the way for a smooth transition into our Monday demo and final presentation.
This upcoming week, I’ll be working with Karthik and Jerry to prepare for our demo on Monday, and will actually get started on buying/building the case for all the components. This is more an aesthetic matter than anything else, but I consider it important for the final presentation.
Team:
This week was generally a success for our team. It was more work than expected, but the C920 PCB was transferred to the Kurokesu enclosure, and the zoom lens successfully integrated. A picture is included at the bottom of this blog post, though regretfully we forgot to take a picture of the intermediate steps. This removes one of the biggest outstanding risks this project faced, and completes a key milestone for the Monday demo.
As an update from last week, and pending any breakthrough regarding the matter, we’ve decided to go through with the power-on/power-off wake behavior for the Ultra96, with boot times around 5s, acceptable for our use case. To compensate for that longer than anticipated delay, we’re working on introducing an intermediate power state with the DPU deactivated, allowing for primitive low-power inference before the full capabilities of the SoC are engaged. This change allows us to remain in a power-on state more liberally, improving the responsiveness of the system within our power profile.
Other tasks for this project’s completion include the construction of the case/package, final tuning of the zoom/focus, and integration of the power system. However, these constitute more of a “to-do” list than actual threats to the project, so we feel quite confident in the project’s state heading into the final stretch, and at this time have a significant majority of the core systems complete.
As for scheduling, there has been some minor clean-up regarding remaining tasks, including breaking up the optimization and final system integration, but there are no major changes to report.
