Author: acyu

After some bug fixes last week, our project is generally in a state that I think is ready to demo. At this point, I think we’ve kind of hit MVP and any further changes are just for additional content to demo. On the software side, I’m doing some further debugging to isolate latency. I think the only place we can feasibly get any decent gains is from optimizing the computer vision pipeline, as network latency is mostly due to our hardware and seems unfixable.

To fix the vision pipeline I implemented some instrumentation to print out latency of specific operations. Once I identify which operations take longer than expected I will try to further improve them. I also made a hardware fix by borrowing a friend’s webcam. The one we used before is a cheap one that seems to have worse drivers which made it repeatedly disconnect under high load. The new webcam seems to not have this problem.

Over the next week I will probably implement minor changes that might further optimize performance. I might also try to find other interesting applications to display.

This week I mainly worked on just improving the software and trying to get the display to look better. We’ve slightly increased speed to about 200 rpm, and though this is much under our goal of 900, I think at this speed the display is decent enough to be able to visually distinguish if there are any software bugs, which I worked on fixing.

There were a bunch of bugs and improvements I made all over the place to try and make it look better. Two that likely made the biggest difference were increasing angular resolution and fixing a display slicing bug.

For the angular resolution, we previously displayed 120 different images for every full rotation the displays made. However, I noticed that when we tried to display straight lines near the edge of the display, they would appear warped (since the distance between 2 successive images becomes farther as you move further from the center). I doubled this number so we now display 240 images for every full rotation while also testing to ensure both the Jetson and Pi were fast enough to meet this.

The major display slicing bug I fixed had to do with how we sampled frames to display. The way the logic works is that the Jetson stores the entire 3D volume of data in memory, and picks out a 128×64 2D plane to send over to the Pi to display. Previously, this plane we sampled crossed the center axis (like a diameter). However, this is inaccurate for our real physical display because the LED panels have width, so they are actually on planes that are slightly offset from the center (more like chords). Fixing this hopefully will help when things look like they are displayed completely wrong or warped.

Tools, Knowledge, Learning:

I didn’t really find the need for tools I hadn’t used before, but there were a bunch of knowledge gaps throughout the design and implementation of this project. For the overall idea itself, I used previous work to understand the fundamentals behind design decisions and architecture. For example, I often looked back on a video of another person who built a similar device to look at how he designed specific parts to fit together. For lower level details where there was no previous work to build off of, I used AI tools to help fill in the gaps so I could get instant feedback on if my understanding of concepts was wrong. I mainly focused on asking questions in order to develop a robust design so that implementation was easier.

After the interim demo, the biggest area for improvement is the refresh rate of our display. We need every piece to be faster for this to happen, so this week I worked on improving the laser cut chassis and some of the software. The chassis we used for the demo had one hole drilled in manually since I forgot to laser cut it, and at higher speeds, the imperfections in the hole caused our tube to rub against wood creating friction and bad sounding noises. I redesigned the laser cut pieces to actually have this hole so we can hopefully reach higher speeds. I also added some holes to hold it better for transportation and some minor improvements to allow for easier assembly.

On the software side, I noticed during the demo at times the display could seem a bit laggy, which motivated me to do more testing. This is likely either due to networking logic or display driving, so I investigated display logic at first (since we also got our full-sized displays which means more data to be displayed).

I created some tests to measure peak performance in terms of how fast we can update both displays, and I originally measured a maximum of 95 fps, which is too low when we are spinning at high speeds. I found that the library we used, rpi-rgb-led-matrix, used a somewhat slow implementation for using pwm to drive the display. To fix this, I replaced the library with custom direct memory access (DMA) so there’s less cpu time spent on driving LED outputs. This vastly improved performance to over 7000 fps, and next week I will work on fully testing to make sure this doesn’t affect any other part of the software.

Other verification will eventually include doing network testing to see if our throughput is enough. If not, we might have to look at improving our wifi hardware or wrapping everything in aluminum foil or something.

I focused a lot on integration this week. For all of the software, I was able to get pretty much the entire pipeline running. The hand detection model is able to be used to generate content on the Jetson. It sends successfully over UDP to the Raspberry Pi, which is able to output data to the LED panels. However, it isn’t super clear whether the data being outputted is correct, as the panel isn’t spinning yet so you only see a single slice of information. I feel like this amount of integration is pretty decent to show for the interim demo.

We also put together a lot of components this week, and some of the integration didn’t go super smoothly. Next week I want to focus on thinking through some of these interactions and possibly redesigning them so that integration feels better/sturdier.

This week I shifted a bit away from software and laser cut the enclosure. I ended up forgetting a hole so there needed to be some manual drilling to make things work, but we’re much closer to fully assembling things now. We also got a microSD card for the Raspberry Pi, which allowed me to load the code on the device and start testing it out a bit. I mainly worked on getting dependencies downloaded and ensuring our OS was configured correctly (ssh, some cpu optimizations for the led, disabling excess features).

I think we might need to speed up progress a bit for the interim demo, so I will try to focus on integrating things more. Next week I want to focus on how to integrate parts such that we can have some form of demoable content.

This week I further flushed out the software. For the Pi code, I created an initial draft with C++ code that takes display data over UDP and outputs it to the LED matrices. I implemented some hall effect sensor logic so we can estimate the angular velocity and determine the timing of when to push GPIO data. I also interfaced with an external library (https://github.com/hzeller/rpi-rgb-led-matrix) to actually send the data to the LEDs. Beyond the Pi code, I also briefly tested the Jetson code to ensure that the logic for spawning a Docker container from C++ and correctly implementing shared memory worked.

I think the progress for the software is on schedule since most everything is done. All that’s left is integrating the software to make sure it works with the hardware. Next week my main goal is to laser cut the enclosure, since I am in another class that gives me access to a laser cutter.

I spent more time flushing out all of the software for the Jetson. Beyond the computer vision Python code, I made the C++ environment for using OpenGL to render 3D geometry from OBJ files, implemented a compute shader to sample 2D slices, and developed some networking logic for compressing and sending data over UDP. Progress is visible on Github where all the source code is.

I think the software is overall on schedule. The current code should be enough to get the MVP working on the Jetson side, so all that’s left is the Pi code. Once initial implementations are done we can do more testing and optimization. Next week I want to try testing some of the software more.

Part A: Our product mainly addresses global factor through our implementation of interactivity. We want 3D interaction to feel natural and smooth compared to commonly available methods. For example, traditional 3D manipulation occurs in complex CAD software with unintuitive controls or VR environments with complex controller mappings. Using simple hand gestures to spin an object around removes these kinds of barriers for access.

This week I began some work on the software side. I did some more in depth analysis for what ML models would work well. The final 2 choices were between trt_pose and mediapipe. trt_pose was made by Nvidia and was optimized for our Jetson hardware, making it much quicker. However, it had much worse accuracy compared to mediapipe. By default, the mediapipe installation doesn’t use the GPU on the Jetson, so I worked a bit on building the package from source to optimize performance. I think I got it working, so now the performance is more comparable to trt_pose while having much better accuracy, making it a clear winner. I also did some research into OpenGL for much of our graphics logic as well as some possible compression protocols to improve networking performance.

I think we are still on schedule overall as the software design is pretty mature, so the only big hurdles left for software are implementing and integrating.

This week I mainly worked on the Design Review presentation. This involved ironing out some design decisions like how the actual hardware components might go together. This also helped mostly finalize our parts list.

I think we are on schedule since we have our design pretty detailed, which means we can proceed already with implementation as integration should hopefully be easier. Next week I hope to begin implementing some of the software. I will try to get one of the machine learning models to work and successfully detect hand landmarks.

Part C: Our project is not really meeting the economic need of any existing product market, as it is relatively unique. However, we are aiming to have it be cost effective compared to any similar solutions. For example, the best possible industry standard for volumetric displays is produced by Voxon and costs $6800. We have also made sure our device can operate by itself without an external computer or any other devices, making it economically easier to use the product once you have it.