Posted on

Andrew’s Status Report for 4/24

After some bug fixes last week, our project is generally in a state that I think is ready to demo. At this point, I think we’ve kind of hit MVP and any further changes are just for additional content to demo. On the software side, I’m doing some further debugging to isolate latency. I think the only place we can feasibly get any decent gains is from optimizing the computer vision pipeline, as network latency is mostly due to our hardware and seems unfixable.

To fix the vision pipeline I implemented some instrumentation to print out latency of specific operations. Once I identify which operations take longer than expected I will try to further improve them. I also made a hardware fix by borrowing a friend’s webcam. The one we used before is a cheap one that seems to have worse drivers which made it repeatedly disconnect under high load. The new webcam seems to not have this problem.

Over the next week I will probably implement minor changes that might further optimize performance. I might also try to find other interesting applications to display.

Posted on

Team Status Report for 4/24

At this point, our project is pretty demoable and there aren’t any significant risks unless we break something. We are trying to prioritize safety for the demo by limiting speed, which should also mitigate risks of anything breaking. There haven’t been any changes to any designs, and we are still on track for demo day.

Tests:

  • Motor RPM: We tested a safe motor speed by incrementally increasing our target duty cycle and carefully observing any changes. At around maybe 24% duty cycle we observed some slight slippage, which might be attributed to PLA warping. So, we will try to reprint in PETG before demo day. If that isn’t achievable we can simply limit speed to 20% duty cycle, which seems to work fine for extended periods.
  • LED Display Time: This metric was measured because this is the hard physical limit for how fast we can update the display. We measured a mean of 184 microseconds and a maximum of 479 microseconds. This informed our decision to increase the angular resolution from 120 to 240 slices per revolution.
  • End to End Latency: We decided the best way to measure the end to end latency was taking a video and seeing when gestures would be processed. In our video we measured around 500 ms of latency, which is a bit higher than we expected. This finding motivated us to do a bit more instrumentation to identify if there are any places we can optimize the computer vision pipeline.
  • UDP gaps: We measured the difference between successive instances of receiving network information. We got around 280 ms, which informs us a lot about where our latency is likely coming from. Unfortunately, this number seems difficult to change because our setup needs to stay wireless and our hardware can’t really be changed at this stage.
Posted on

Andrew’s Status Report for 4/17

This week I mainly worked on just improving the software and trying to get the display to look better. We’ve slightly increased speed to about 200 rpm, and though this is much under our goal of 900, I think at this speed the display is decent enough to be able to visually distinguish if there are any software bugs, which I worked on fixing.

There were a bunch of bugs and improvements I made all over the place to try and make it look better. Two that likely made the biggest difference were increasing angular resolution and fixing a display slicing bug.

For the angular resolution, we previously displayed 120 different images for every full rotation the displays made. However, I noticed that when we tried to display straight lines near the edge of the display, they would appear warped (since the distance between 2 successive images becomes farther as you move further from the center). I doubled this number so we now display 240 images for every full rotation while also testing to ensure both the Jetson and Pi were fast enough to meet this.

The major display slicing bug I fixed had to do with how we sampled frames to display. The way the logic works is that the Jetson stores the entire 3D volume of data in memory, and picks out a 128×64 2D plane to send over to the Pi to display. Previously, this plane we sampled crossed the center axis (like a diameter). However, this is inaccurate for our real physical display because the LED panels have width, so they are actually on planes that are slightly offset from the center (more like chords). Fixing this hopefully will help when things look like they are displayed completely wrong or warped.

Tools, Knowledge, Learning:

I didn’t really find the need for tools I hadn’t used before, but there were a bunch of knowledge gaps throughout the design and implementation of this project. For the overall idea itself, I used previous work to understand the fundamentals behind design decisions and architecture. For example, I often looked back on a video of another person who built a similar device to look at how he designed specific parts to fit together. For lower level details where there was no previous work to build off of, I used AI tools to help fill in the gaps so I could get instant feedback on if my understanding of concepts was wrong. I mainly focused on asking questions in order to develop a robust design so that implementation was easier.

Posted on

Andrew’s Status Report for 4/4

After the interim demo, the biggest area for improvement is the refresh rate of our display. We need every piece to be faster for this to happen, so this week I worked on improving the laser cut chassis and some of the software. The chassis we used for the demo had one hole drilled in manually since I forgot to laser cut it, and at higher speeds, the imperfections in the hole caused our tube to rub against wood creating friction and bad sounding noises. I redesigned the laser cut pieces to actually have this hole so we can hopefully reach higher speeds. I also added some holes to hold it better for transportation and some minor improvements to allow for easier assembly.

On the software side, I noticed during the demo at times the display could seem a bit laggy, which motivated me to do more testing. This is likely either due to networking logic or display driving, so I investigated display logic at first (since we also got our full-sized displays which means more data to be displayed).

I created some tests to measure peak performance in terms of how fast we can update both displays, and I originally measured a maximum of 95 fps, which is too low when we are spinning at high speeds. I found that the library we used, rpi-rgb-led-matrix, used a somewhat slow implementation for using pwm to drive the display. To fix this, I replaced the library with custom direct memory access (DMA) so there’s less cpu time spent on driving LED outputs. This vastly improved performance to over 7000 fps, and next week I will work on fully testing to make sure this doesn’t affect any other part of the software.

Other verification will eventually include doing network testing to see if our throughput is enough. If not, we might have to look at improving our wifi hardware or wrapping everything in aluminum foil or something.

Posted on

Team Status Report 4/4

We did not face any significant challenges this week. We successfully pulled off the interim demo and are now preparing for our final demo.

No significant schedule or design changes to our project at this time. Our LED panels have finally arrived and we’re now ironing out how to program the displays.

For full system validation, we’ve been testing while integrating to make sure everything would work well. For example, for power draw we have measured components individually and ensured that they pull around the amount we estimate. Once we have a more final product, we will also take the time to measure framerate and latency to ensure they are within reasonable bounds. We will also make some qualitative observations to see which aspects we will focus on improving.

Posted on

Steven’s Status Report 4/4

This week marked our interim demo, where we completed our first full system integration. Overall, the demo was quite successful, and we received largely positive feedback from both our instructors and TAs. While the LED display has not yet achieved 3D image rendering and we are not operating at our target RPM, we were still able to present a strong proof of concept. I was honestly surprised by how much we were able to accomplish in time. Andrew and Kendric both put in significant effort to make the interim demo a success.

One major challenge I overcame prior to the demo was getting the hall-effect sensor to work with the Adafruit HUB adapter. Because the adapter occupies all of the Raspberry Pi’s GPIO pins, we needed to route the hall-effect sensor through the HUB itself. Although the adapter supports up to three LED panels and we are only using two, leaving some pins physically available, we initially overlooked an important detail: data direction. LED panels are designed to receive data, while hall-effect sensors transmit data. As a result, when we first connected the sensor to the available GPIO pins, we were unable to read any signal.

After further debugging, we discovered that the Adafruit HUB adapter only supports unidirectional data flow on most pins, with the exception of the SDA and SCL lines. We then rerouted the hall-effect sensor to these pins, allowing proper data transmission back to the Raspberry Pi. Fortunately, this resolved the issue, and we were able to demonstrate a fully functioning hall-effect sensor during the interim demo.

After having a working project for the interim demo, I’m much more confident about where we are in our progress. I feel that we are now back on schedule and in good shape for the final demo.

Next week, I’m planning on working on the Raspberry Pi software to drive images to display as well as preparing final demo logistics.

Posted on

Andrew’s Status Report for 3/28

I focused a lot on integration this week. For all of the software, I was able to get pretty much the entire pipeline running. The hand detection model is able to be used to generate content on the Jetson. It sends successfully over UDP to the Raspberry Pi, which is able to output data to the LED panels. However, it isn’t super clear whether the data being outputted is correct, as the panel isn’t spinning yet so you only see a single slice of information. I feel like this amount of integration is pretty decent to show for the interim demo.

We also put together a lot of components this week, and some of the integration didn’t go super smoothly. Next week I want to focus on thinking through some of these interactions and possibly redesigning them so that integration feels better/sturdier.

Posted on

Team Status Report 3/28

Successfully integrated everything (except the display) this week.

One big issue we ran into this week was losing both of our displays. We realized that one of our displays was bricked, and shortly after, we proceeded to short our only remaining working display, so we’re currently working blind. We got the Pi working, so we can write the code, but we can’t verify if anything we’re writing is correct. We’ve since ordered express shipping on four new LED panels (we learned our lesson on redundancy), so that we can begin testing asap.

Other than the LED Display roadblock, we haven’t faced any additional significant challenges. One small component we may be redesigning in the near future is the chassis of the box. The individual components are currently a bit cramped inside our box, so we might redesign the box with a bit more room, as well as laser-cut a precision hole for the spinning metal rod to go through so it’s no longer rubbing against the plywood box.

As an update for our interim demo, here are some of our relevant figures and our updated schedule, which haven’t changed much since we began: Interim Demo Slides.

Posted on

Steven’s Status Report 3/28

 This week, my main focus was getting the motor running predictably. After digging through the ESC documentation, I figured out it’s controlled by a 50 Hz PWM signal (1ms for minimum power, 2ms for maximum). With that in mind, I wrote an Arduino program that lets us customize the motor speed using a simple percentage input. I also wired up a two-button setup on a breadboard: one to turn the motor on, and one to turn it off, which safely lets the motor coast to a slow stop at any point during operation.

Hardware-wise, I think we are finally back on track. Andrew and Kendric did some great work on the CAD side this week. The 3D-printed clamps are working surprisingly well and have pretty much solved the mechanical fit issues we were stressing about last week. We also finally got our microSD card, so the Pi is fully functional. But, ironically, we hit a massive new roadblock: both of our displays are dead (one is bricked, the other shorted out). So even though the Pi is ready to go, we can’t actually test any of our display code. We’ve express-ordered new 64×128 LED panels from China, but they definitely won’t be here in time for the interim demo.

Even without the physical screens, my goal for next week is to keep pushing forward on the software side. I’ll be writing the display code blind and working on integrating the Jetson Nano output so we’re completely ready to display real images the second those new panels arrive.

Posted on

Andrew’s Status Report for 3/21

This week I shifted a bit away from software and laser cut the enclosure. I ended up forgetting a hole so there needed to be some manual drilling to make things work, but we’re much closer to fully assembling things now. We also got a microSD card for the Raspberry Pi, which allowed me to load the code on the device and start testing it out a bit. I mainly worked on getting dependencies downloaded and ensuring our OS was configured correctly (ssh, some cpu optimizations for the led, disabling excess features).

I think we might need to speed up progress a bit for the interim demo, so I will try to focus on integrating things more. Next week I want to focus on how to integrate parts such that we can have some form of demoable content.