Category: William Shaw’s Status Reports

This week, the final design for the actual mount was completed. I did a final revision, adding a front plate, the neoprene padding, and using black and smoke grey acrylic (to make it look less suspicious). I also found a way to mount the Wifi+Bluetooth module, although it is not the most secure way of doing it (though necessary due to space constraints). 

The IMU integration needs to be finalized, with resync signals based on drift and finding the new walk sign. This should not be too bad, but it requires us to find accurate angles based on the detected traffic lights from the Object Detection ML model, which involves some math based on the bounding box location and the camera’s field of view (FOV). 

Regarding the schedule, I am in sync with the Gantt Chart. I need to finalize some things with the IMU, but it’s good for the most part. Another component is finalizing the audio signals. We are currently using a Python text-to-speech library, but it may also be better to have some pre-recorded audio signals to just play (including beeps and whatnot). 

This week, I focused more on fine-tuning the chest mount. We also continued working on integration and testing all our components. Last week’s changes to the chestmount were well received. We tested the chest mount of a few women, all of whom stated that the mount was secure and comfortable. This resolved our worries about non-flat chests being incompatible with our mounting design. The standoff components arrived, and I had to make some tweaks to the mounting dimensions to get them to work properly. However, after the modifications, everything works perfectly. Currently, the Jetson Orin Nano, camera, and IMU are nicely and securely mounted onto the chestpiece. One small issue was that I didn’t consider mounting the WiFi module, along with its antennas. These were connected to the original case. I had assumed that we would have our stuff running offline, so I thought I wouldn’t need to mount these. I will look into fully mounting these, but for now, they are safely tucked into the IMU standoffs. Lastly, I have begun testing the addition of neoprene padding for comfort. 

The power bank also worked well, providing more than enough power to last the whole day (around 11 hours). With the components all nicely mounted to the chest mount, we were much more easily able to gather data. Previously, data collection needed to be done with two people, as we had to hold all the components together carefully. Now the device is basically hands-free, save for the laptop needed to SSH in and run the scripts. Hopefully, this can be replaced with a startup script once we finalize integrating all our software components together. 

Regarding the schedule, I am in sync with the Gantt Chart. I aim to continue working to get all the components mounted to the chest mount. I expect to work through some issues with screw hole dimensions, so I have blocked out a decent chunk of time for iterating through designs. 

As I designed, implemented, and debugged my project, I had to use some tools to fix issues with the camera. Specifically, I had to use nvarguscamerasrc, Argus daemon, GStreamer, and v4l2-ctl for debugging. Most of my knowledge for this was gained by using Google and the NVIDIA Developer Forums for people with similar issues. I also had to debug many issues by checking versioning issues with packages. This issue is still in progress, as we have some conflicts with what packages are needed.

This week, I focused on the refinement of the chest mount, as well as some integration and finalization. For the chest mount, I worked on calibrating the designs for comfort. Now, the main part of the chest mount is made of 3mm acrylic (vs the 6mm plywood previously used). This allows for much smoother contact with straps, as the previous design had issues with fraying fabric.  It is also much lighter and more flexible, making the design more comfortable to wear. The standoff components are arriving tomorrow, which will let me mount the components to the chest piece. 

In other news, the power bank arrived this week. It took a while for this, as I had to ensure that the battery could consistently output the required wattage over the ports (an issue we ran into with existing power supplies). This allowed us to take the system out for testing, albeit awkwardly, as we held all the components, laptops, and wires by hand. The results from this testing were not great, as the classifiers could not correctly detect the correct signals most of the time. As such, we recorded a bunch of new data, which my teammates are using to retrain the models to make them more accurate. 

Another issue was that my default settings for the camera were lackluster. I believe that with some fiddling, I can get a better picture quality for the models. However, this higher resolution may not help or may even cause the models to take longer to run. Ultimately, more testing needs to be done for this. On a more positive note, the red tint issues from last week have been completely resolved using a .isp file I found online. 

Regarding the schedule, I am more or less in sync with the Gantt Chart (which was updated during demos). I aim to continue working tomorrow to get all the components mounted to the chest mount. I expect to work through some issues with screw hole dimensions, so I have blocked out a decent chunk of time for iterating through designs.

This week, I mainly focused on the chest mount. I have finished my second iteration of the design. I plan to cut it out tommorow for testing. It is a no sew design, where I can slide on straps through some holes into the mount. I also took measurements of all the mounting holes and strap dimensions with a pair of calipers, so the dimensions of the design should be more or less accurate (at least to the point where everything fits and can be mounted. 

With regards to peripherals. I am doing more calibration on the camera. It has a reddish tint, and I have been trying to apply a filter/patch to remove it. I am also testing out the different frame rates and resolutions to find out what gives us the best picture quality. The GPS and IMU sensors have taken a backseat, as there is not much to demo for them yet. 

Regarding the schedule, I am more or less in sync with the Gantt Chart. I aim to continue working tomorrow (before the deadline) to cut out and test the second iteration of the chest mount. I also need to go and find some mounting hardware (standoff screws and screws).

This week, I began work on designing the chest mount. This is being done in Fusion. I plan to laser-cut my initial design out of wood as a proof of concept. The final version will be made of either ABS or wood, depending on which is more durable and easy to mount the devices onto. I will also likely add either a foam or TPU pad to the underside of the mount, as having a hard chestpiece would be uncomfortable for the user. This will be a focus over the next week, as I want to wear the device. I also need to figure out how the harness straps will work, as I do not know how to sew. 

With regards to peripherals. The camera may have broken. It no longer appears in “ls -l /dev/video*”, so I am unsure what happened. I plan to resolve this as soon as possible. I also got the VNC to work better by using a dummy DisplayPort plug, which tricks the Jetson Orin Nano into thinking it is plugged into a monitor. I opted to do this over running a virtual desktop, as I assumed it would be easier to turn off (just unplug it) and take less system resources. Another update is that I have confirmed that the Jetson Orin Nano works at full power over the USB-C PD to  DC barrel adaptor!

Regarding the schedule, I am out of sync with the Gantt Chart. This is because some peripherals stopped working, and I must fix them. Next week, I hope to get the final dimensions sorted for the chest mount and try attaching the parts to the prototype. I have included some images of the current design (which is still being worked on) below.

This week, I continued my work testing how to connect each peripheral device to the Jetson Orin Nano. There’s not much to update here, as I am still working on getting it done. I expect to have this completed by the end of next week. In particular, the I2C is a bit off, but I think it should just be a minor code fix that I need to resolve. The camera is in a good place, and I believe the data is being properly sent over the GStreamer pipeline. 

One easy test was the USB C PD to DC cable. I have verified that with a 45W wallwart, it is able to power the Jetson Orin Nano from a USB-C power source. However, I could not get the device to power on using the power bank that I had at home. This means I must find a suitable power bank that can successfully supply the voltage and has more modern PD standards. It is important to note that the Jetson Orin Nano powers on, but I do not know the wattage that it is running at over the adaptor. It is entirely possible that it is only running at a lower wattage setting, so I will find a way to verify that it works at full power next week. 

I have also begun designing two mounting systems for the device. One head mount and one chest mount. This is based on our weekly meeting, where we discussed that there may be some benefits to mounting the camera on the chest rather than the head. Once both designs are finalized and printed (or laser cut), we will test and compare both for user comfort and camera shake. 

 

Regarding the schedule, I am still about a week behind the Gantt chart. This is a hold-over from last week, where I forgot to account for spring break on the schedule. For next week, I plan to finalize the peripheral integration and to continue working on the mounting designs. 

This week, the rest of the critical parts arrived. As such, I was able to move into the testing phase for the components. Since I am still in the earlier stages of testing and integrating the components, my focus has been primarily on the setup process and ensuring basic connectivity. First, for the camera (IMX219), it is connected via CSI-2, so the system detects it under “/dev/video*; v4l2-ctl –list-devices”, instead of with “lsusb”. I then made sure that v4l-utils and gstreamer were installed and updated to interact with the camera. More testing needs to be done to actually access the video feed, but the device is being detected. Second, for the IMU (BNO055), it communicates over I2C. As such, I use the command “i2cdetect” to check that the module is detected on the I2C bus of the Jetson Orin Nano. Next, I will use the smbus python library to read the raw sensor data. 

I also worked on configuring the Jetson Orin Nano for headless operation, ensuring that we can all access and interact with the system without needing an external monitor, keyboard, and mouse. Headless will be the operation mode for the project’s final phase, as we will not be able to have a monitor attached to the user. This is done through SSH’ing on my laptop. I also set up VNC (Virtual Network Computing) to get a visual remote desktop. Initially, I was experimenting with using Vino, but that has varying performance depending on the exact Jetson device being used. As such, I ended up swapping to x11vnc. This can be connected to using the built-in VNC client on a MacBook (“Screen Sharing” app). Separately, there were some initial issues with getting the wifi to work properly (due to some user privilege issues), but they have been resolved. 

Regarding the schedule, I am about a week behind the Gantt chart. This is because I did not consider that Spring Break was my “Week 5”, so I misjudged the actual dates. I plan to finish testing of each component by this week to get back on schedule. I also want to double check that the Jetson Orin Nano works on CMU-Secure/Device, as I have just been testing on my home network.

This week, I ordered the other essential parts. This included the GPS module and a USB-C to DC barrel jack adaptor. Neither of these are technically system-critical as of right now, but ordering them together saved on shipping costs from Adafruit. I opted to wait on ordering the USB Sound Card, as the driver used for audio output should not change, and we may opt to use wireless bone conductive earphones instead of the on-ear headphones. This could be a safer alternative, as they block the least ambient noise.

Regarding the Jetson Orin Nano, I spent this week setting it up in preparation for future tasks. This included updating the board’s firmware, loading a new boot image with JetPack SDK, and setting up Ubuntu. After completing these preliminary steps, I moved on to installing the dependencies we would need for future tasks. Many of these were included in the JetPack SDK, so it took less effort than expected. I also began trying to run a few demos like Ollama on the Jetson. Ideally, this makes me more familiar with the platform, which should make later work smoother.

In terms of schedule, I am right on track. The parts should arrive in a few days, which is on schedule for me to begin testing. Next week, I plan to complete much of the testing for interfacing the hardware to the Jetson. I will focus on the camera and the IMU first, as these are our most system-critical components. I also want to begin drafting our overall mounting mechanism.

This week, I primarily focused on ensuring that all our parts were in order. This was a much deeper dive into each hardware component than before, and I checked for things like compatibility, interfacing, and outputs. This caused me to revise a few of the prior hardware choices and realize that more needed to be added. In particular, the GPS module we planned to use did not give accurate heading data for stationary subjects. As such, I added a new IMU module to act as a compass, the Adafruit 9-DOF IMU Fusion Breakout BNO055. The module allows us to get accurate user heading without movement up to ±2°, while automatically compensating for head-tilt and sensor noise. 

Another update regarding parts is the audio output and power supply. I had previously thought a Type C Power delivery power bank could power the Jetson board. However, for the Jetson Orin Nano Development Kit, the Type C port is data only and does not power the board. As such, I am looking into alternative power supplies/options for when we make the system portable. Additionally, the board does not come with a 3.5mm audio jack. While there is audio over the HDMI port, that is not a viable solution since we will not connect the board to a display. As such, I need to find a compatible USB sound card for the board. 

So far, I have the Jetson Orin Nano on hand. The Arducam IMX219 (camera) and BNO055 IMU (Compass) are being shipped. These are necessary for us to begin testing the navigation system of our project, so we should be able to start testing actual inputs when they arrive (assuming that interfacing goes smoothly). There are a few remaining components to order (speakers, soundcard, portable power supply), but they are not system-critical for the work that needs to be done so far. I plan to order these components by the following weekly report. I am on schedule so far. By next week, I hope to have placed orders for all the components. I also aim to successfully interface the IMU and camera to the Jetson Orin Nano board. 

This week, I spent most of my time looking into hardware components for our project. More specifically, I looked into the camera, GPS/Compass, and battery components. I also began drafting plans for mounting these components cleanly and comfortably. Currently, I plan to mount the camera to the helmet with a GoPro mount. This would let us adjust the camera angle based on testing results. Furthermore, I would like to create a case for the Jetson board and its components using laser cutting or 3D printing. If it is light enough, the board and battery can be mounted to the helmet, but otherwise, we may need to use a hip-mounted pack for comfort.

I also placed the order form for the Nvidia Jetson Orin Nano. I noticed a 4GB variant (non-Orin) was also available in the existing ready-to-use stock, but I am unsure if it is sufficient to run our models.

I am on track for our Gantt Chart Schedule. Although I have not placed all the orders for the other parts yet, testing of the code can begin once we receive our Jetson board. I would like to get this before placing orders for some of the parts (like the camera), so we can test the interfacing available. I also want to check how well it runs the model, as the top contender for the camera (Arducam IMX219) outputs a 3280 x 2464 pixel image. Although the FOV seems promising, the resolution might be too high to run our model at a suitable refresh rate.

For next week, I hope to finalize all of the parts and place the orders! However, I will emphasize that I want to ensure the components work first, which may require testing on the Jetson board beforehand.