Category: Shakthi’s Status Updates

Accomplishment: Worked on integrating the OR Module, Speech Module and the Proximity module into the NVIDIA Jetson alongside Josh. We found that the proximity module that I was unable to debug in the past is going to pose a problem to the flow of our project. We spent a whole day trying to figure out if the problem was either in our software implementation, or possibly in our circuitry, but found that the NVIDIA Jetson may simply not be capable of real time processing that the ultrasonic sensor requires. We concluded that we will need to pivot to a different approach for the proximity module and weighed the pros and cons of either solution. The solutions are outlined in detail in our team report but one would be a more robust solution that will require a big design change and implementation time, while the other will be a smaller quicker fix but one that may lower the accuracy of our device. We have yet to make the call on which route we will take but Josh and I worked hard to find ways to address this. We also completed most of the system setup to make the program run upon bootup of the jetson, so no manual command is needed.

Progress: I made significant progress with the integration into the Jetson along with Josh as well as trying and debugging the headless setup for the device. We went through many ways and debugged many errors together and I feel good about the progress we made.

Projected Deliverables: In the coming days the goal is for us to make a call on which way to address the problem that has risen with the ultrasonic sensors, whether the software or hardware solution. I also hope to have flushed out the speech module to address any and all cases for the final demonstration.

As you’ve designed, implemented and debugged your project, what new tools or new knowledge did you find it necessary to learn to be able to accomplish these tasks? What learning strategies did you use to acquire this new knowledge?

During the design, implementation, and debugging of my project, I found it necessary to learn several new tools and gain new knowledge to accomplish these tasks successfully. Some of the new skills I acquired include using sudo for executing commands with superuser privileges, grep for searching text patterns, understanding ALSA (Advanced Linux Sound Architecture) for audio setup on the NVIDIA Jetson Nano, configuring Jetson audio setup, working with threading for concurrent execution, and implementing real-time processing for audio and other data. I also delved into watching youtube videos to visually grasp concepts, reading through github project repos with READMEs to understand implementations and dependencies, and did extensive research on various topics related to my project. Reading documentation was a crucial part of my learning process, as it provided valuable information and guidelines on how to use these tools and technologies effectively. Additionally, being patient and reading through people’s errors and solutions on forums and communities helped me understand common issues and their solutions, allowing me to troubleshoot and debug more efficiently. By combining these learning strategies, such as watching videos, reading documentation and GitHub repos, doing research, and being patient, I was able to enhance my skills and successfully implement and debug my project on the NVIDIA Jetson Nano platform.

We recognize that there are quite a few different methods (i.e. learning strategies) for gaining new knowledge — one doesn’t always need to take a class, or read a textbook to learn something new. Informal methods, such as watching an online video or reading a forum post are quite appropriate learning strategies for the acquisition of new knowledge

Accomplishment: Began initial testing of speech module, and customized the voice output parameters using the pyttsx3 and espeak TTS engines. This involved reading up on the documentation of these python librarie and following the demo code available to adapt to our use case. I faced some complications in testing due to the various OS environments we were working on (linux, vs windows vs macOS), and so finally getting it functioning on the NVIDIA Jetson was rewarding. With Meera’s help on the hardware side we have gotten the jetson to output audio via wired headphones. Here is an image of the initial stage of the speech module implementation:

Progress: Once I complete some more testing of the speech module and make it a continuous process in the main loop, I will get an idea of the latency of the object recognition to final output. Based on that we can fine-tune the different parameters and runtime of the programs written. This coming week I hope to have both the proximity and speech modules close to completion.

Projected Deliverables: I hope to begin testing and also integrate the speech and proximity modules together and prepare for the OR module to be inserted into the overall system. I also hope to begin brainstorming and possibly purchasing the necessary items for the device (to make it wearable).

Verification (Extra Qn):
For the proximity module, I have conducted unit testing to measure objects of known distances away from the sensor. This has been of good accuracy (within +/- 1 cm) for the requirements of our project. This testing has only been unit testing as the implementation of the parallel threads to run the proximity module continually is still buggy, likely due to some timing/synchronization problems. Once I debug these problems I will repeat the verification process and produce a testing dataset to include in our final project report. It is also possible that the bugs I am facing are caused by a hardware problem, perhaps due to residual signals triggered by other programs, and hence Meera and I plan on separating the proximity module from the rest of the system and testing it in an isolated manner (on a breadboard). As we purchased extra pieces of hardware, this testing will not add any additional costs and will help us isolate the problem.

As for the speech module, we have done some basic user-testing to ensure that the rate, tone, volume, etc of the speech output is conducive to the user. We also intend on purchasing bone-conducting headphones so as to test the usability of the speech module in a loud (indoor) environment. The latency of the speech output (which is a part of the initial design requirements we discussed for the performance), I will do the testing once some part of the OR module is integrated into the Jetson.

Accomplishment: This past week I worked on testing the proximity module’s distance detection part. With the PCB design by Meera, I wrote the code to set up the GPIO pin connections and got tests done with the NVIDIA Jetson environment. I have also begun working on the speech to text output functionality, with the plan of running test cases on  my code in the coming days. Here are some images of the work I’ve done this week:

Progress: I need to begin working on the external look of the device – 3D printing/ crafting the case for the device and sourcing neck strap. I also need to finish the Speech module and move it to testing ASAP.

Projected Deliverables:  Testing the speech to text module it the main goal for the coming week, along with looking into building the external look of the device.

Accomplishment: This week I worked on the implementation of the vibration module. This involves setting up the GPIO pins to the various inputs and outputs of the system and using the ultrasonic sensor to calculate the distance between the user and the nearest object. As a recap of the main system design, the ultrasonic sensor is used to detect objects nearest to the user and if the user the vibration setting on (control A was pressed once), this means that they enter the vibration mode and all objects under a 2m radius is indicated to them through the vibration motors. Hence, the vibration module not only measures distance using the sensor, but also includes a program to determine the mode that we are currently using and output the appropriate vibration to the user. Here is the code I have written- as the OR module solidifies, there may be changes to this work. Additionally, the setup of the GPIO pins and any hardware-side code is only a sample set-up and will be replaced with the work done by Meera.

Progress: Completing a first version of the implementation of the proximity module puts me in a good position in terms of the schedule. In the upcoming week I want to implement the TTS engine and test the speech module.

Projected Deliverables: By the end of the week I aim to have a first version of the speech module implemented and also modify the existing proximity module code depending on any changes made to the output of the OR module.

Accomplishment: Completed a first version of the post-processing of data from the OR module to the TTS engine. There will be some changes to this work as the OR module’s distance estimation aspect might undergo some implementation changes as explained in the team report, and Josh’s report. 

 

Progress: I think I’m falling behind a little bit due to a pile up of work from other classes but hope to be back on track by the end of the week.

Projected Deliverables: The goal for the week is to start working on implementing the speech module and vibration modules and integrating it into the NVIDIA jetson environment. This will be only be the initial framework to these 2 modules and will be flexible to support changes made to the OR module.

Accomplishment: I have flushed out an algorithm for the post-processing of data from the OR module, which will then be redirected to the speech module. In each iteration of the OR module’s output, it will provide me with an array of all the objects (by name) identified in the frame under a specific distance threshold (2m). Along with the object’s name, there will be a confidence score and the distance measured attached to the data point. In the post-processing program, I will filter all the identified object from closest to furthest and the closest object will be reported to the user (by means of the speech module) should it pass our confidence score test (ie. confidence > 0.8).  I also have created an interim method to store the past 10s of history in an array that will get referenced should the object not pass the confidence score test. However, all of this is subject to change based on the performance of the OR module that Josh is still working on. Furthermore, as we have decided to upgrade to Yolov9 we expect to have improved performance in the OR model.

 

Progress: I think I am relatively on track. I have decided to hold off on writing out the speech module as the post-processing of the data from the OR module precedes the speech module in the overall data flow.

 

Projected Deliverables: This coming week I will be working on solidifying the post-processing code and hopefully have some testing done on dummy data.

Accomplishment: Looked into NVIDIA Jetson OS and how we can integrate the different modules into the on-board computer. We will be using the NVIDA JetPack SDK that provides linux developer tools to begin building the programs we need for this project. We will use Python as the main language to program the Jetson. As for the speech module, I did some brainstorming on the different possible cases (outputs from the OR module) that will need to be considered when implementing this module.

Progress: I have made some good progress on brainstorming the implementation of the speech module but will need to begin writing some actual code to test out on sample data. This will be my goal for the coming week.

Projected Deliverables: Before Spring break I aim to have more clarity on the speech module and integration into the rest of the device, as well as some code to build off of.

Accomplishment: Settled on the text-to-speech (TTS) engine to convert the OR model’s output into speech dictation should the user use that setting. The TTS engine we will be using is espeak, with the wrapper library pyttsx3 that will provide additional functionality such as adjusting the rate and selecting voice. Also looked into integration of the speech module into the overall device.

Progress: Given the design changes we made, to use the NVIDIA Jetson and forgoing the network-based approach, I believe I am back on track with my progress.

Projected Deliverables: I will next begin researching and developing ways in which to implement the vibration module of our device, which involves the ultrasonic depth sensors that will feed data into a program, which is eventually routed to the vibration motors. This program will act as a mediator between the depth sensors and the vibration motors, allowing us to set a specific threshold for detecting objects (2m for example), along with determining the frequency of the vibrations. I will likely be working alongside Meera, who leads the hardware for the project, towards implementing this module

Accomplishment: Early this week we had our proposal presentation so I spent some hours working on my delivery to ensure that I covered all the key points of our project and kept within time. We held our weekly meeting where we discussed the overall latency of our device and I proposed a plan to implement a network-based design so as to offload the processing from the RPi to a server.

 

Progress: I am a little behind on the research into how doable this network-based approach will be, but I did talk to a friend who has experience on similar work and believe we are capable of handling it. I plan on spending time tomorrow and Monday to flush out a solid execution plan along with system diagrams.

 

Projected Deliverables: By end of the week I hope to have some clarity on this new network-based design choice, along with deciding on some metrics such as how frequently the RPi will send packages to the server. I also will look up on how servers can be hosted over the cloud, as well as brainstorm the design of API endpoints to facilitate the communication.