Carnegie Mellon ECE Capstone, Spring 2025: Kaya Akinci, Cynthia Meah, Maya Voelkl
Accomplishments this week:
Worked on debugging, testing, final presentation/report, and implemented one last feature for detecting a step down.
Reflection on schedule:
On schedule!
Plans for next week:
Working on the poster and final report and our final category of testing (differentiation and latency tests).
Accomplishments this week:
This week, we worked on all of our testing, and the only thing we have left to test is latency. We also implemented a feature for detecting downward stairs.
Reflection on schedule:
On schedule!
Plans for next week:
Over the next week, we will be working on the poster, final video, and any last-minute adjustments our device may need.
Risks:
Lower accuracy than anticipated, but no large risks!
Changes:
Changed our algorithm for how we are doing downward stairs detection.
Unit Tests and Overall System:
I will list the test followed by the findings we had from each test:
Object Detection –> Pretty solid all around. We tightened up the range that we accept objects so that it will only detect objects in a shorter range.
Steps –> With data augmentation, our model is very accurate in dim/weird angle areas.
Wall Test –> Mainly very accurate, only inaccurate on slanted walls where the closer wall is in a distance hole (distance of 0).
FSR -> mainly accurate, only inaccurate on carpets
Haptics –> completely accurate
Integration –> mainly accurate, only inaccurate on moments where there is a person on the stairs.
Accomplishments this week:
This week, I worked on performing the extensive tests for our device with Cynthia and Maya. This involved FSR testing, CV testing, haptic testing, wall detection testing, stairs testing, and integration testing. Additionally, we were able to get downstairs detection working with our device.
Reflection on schedule:
on schedule!
Plans for next week:
Work on the poster, final video, and preparing our device for the demo.
Accomplishments this week:
This week, I improved the assembly of the cane by securing everything in a less temporary way. This included some woodworking for our camera holder and some additional aspects to hide our wires and such. I also improved the haptic feedback responses to be much stronger. The second half of the week was spent testing our prototype and putting together parts of our final presentation.
Reflection on schedule:
On schedule!
Plans for next week:
Over the next week, we will be completing the rest of our testing and documentation and getting some materials together for our demonstration. This includes stairs, figuring out lighting, and putting together our poster.
New tools:
On the hardware side, I became familiar with Adafruit’s QT Py RP2040 microcontroller, and the DRV2605L haptic driver, which required us to learn how to communicate between these two devices and with the jetson. Additionally, I learned how to use FSRs to trigger our threshold for the CV.
To gain this knowledge, I relied heavily on informal learning strategies. I used GitHub example code, Adafruit and NVIDIA forums, and documentation pages to understand how each component worked and how to debug integration issues. We also used trial-and-error testing and peer troubleshooting within our team to identify bugs and refine our software logic, especially when integrating the camera, haptic feedback and FSR with our Jetson.
Accomplishments this week:
This week, I worked with Cynthia on improving the accuracy of our object detection model in harsher environments, specifically places with low lighting. This way we did this is by retraining our model but with a learning rate scheduler and data augmentation. After retraining, we did notice better results in the harsher environments. Additionally, I started performing tests to verify the accuracy of our project, specifically the FSR test, the weight test, and the beginning of the CV test.
Reflection on schedule:
On schedule
Plans for next week:
Finish CV testing and working on poster.
New tools:
As I designed the project, some new tools I learned were general linux and OS commands to debug the Jetson errors. Some learning strategies I used to acquire this knowledge were online NVIDIA discussion boards and online tutorial videos.
Accomplishments this week:
I retrained our object detection model by changing fine-tuning parameters to improve performance, such as increasing the starting learning rate for the learning rate scheduler and changes to lower memory usage. Additionally, I performed more complex data transformations to augment part of our dataset to work better in different indoor lightings by editing features like saturation, shadows, and rotations/flips. Additionally, debugged with my teammates, helped Maya with woodworking, started testing with Kaya, and started on our final documentation.
Reflection on schedule:
On schedule!
Plans for next week:
Finish testing and poster.
New tools/knowledge:
As I worked on our project, the main knowledge I had to learn was deep learning techniques to fine-tune and improve model performance and also integration knowledge with our peripherals. The learning strategies I used were applying knowledge from the deep learning class I am currently in and going through forum posts such as stack overflow posts and YOLO support posts of similar problems to ours with fine-tuning. Additionally, I learned how to efficiently go through technologies’ documentation and support websites to learn integration techniques for the technology I have not used before.
Risks:
Lower accuracy than anticipated, but no large risks!
Changes:
Moved to a fine-tuned model with an augmented portion of the dataset and different training parameters.
Accomplishments this week:
This week I spent most of my time debugging the fine-tuning code to make our system faster, which ended up not performing as expected, so further debugging will need to be done (but the current model is still working well, just slightly laggy). I also worked with Kaya to integrate wall detection with our code and get the correct response sent to the user.
Reflection on schedule:
We are on schedule, but because of laggy-ness our project will likely have a lower accuracy than our design requirement.
Plans for next week:
Testing and verification, further debugging, and starting our final report if we have time.
Verification:
I will focus on hazard and stair detection testing.
I will test the model (after removing the display of the frames which has been making the program slower) by analyzing the distance/location accuracy of objects detected, whether hazards vs non-hazards consistency get identified or not identified as expected, and the overall latency of the system from detection to user response with Maya. I will be performing the same analysis for the stairs hazard, with the addition of measuring how accurate the classification of the class stairs is. Note that I will not be testing the accuracy of specific object classifications because the response for different objects which pose as a hazard does not depend on what specific object it is, but on its overall position and size.
For hazard detection, I will perform an equal number of tests on large indoor items (such as tables and chairs), smaller items that should be detected (such as a laptop), and insignificant objects (such as flooring changes) to ensure false positives are not occurring. I will record true positives, false positives, and false negatives (missed hazards), aiming to achieve at least 90% true positive rate and no more than 10% false positive rate across these tests. I will also measure the latency from visual detection to haptic response with Maya, expecting a response time of less than 1 second for real-time feedback.
For stair detection, I will perform tests consisting of different staircases, single-step elevation changes, and flat surfaces used as negative controls (to ensure stairs are not falsely detected). Each group will be tested under varied indoor lighting and angles. The stair classification model will be evaluated on binary detection (stair vs. not stair). I aim to achieve at least 90% stair detection accuracy and 84% accuracy in distinguishing stairs from walls and other obstacles.
Reflection on schedule:
We are right on schedule since the integration has been going smoothly. We should begin testing this upcoming week.
Plans for next week:
Perform extensive tests on each feature of the cane.
Verification: Wall Detection