Carnegie Mellon ECE Capstone, Spring 2025 | Nicholas Mesa-Cucalon, Joe Sunil Inchodikaran, Lohith Tummala
This week I have been working on finishing integration with the ML model and starting to get the User Survey working. There were a few issues with the original communication protocol, so I am ironing them out and making sure everything is optimized to work with the ML Model. Besides that, I am mainly helping with final deliverables besides the survey and ML model to round out our project.
This week was focused on testing and some last parts of integration. The ML Model has been optimized to better fit the Jetson Platform with TensorRT, and now I am just sorting out some last minute problems with communications with the WebApp, which should be sorted out shortly. I am also beginning to turn my attention to the final deliverables, since we just now need to finish some testing and light optimization for communication between the WebApp and ML Model.
We have tested all of the metrics, and we are in the process of developing the survey. This only requires the full functionality of the web application, so realistically, we could perform the study with just that part functional, but we intend on finishing in the next few days, so having users work with the full product would be nice.
As for unit tests, we have latency measurements for the camera, card detection system, and the processing algorithm. We also have a latency measurement for the full system in total for the total system latency spec of 1s. We have accuracy tests for the card detection and processing algorithm. We intend to perform the survey as well, but do not have results at this time.
I have worked on the poster for the demo and am working on parts of the final report that can be completed prior to the final construction of our product. I have also hot-glued the planks back on, so hopefully, we do not have to worry about the box tipping over again, though even if it does, we can just place the planks back on, this time using tape or a stronger adhesive. In the final days before integration, I may paint the box to give a more visually appealing design, and also to remove any wooden dust off the box.
As a team, we are attempting to work through two main tasks. The first is the integration of the web app and the ML model. As mentioned in individual reports, the web app can now receive data from a local source, so now we are trying to test this process in a real-time system. There are some small issues to consider, such as duplicate cards (which can happen with multiple decks), not double counting cards, etc. We hope to finalize this process by next week, but for now, we can start the second main task, which is testing the metrics. For the presentation, we hope to get most of these metrics done, but for some metrics, such as the user survey, we won’t have results immediately. We have finalized our methodology, and very soon, we will conduct these tests.
I was able to address the changes mentioned in the last status report. I laser cut two long planks and glued them to the bottom of the box, with each of them protruding out in the same direction, so that if the camera were to stick out in that direction, the box would no longer tip over. Some of the planks fell apart, so I plan to hot glue them soon. Also, to further stabilize the camera mount, I laser cut a plank with a hole in it. The camera mounting hole almost perfectly sticks through it, so once I hot glue that to the top, the camera mount should exhibit very limited movement when the user moves the camera. On top of this, I have been working on the slides for the final presentation.
This week I have managed to get almost all of the website applications functionalities up and running. I managed to get the web application to have a decent amount of functionality on phones and tablets. Additionally, it is fully capable of taking streaming updates from a local source. I’m hoping to fully secure it next week and complete integration as a whole next week
Personally, I did not have to deal with streaming updates to a web application in the past when it came to developing web applications. I think my experience this time allowed me to learn a lot when it came to this area. I had to research a lot of new things to properly implement features when it came to this area.
This week was focused on finishing integration and optimizing the ML Model. I was able to integrate the ML model with the Jetson by retraining it on the ECE Clusters on a Python 3.8 environment, and I tested it with TensorRT and it functioned properly. However, something that we anticipated and encountered was how the Yolo11L model was a bit slow for inference, so we retrained for 2 epochs with the Yolo11S model. We will continue to optimize the logic and finish integration with the WebApp over the next week, as we are almost done with the vision model.
I was aware of ways to write CUDA kernels to speed up and optimize inference, but I was not aware of TensorRT for this optimization, which I was made aware of during the 2nd presentation. Overall, the most important skill I have learned throughout this project is how to rapidly prototype Vision Models. I learned how to do so via blog posts and reading other peoples experiences online, through NVIDIA forums and Stack overflow posts. This has greatly decreased the time I need to go from a base idea to a working Vision Model, which I feel has been a great thing to learn throughout Capstone.
We were able to run the ML model on the Jetson Nano at last. We initially thought that this would entail reflashing the microSD card, but it was actually possible with just a few Python installations to get to 3.11. Now, this leaves the final major integration step, which is the transmission of card detections from the ML model to the processing algorithm, which we are working on. We also would like to broadcast the camera feed to the web app so that the user can calibrate the camera. Once we finish these integration steps, we then will start testing the key parameters of our product to assess viability.
Since the last status report and demo, a lot of work has been done on improving the apparatus that houses the Nano and the camera mount. This improved design is made of plywood and stuck together with hot glue. After piecing the structure together, I tested the integrity of the structure, and it seems to be good, except for the case where the camera significantly tilts over the edge, at which point the structure falls. The solution here, which I will implement shortly, is to glue the box to long flat planks that jut out forward. In this case, even if the camera leans forward, the box won’t fall forward.