Posted on

Bhavya’s Status Report for 4/27/24

Given that I was in charge of the final presentation a lot of my time went into working on the slides for the presentation. I made the entire slide deck and spent time practicing my presentation. Given that this was about testing and verification, I spent time with Jae gathering as much data as we could to provide reliable statistics on how our use-case requirements we being met – both coming up with the testing conditions and running the tests themselves.

After giving the presentation on Wednesday these are my ideas on how I could have improved it:

  1. We have a working model. I should have leveraged this to show off a lot more pictures, diagrams, and videos in the presentation. Our project is openCV-based and I think showing the result that each component produces in the computer vision please would have been nice for the viewers.
  2. Kept my explanations more concise and kept on time
  3. Kept it a little more casual and upbeat to engage the audience.

Other than the presentation: I also came up with the idea for the preswitchnig using the ordering of the cameras as the only way we could make the viewing experience more robust after we hit the wall on our detection limits. Given that the cameras cannot detect beyond a certain distance, having the preswitching allows for the viewer to see the car coming into the frame rather than just appearing suddenly. I think this will greatly improve the watching experience. I have been helping refine this idea.

Posted on

Bhavya’s Status Report for 4/20/24

I have made as many changes to the detection algorithm as I think I can given our time constraints. When running our project in different lighting conditions few changes will have to be made to the detection to best fit it for that particular setting. This includes changes to the contour size for mapping and adjusting the initial bounding box when accessing the colors. I could have made the minimum contour size a dynamic feature like I did for getting the number of colors to represent the car by, but our group is more focused on making the stream more watchable and the switching better.

One of the ideas we discussed (to make the switching better) is to use prediction to know where the car was headed. If we knew the car’s direction maybe we could know where it was going and switch preemptively. So I implemented Kalman filters – which allow me to track the card’s trajectory, and constantly update it to get the velocity and acceleration of the car. I then output it as a line of dots that shows the car’s predicted path based on the past few frames.

Unfortunately, since we do not know where the next camera is going to be, this new addition did not prove to be useful for switching. But it could still be helpful for adjusting the camera pan preemptively so that the camera does not lag behind the car – instead allowing the car to constantly be in the center of the frame.

We still have to fully integrate this feature.

Posted on

Team Status Report for 04/27/24

This week was our final presentation in front of our Capstone group. We wanted to get a good working demo before our presentation and possibly have a video to show off our progress. We were able to achieve this!

Given that the report was more so based on our testing, verification, and validation, there was a significant push towards finalizing our testing strategies and collecting data.

Here is a brief outline of our major tests:

  1. Understand how many frames we get per camera for different power levels. This helps us process how well the feed would look for any single-camera
  2. Testing the switching of the camera. This included proximity tests where the car would be moved back and forth between the cameras to test whether the switch would take place as predicted
  3. Detection tests through positioning the car in different orientations and (a few different lighting conditions). These tests did not seem to be conclusive as the number of orientations and lighting conditions can vary a lot. We ideally should have tested at the extremes of the design requirements but reproducing lighting conditions proved to be challenging.

While we could not fully account for different lighting changes we did make a change to accommodate the slower switching process that takes into account what the order of the cameras is. This order is recorded after the initial lap and updated to ensure any camera removals or additions are seamless. (In the case that a camera has to be shut down/ started up mid-race)

To fully accommodate this new change we will need more testing to make sure the ordering works. One key edge case here is when there is rapid switching between the cameras due to detection of noise/ due to true detection of the car, but given their rapidness, it makes the stream unwatchable. Given that quality of stream is an important design requirement, we will need to make some provisions in the switching algorithm to make sure we deal with this edge case.

Posted on

Bhavya’s Status Report for 4/6/24

In preparation for the interim demo, there was a lot of integration testing. We constantly had to test on different track configurations, different lighting conditions and different speeds of the car. I continued to refine the detection and tracking algorithms to based on our tests and the requirements we had for the demo. A key flaw in our system seems to be the lack of memory used to predict car position. I am currently working on integrating kalman filters into the tracking algorithm to use path based trajectory to ensure more correct location of the car.

Posted on

Bhavya’s Status Report for 3/30/24

This week I spent refining my detection algorithm. After moving back to simple color and edge detection (from ML based models due to latency issues) there were challenging edge cases that I had to tackle like if there are other similarly colored objects on the feed and if there are different lighting conditions on the track. Here are the various strategies I used to solve these issues partially

  • The program first requires you to select the car (by letting you draw a box around it) so that it can perform a color analysis.
  • I could have done the color analysis in two ways that I have detailed below. I have implemented both and am still running tests to see which one performs better.
    • Either select the top colors in the box
    • Or selecting the top color and then looking for other similar shades – this method I thought would help in particular with different lighting conditions on the track where different shades could be more prominent. This is done by restricting the range of colors around the most prominent color.
  • Once the top colors are selected I also need to decide how many I need to best represent the car – too many colors makes the masking of the frame useless as it captures a lot of the background. But sometimes a few colors represent the car better than just one. I let the detection algorithm decide by observing how much of the car it could detect without detecting additional environment for different numbers of colors.
  • Once the camera detects the car, the detection algorithm is only permitted to search for the car in its immediate neighborhood in the next run.
  • Testing what is the minimum threshold to qualify for detection that prevents capturing noise. (Changing the minimum contour size that the edge detection provides)
  • Restricting the speed in which the detected box can grow (this will prevent noise from affecting it immediately)

Overall, I can currently track the car pretty well in stable lighting conditions and some color obstacles. However, the system is not completely robust and will require more testing in the final stretch of the project.

Along with this we also performed some integration testing and those details are present in the team report.

Posted on

Team Status Report for 3/23/34

Everything seems to be on track as far as implementations. Although progress has still been slow. A few problems with Git and transferring large files caused problems in collaboration. By using Git LFS we seemed to have worked through that problem.

Secondly, on testing the car speed we felt like it might be too quick for our detection and tracking. As far as the speed is concerned we felt like we had two options: using the car at its full speed or scaling real-world f1 conditions down to the toy track.

Since the current detection (yolo model) and tracking (GOTURN model) seemed too slow for the system at max speed, we decided to pivot to a hybrid between purely color-based tracking and GOTURN. This seems to be able to keep up with the toy set speeds.

There has been progress on the switching algorithms. We still need to receive the connecting pieces for the motors and the cameras to finish the stand.

We aim to have a working demo by Friday and spend the rest of the semester on integrating and fine-tuning the parameters.

Posted on

Bhavya’s Status Report for 3/16/24

I finished creating the detection algorithm. Instead of using the yOLOv4, I used the R-CNN instead. R-CNN typically provides better accuracy by employing region-based convolutional neural networks, which allow for more precise localization of objects in images, albeit at the cost of increased computational complexity during inference. The bounding boxes I was able to create were highly accurate but took a long time to be produced. Ran tests for the detection using static images of the slot car from various angles and distances. Then I integrated the pre-processing, detection, and tracking. Ran tests on a video of the slot car. Currently, the detection algorithm might be too slow, and the preprocessing needs to be tuned after testing which configuration works best for latency. I also have actual footage of the toy car on the track that I will be testing my algorithm on now.

For next week, I will be integrating the code with the live stream offered by our cameras and relaying panning instructions to the motors. Further testing on what type of tracker/detection will be best for our use case will be required.

Given the amount of fine-tuning our system will require for the live stream to be watchable I think we are slightly behind schedule. Sufficient testing in the following week should help put us back on track

Posted on

Bhavya’s Status Report for 03/09/2024

After switching to the F1 track camera idea (post the design presentation) the team had to scramble to establish and work on this new idea.

I started the week off by writing the object tracking algorithm using GOTURN. I also tested several image preprocessing strategies that could possibly reduce the latency of the tracking system. I was able to achieve a preliminary tracking algorithm.

The next major task was the design document submission. I was in charge of the trade studies on computer vision strategies, implementation details of detection and tracking systems, and the outline of the testing and verification in accordance with the use case requirements. We split up the work and conducted peer reviews before finalizing the document.

Over the break, I have been working on the detection algorithm, integrating it with the tracking algorithm, and testing it on the actual slot racing car that we plan to use for the demonstration. I hope to integrate it with the camera to have a one-camera system ready by Wednesday and start arranging the muti-camera system by the end of this week.

Posted on

Bhavya status report for 02/24/2024

After correspondence with Professor Shamos (who has experience in measuring the game of pool) in which I explained the concept of the double hit detection project, I learned that our project would not be feasible in the way we were planning due to the low shutter speed cameras.

This has put our work behind schedule. But the idea we pivoted too will still have some tracking so not all of my work has gone to waste. I simply have to adapt the pool tracking to car tracking now. I hope to have it done by Wednesday to show off a working model.

I helped flesh out the new idea, select materials and set the use case requirements that we aim to fulfil in this new project.

Posted on

Team status report for 02/24/2024

After fully defining our project idea for detecting double hits in a game of 8-ball pool, we went ahead to film shots/fouls and get a better understanding of how these shots work. Our research, accompanied by feedback from a professor who has experience in detecting these types of fouls and other pool-based projects, concluded that it would be extremely challenging to detect double hits with only 240fps cameras. Given that our budget does not allow us to purchase cameras with greater shutter speeds,  we have decided to shift our idea. Even though we had some contingency plans to assist our prediction system (accompanying the cameras), the accuracy we were hoping to achieve would simply not be possible.

Instead, we have decided to proceed with an earlier idea of a multi-camera system for tracking a car around a circuit. Given that we cannot test it on a real car, we will be using a slot car track. Cameras placed at various points on the track will pan as they detect and film the slot car. Our system will also stitch together the footage using the feed from the camera that is closest to the car at any given time to output a seamless live feed. The idea comes from trying to automize the filming of trackside cameras in circuit races eg. F1.

Due to the change in plans, we are significantly behind schedule.  We have updated the Gantt chart to adapt to our new idea.  We have ordered the track and redefined all the design presentation goals for our new idea. To get anywhere close to being on track, we are working on having some working model before Wednesday.

To do so we will be working primarily on the tracking algorithm and assessing the racetrack parameters to understand the speed and camera positions we would need to represent real-world scenarios the best we can.