Kaitlyn’s Status Report for 4/6/24

Work Done

This week I spent most of the time working on the q-learning model that optimizes the traffic light intervals. I finished the basic model and it is working, however it seems like the model keeps converging to really small intervals for the traffic lights, which I am working on debugging. However, the model technically runs and trains at the moment. I think the problem right now is I might be using the Q-values incorrectly and representing them inaccurately. I plan on taking a deeper look the next couple of days and also trying to represent the action differently.

The code below shows the Agent class that is used to implement Q-learning.

The code below shows the neural network used to implement deep Q-learning. The Q-values are represented by the neural network.

Additionally, I refactored a lot of the existing code to make the utility functions I previously designed more accessible to the Q-learning code. I also made some changes to how the code uses the API data to adjust the simulation, however I realized this led to some bugs when running the simulation for over 5 minutes, so I plan on reverting to the previous API code integration instead. If time allows, I will optimize this further after the other tasks are done.

Schedule

I am mostly on track, however I might need to push the ML code task a day or two longer since I still haven’t had a chance to optimize for hyperparameters. I plan on spending additional time on the project the weeks leading up to the deadline, so I think we will still finish on time.

Tasks this Week

• Finish ML debugging
• Finish ML hyperparameter optimization
• Implement pedestrians in SUMO simulation

Testing and Verification

Reduced wait times: We have a week dedicated to testing where I will first simulate a simulation without any traffic light optimization. I will use the functions that are used by our Q-learning model to calculate average light waiting times at the intersection to then calculate the average light waiting time of 10 different averages over 5 minutes in the simulation and average the 10. I will then repeat the same for 10 averages over 5 minutes using our optimization model and average the 10 to compare the improvements to see if they match our target reduction in wait time.

Safety: I also plan on running the model for a simulation time of an hour to ensure that the system remains working for long periods of time as well as no safety violations occur. SUMO simulates car crashes, so in the event that safety violations do occur due to light intervals being too short, etc, we would be able to adjust our model to fix these concerns.