GRIP, Internships Abroad Learning about Reinforcement Learning

Engineering Research in Singapore

As part of the Engineering Research in Singapore program, I am doing research at the National University of Singapore (NUS) for 2 months. The research project I’ve been assigned to is called decentralized traffic signal control for urban mobility under professor Guillaume Sartoretti. However, I work more closely with my assigned Ph.D. student, who has been helpful with kick-starting my project.

Though this project is under the Mechanical Engineering department, I have found that it aligns very well with my major, which is Systems Engineering at Penn. Up until now, my tasks have consisted of getting more familiar with the python programming language, familiarizing myself with some of the libraries extensively used in the project which includes PyTorch and TensorFlow, creating small projects with SUMO (simulation for urban mobility) software, and learning the overall structure of reinforcement learning.

The overarching problem this project helps address is optimizing urban traffic – such as decreasing wait times, vehicle queue lengths, increasing vehicular throughput, and much more – which has many implications in the real world. This includes increasing efficiency when traveling, lowering traffic congestion, cutting costs, and lowering fossil fuel emissions that contribute to climate change. After taking the course Engineering Probability (ESE 3010) this past spring semester, I am able to have a better understanding of much of the math going on behind the algorithm implementations. Concepts such as wait times and using the binomial distribution, and methods of random sampling are some connections to this course.

The main project I am assigned to is code integration. Our lab has developed a distributed reinforcement learning algorithm for traffic optimization in a Manhattan environment; however, we want to integrate this code to be compatible with other benchmark environments. We also want to be able to run the algorithm besides the other existing state-of-the-art RL algorithms. By doing this, we can examine how our algorithm performs in a variety of different benchmark environments and compare its performance directly with the existing algorithms.

My assigned Ph.D. student has been supportive in getting me on board with the project by providing me with many resources ranging from relevant research papers to online tutorials to university courses to blog posts. A lot of what I’m doing is learning independently and figuring things out, which is an important skill in any discipline. The stuff I’ve been learning is super interesting, and I look forward to diving in even deeper.