Kernel-based Reinforcement Learning for Traffic Signal Control with Adaptive Feature Selection

Reinforcement learning in a large-scale system is computationally challenging due to the curse of the dimensionality. One approach is to approximate the Q-function as a function of a state-action related feature vector, then learn the parameters instead. Although assumptions from the priori knowledge can potentially explore an appropriate feature vector, selecting a biased one that insufficiently represents the system usually leads to the poor learning performance. To avoid this disadvantage, this paper introduces kernel methods to implicitly propose a learnable feature vector instead of a pre-selected one. More specifically, the feature vector is estimated from a reference set which contains all critical state-action pairs observed so far, and it can be updated by either adding a new pair or replace an existing one in the reference set. Thus the approximate Q-function keeps adjusting itself as the knowledge about the system accumulates via observations. Our algorithm is designed in both batch mode and online mode in the context of the traffic signal control. In addition, the convergence of this algorithm is experimentally supported. Furthermore, some regularization methods are proposed to avoid overfitting of Q-function on the noisy observations. Finally, A simulation on the traffic signal control in a single intersection is provided, and the performance of this algorithm is compared with Q-learning, in which the Q-function is numerically estimated for each state-action pair without approximation.