consider the optimal routing problem in a discrete-time system with a job dispatcher connected to M parallel servers. At every time slot, the job dispatcher sends the incoming jobs to a server for execution, with each server having a queue that stores the jobs. The arrival process of incoming jobs, and the service processes of the servers are stochastic with unknown and possibly heterogeneous rates. Each server sm is associated with an underlying utility vm that is initially unknown. Whenever server sm completes a job, a utility of vm is obtained and a noisy observation of vm is received. The goal is to design a policy that makes routing decisions to maximize the total utility obtained by the end of a finite time horizon T. The performance of policies is measured in terms of regret, which is the additive difference between the expected total utility obtained by the policy and the supremum of the expected total utility over all the policies. The optimal routing problem can be interpreted as a problem of multi-armed bandit with queues where each server is viewed as an arm and the completion of a job is viewed as a pull of an arm. The key distinction between the optimal routing problem and traditional multi-armed bandit problems is in the queueing dynamics at the server, which arises due to the stochastic nature of the arrival and service processes. Our results combine techniques from control of stochastic queueing systems and stochastic multi-armed bandits to provide insights to the design and analysis of policies for the optimal routing problem. We first present analytical bounds that link the regret to the utilization and queue length of servers. Next, we start by assuming that the ordering of the underlying utilities is known and introduce the Priority-K routing policy which makes priority-based routing decisions that send the incoming jobs to the server of the highest underlying utility with queue length no larger than a threshold K. We prove that Priority-K achieves O (log T )-regret with an appropriately chosen K. Next, removing the assumption of known utility ordering, we pro -pose the Upper-Confidence Priority-K policy, which essentially combines the Priority-K policy with the ordering based on the upper-confidence bounds of the underlying utilities, and establish that the Upper-Confidence Priority-K policy achieves an instance-dependent O(log(3) T )-regret. Finally, we extend our results to the a generalized version of the optimal routing problem with multiple job dispatchers in a bipartite network. Our theoretical results are also validated by simulations.
