Learning and Selection of Pareto Optimal Policies Matching User Preferences

Reinforcement learning, which is attracting attention as a method for optimizing sequential decision-making, primarily focuses on scenarios with a single objective. In contrast, real-world decision-making involves multiple objectives, often with trade-off relationships among these objectives. Multi-Objective Reinforcement Learning (MORL) addresses the sequential decision-making process with multiple objectives. Previous research in MORL has mainly focused on learning the set of Pareto optimal policies, known as the Pareto front. However, these studies do not discuss methods for selecting policies from the Pareto front that align with user preferences. The utility values of each policy in the Pareto front may not necessarily align with user preferences. Therefore, an evaluation metric illustrating the relationship between user preferences and policies is necessary to select the policy that matches user preferences. We introduce a MORL method that incorporates "Mismatch" metric to evaluate the similarity between preferences and policies. The proposed method treats Mismatch metric as a penalty term in the evaluation function and learns the Pareto front using MORL with an evolutionary algorithm. Users can specify preference weight parameters and select policies from the learned Pareto front that minimize Mismatch, aligning with their preferences. We verified the performance of the proposed method by computer experiments in a multi-objective reinforcement learning benchmark environment.