Multiresponse surface methodology for hyperparameter tuning to optimize multiple performance measures of statistical and machine learning algorithms

Hyperparameter tuning is an important task in machine learning for controlling model complexity and improving prediction performance. Most methods in the literature can only be used to tune hyperparameters to optimize a single performance measure. In practice, participants may want to optimize multiple measures of the model; however, optimizing one measure may worsen another. Therefore, a hyperparameter tuning method is proposed using the multiresponse surface methodology to solve the tradeoff problem between multiple measures. A search algorithm that requires fewer tuning runs is developed to find the optimal hyperparameter settings systematically based on the preferences of participants for multiple measures of the model. An example using the random forest algorithm is provided to demonstrate the application of the proposed method to improve the prediction performance of the model on unbalanced data.