Tolerance allocation optimization (TAO) is a crucial step in the manufacturing process of the portal milling machine (PMM). Traditional design approaches often consider only the machining errors arising from geometric errors. However, the thermal characteristics of the bi-rotary milling head (BRMH), which significantly machine tool accuracy, are frequently overlooked. Hence, a novel TAO model of PMM is proposed, considering the thermal characteristics of BRMH across multiple operating conditions. Firstly, based on the multi-body system (MBS) theory and homogeneous transform matrix (HTM), a mapping function relating geometric errors to tolerance and volume errors is developed. Secondly, the heat generation and dissipation mechanisms of the BRMH are analyzed, and precise boundary conditions for thermal simulations are determined. Notably, the spin friction torque of the rolling body on the bearing, which contributes to heat generation, must be considered. Finally, the established accuracy design optimization model, which balances total cost and machining accuracy, is solved using the NSGA-II algorithm. The Pareto optimal solution set reveals an 11.74% reduction in total cost and improved machining accuracy for the PMM. Besides, the proposed framework for accuracy design optimization with thermal characteristics is applicable to the manufacturing processes of other machine tools.