NONCONVEXITIES IN A STOCHASTIC-CONTROL PROBLEM WITH LEARNING

This paper examines the benefits from active learning in a stochastic control problem. In a linear model with parametric uncertainty, there are gains to probing, but the probing component of the loss function often has nonconvexities. I show that they can arise for two reasons: (1) failure of the precision matrix of the parameters to increase monotonically with the control variable (the covariances between a state variable and the random parameters can reduce the information gained from probing) and (2) changes in the path of future state variables induced by modifying the certainty-equivalent control. If the parameter on the control variable is large, a small change in the control can cause a much larger change in future state which, for a given level of uncertainty, makes probing more costly. © 1991.