Minimax Estimation of Information Measures

We propose a general methodology for the construction and analysis of minimax estimators for functionals of discrete distributions, where the support size 5 is unknown and may he comparable to the number of observations n. We illustrate the merit of our approach by thoroughly analyzing non-asymptotically the performance of the resulting schemes for estimating two important information measures: the entropy H (P) = Sigma(S)(i=1) -p(i) In p(i) and F-alpha (P) Sigma(S)(i=1) p(i)(alpha), alpha > 0. We obtain the minimax L-2 risks for estimating these functionals up to a universal constant. In particular, we demonstrate that our estimator achieves the optimal sample complexity n >> S/ In S for entropy estimation. We also demonstrate that the sample complexity for estimating F-alpha(P), 0 < alpha < perpendicular to is n >> S-1/alpha/In S, which can he achieved by our estimator and not by the popular plug-in Maximum Likelihood Estimator (MLE). For I < alpha < 3/2, we show the minimax L2 rate for estimating F-alpha(P) is (n In n) -2((alpha-1)) regardless of the support size, while the exact L-2 rate for the MLE is n(-2(alpha - 1)). For all the above cases, the behavior of the minimax rate-optimal estimators with a samples is essentially that of the MLE with o In n samples. Finally, we highlight the practical advantages of our schemes for the estimation of entropy and nmtual information.