Information gains from Monte Carlo Markov Chains

In this paper, we present a novel method to compute the relative entropy as well as the expected relative entropy using an MCMC chain. The relative entropy from information theory can be used to quantify differences in posterior distributions of a pair of experiments. In cosmology, the relative entropy has been proposed as an interesting tool for model selection, experiment design, forecasting and measuring information gain from subsequent experiments. In contrast to Gaussian distributions, these quantities are not available analytically and one needs to use numerical methods to estimate them which are computationally very expensive. We propose a method and provide its python package to estimate the relative entropy as well as expected relative entropy from an MCMC sample. We consider the linear Gaussian model to check the accuracy of our code. Our results indicate that the relative error is below 0.2% for sample size larger than 105\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$10^5$$\end{document} in the linear Gaussian model. In addition, we study the robustness of our code in estimating the expected relative entropy in the Gaussian case.