Entropy and Information Content of Geostatistical Models

Geostatistical models quantify spatial relations between model parameters and can be used to estimate and simulate properties away from known observations. The underlying statistical model, quantified through a joint probability density, most often consists of both an assumed statistical model and the specific choice of algorithm, including tuning parameters controlling the algorithm. Here, a theory is developed that allows one to compute the entropy of the underlying multivariate probability density when sampled using sequential simulation. The self-information of a single realization can be computed as the sum of the conditional self-information. The entropy is the average of the self-information obtained for many independent realizations. For discrete probability mass functions, a measure of the effective number of free model parameters, implied by a specific choice of probability mass function, is proposed. Through a few examples, the entropy measure is used to quantify the information content related to different choices of simulation algorithms and tuning parameters.