Information conversion, effective samples, and parameter size

Consider the relative entropy between a posterior density for a parameter given a sample and a second posterior density for the same parameter, based on a different model and a different data set. Then the relative entropy can be minimized over the second sample to get a virtual sample that would make the second posterior as close as possible to the first in an informational sense. If the first posterior is based on a dependent dataset and the second posterior uses an independence model, the effective inferential power of the dependent sample is transferred into the independent sample by the optimization. Examples of this optimization are presented for models with nuisance parameters, finite mixture models, and models for correlated data. Our approach is also used to choose the effective parameter size in a Bayesian hierarchical model.

I. Introduction

In recent years and in many fields, ever more datasets have been generated for similar purposes. However, because they are from different sources, they are not directly comparable. In an effort to bring the information in different datasets closer to comparability, we want a generic way to express the information content of one dataset relative to another. The information content of interest may be the equivalent independent sample size for a set of dependent data, or the minimum number of parameters required for good data summarization, or may be a representation of the data in term of a different model. In the latter case, we get a virtual sample that is as similar as possible to the original dataset.