Maximum entropy versus minimum risk and applications to some classical discrete distributions

The game which can be taken to lie behind the maximum-entropy principle is studied. Refining previous techniques, new theoretical results are obtained. These results are illustrated by concrete examples pertaining to well-known classical models.

I. Introduction and Background Information

Table I A codebook

A	codeword	K
a	100	3
b	1110	4
c	101	3
d	110	3
e	0	1
f	1111	4

Let , the alphabet, be a finite or countably infinite set. The notion of (idealized) codes will play an important role in the sequel. For the purpose of motivation, we remind the reader of the usual notion of a binary prefix-free code. This is a map which to any assigns a binary codeword in such a way that no codeword which appears in this way is a prefix of another such codeword. An example is shown in Table I which displays a codebook for the first six letters of the English alphabet. We may use the code for identification of an unknown letter from or we may conceive the code as a strategy for observation, assuming that an observation is broken down in units of binary questions, and that any such question is feasible. For the example shown, the code points to the question “is the letter one of or ?” as our first question (equivalent to the question “is the first binary digit in the codebook a 1?”). Continuing in this way, enquiring about the further binary digits until identification is possible, we realize that the code length function which is shown in Table I gives us the number of bits needed for identification.