Maximum entropy estimation of the probability density function from the histogram using order statistic constraints

An analytical expression for a probability density is usually required in detection and estimation problems, yet it is usually only assumed or selected from contenders by parameter estimation, or the histogram is smoothed with an arbitrary window function. In contrast, given a histogram containing R sample points, we derive a nonlinear differential equation (NDEQ) whose solution is a maximum entropy density given constraints that arise from assumptions that the samples are means of the order statistics of the parent distribution. We solve the NDEQ for R=1 and approximate the solution for general R using the fact that order means partition the density into equal probability regions, which we require to independently be maximum entropy. Finally we show with a Rayleigh density example what errors may result.

1. INTRODUCTION

Maximum entropy (Maxent) is an optimization technique that is used to produce analytical functions that have maximum uncertainty while simultaneously satisfying apriori knowledge. If the unknown density (pdf) is and its distribution function (cdf) is , the density of the -th ordered sample [1] from is , where, if is the sample size,

$$f_{(r)}(x)={R!\over (r-1)!(R-r)!}F^{r-1}(x)[1-F(x)]^{R-r}f(x) \eqno{\hbox{(1)}}$$ In the following we maximize the pdf entropy, $$H=-\int_{-\infty}^{\infty}f(x)\log f(x)dx \eqno{\hbox{(2)}}$$ by assuming that the samples are the means of the respective order statistics of the parent pdf. The result is a nonlinear differential equation, which, when , is linear and an exponential pdf is the solution. Further, by utilizing the density-partitioning property of means of order statistics and simply constraining the cdf at the sample points we produce a piecewise contiguous Maxent solution and an example when the parent distribution is Rayleigh.