I. Introduction
Ideally, a lossless or lossy source coder would compress data into an independent, identically distributed (i.i.d.) nearly uniform bit-stream (for sufficiently long blocklengths). However, most practical source coding methods are not ideal; hence there exists a residual redundancy (in the form of non-uniform distribution and/or memory) at their output which will be present at the input of the channel encoder. For example, the line spectral parameters at the output of codebook-excited linear predictive (CELP) speech vocoders may contain up to 42% of (residual) redundancy due to non-uniformity and memory (see, e.g., [3]). Another example is the bit-stream at the output of vector quantizers with moderate blocklengths. Furthermore, natural data sources, which in certain complexity-constrained applications (e.g., wireless sensor networks) are transmitted uncompressed over the channel, exhibit even higher amounts of redundancy. For example, binary images may contain as much as 80% of redundancy due to non-uniformity; this translates into a probability as high as 97% for having a “0” (as opposed to a “1”) in the image bit-stream (see, e.g., [34] and the references therein).