Bound on the ML Decoding Error Probability for Coded QAM Signals with Shaping

An approach for estimation of the maximum-likelihood (ML) decoding error probability for coded shaped QAM signals is proposed. It is based on approximation of the squared Euclidean distance spectra of the coded shaped signal constellations. Two variants of shaping used in the communication system with coded QAM signaling are studied: shaping before coding (SBC) and shaping after coding (SAC). First, the new approach is verified by applying it to short codes. Then, it is used to derive a random coding bound on the ML decoding error probability for the ensemble of binary linear codes. Simulation results for the frame error rate (FER) performance of belief-propagation decoding for the optimized nonbinary (NB) quasicyclic (QC)-LDPC codes used with SBC and SAC, as well as the same performance of the binary QC-LDPC block codes in the 5G standard are compared to the new bound.