1. INTRODUCTION
In the digital video broadcasting-satellite services to hand-held (DVB-SH) standard, hierarchical modulation (HM) is selected as an alternative to conventional modulation schemes such as quadrature phase shift keying (QPSK) and quadrature amplitude modulation (16-QAM) [1]. HM is one of the promising technologies to upgrade existing systems with backward compatibility, i.e., receivers that have been designed to use conventional modulation schemes will remain functional by detecting the data in the high priority constellation points [2]–[4]. Furthermore, the additional complexity and cost for HM-based systems are relatively low [5], [6]. However, HM usually suffers from the interference between the enhancement-layer (EL) and basic-layer (BL) streams, which is denoted as inter-layer interference (ILI) [7], [8].