I. Introduction
Multiple-input multiple-output (MIMO) antenna technique can significantly improve the capacity of a wireless system with respect to the number of antennas [1], [2]. If channel state information at the transmitter (CSIT) is available, the capacity can be simply achieved by the linear singular value decomposition (SVD) beamforming and water-filling power allocation [3], with no need to consider complex algorithms such as the joint maximum likelihood (ML) decoding. In practice, however, the acquisition of CSIT is often quite challenging due to limited feedback or channel variation. In the lack of CSIT, the performance of conventional linear receivers such as the minimum mean-squared error (MMSE) receiver becomes far apart from the capacity, especially when the channel matrix is near-singular [4]–[7]. Non-linear MIMO schemes such as sphere decoding can provide an enhanced performance and its complexity reduction algorithms have also been proposed in [8]–[11]. Their complexities, however, are yet considerably high to be applicable to practical systems, especially when the number of antennas is large and/or the modulation order is high. Note that in the current Long Term Evolution-Advanced (LTE-Advanced) system, the fourth generation (4G) communications system, up to eight streams with 256-quadrature amplitude modulation (QAM) are supported for high data rate transmission, and it is highly likely that the number of supported streams and/or the supported modulation order will further increase in the next fifth generation (5G) communications system. Hence, it becomes more and more important to design a practical MIMO receiver that can support high spectral efficiency with low complexity.