MMSE hybrid precoder design for millimeter-wave massive MIMO systems

This paper studies hybrid RF/baseband linear pre-coding design to minimize the mean square error (MSE) for millimeter-wave massive multiple-input multiple-output (MIMO) systems using optimal linear equalizer. Instead of dealing with the objective function of sum MSE, which involves matrix inverses, we approach this problem by minimizing the Euclidean distance between the hybrid precoder and the optimal minimum MSE precoder. In an effort to impose the optimal structure of channel diagonalization, we separate the design of modulus-constrained RF precoder from the design of unconstrained baseband pre-coder. Magnitude-least-squares approximation is introduced to formulate the RF precoder design problem, and is subsequently transformed into a simultaneous matrix diagonalization problem. Such transformation enables application of a simple and numerically stable Jacobi-like algorithm. The effective channel representing a cascade of the derived RF precoder and the MIMO channel, is diagonalized by the baseband precoder. The error performance of the proposed solution is examined by numerical results where the effectiveness is verified by its closeness to the optimal design and its noticeable gain over sparse approximation based schemes.