I. Introduction
Directional of arrival (DOA) estimation is a fundamental issue for array signal processing and has received extensive concentration in radar, sonar, aerospace, and wireless communications [1], [2], [3]. For decades, various algorithms have been proposed for DOA estimation, such as the MUSIC method [4] and the ESPRIT approach [5]. Then, an improved beamspace root-MUSIC algorithm [6] is proposed to reduce the computational complexity. In [7], [8], spatially smoothed decoherence algorithms are proposed for two-dimensional (2-D) DOA estimation. Although the two algorithms are capable of decoherence, they also have high computational complexity and sacrifice array aperture. In [9], the unitary ESPRIT and beamspace ESPRIT algorithms are proposed for uniform rectangular arrays (URAs), which can estimate 2-D DOA simultaneously and have lower computational complexity than the MUSIC algorithm. The algorithms above are all based on the uniform linear array (ULA) structure, and the inter-element spacing is less than or equal to the half wavelength. However, the simple array structure usually causes a mutual coupling problem between the array elements. In addition, limited by the aperture of the arrays, the DOA estimation performance cannot be further improved. MIMO radar is proposed in [10] to increase the array aperture, and the subspace-based algorithm can be extended to MIMO radar to obtain a better DOA estimation performance. Regardless, MIMO radar still cannot avoid the mutual coupling problem.