I. Introduction
Direction of arrival (DOA) estimation of multiple narrowband sources is a major research issue in array signal processing. Many high-resolution DOA estimation methods, such as the famous MUSIC and ESPRIT, have been developed over the years [1]–[3]. However, in real environments, those high-resolution methods may fail to work when there are highly correlated or coherent sources due to multipath propagation, since the methods inherently require the sources to be uncorrelated or lowly correlated. Some preprocessing techniques [4], [5] can be used to decorrelate the coherent source signals, but they usually reduce the effective number of array elements. In [5], a Toeplitz matrix reconstructed from the covariance matrix is used to realize a better DOA estimation no matter whether the sources are coherent or not. A main disadvantage is that in this approach the number of sources can be resolved is at most half of the number of array elements.