On the False Alarm Probability of the Normalized Matched Filter for Off-Grid Targets: A Geometrical Approach and Its Validity Conditions

Off-grid targets are known to induce a mismatch that dramatically impacts the detection probability of the popular Normalized Matched Filter. To overcome this problem, the unknown target parameter is usually estimated through a Maximum Likelihood strategy, resulting in a GLRT (Generalized Likelihood Ratio Test) detection scheme. While the test statistic for the null hypothesis is well known in the on-grid case, the off-grid scenario is more involved, and to the best of our knowledge, no such theoretical result is available. This paper fills this gap by proposing such an expression under circular compound Gaussian noise with a known covariance matrix thanks to a geometrical approach. It is exact, provided that the probability of false alarm is low enough: we derive the conditions guaranteeing the exactness of the relationship, and we show numerically that the formula still yields a good approximation of the probability of false alarm when this condition is not met.