On the Out-of-Distribution Evaluation of ML-Based End-to-End Communications Systems

Machine learning (ML)-aided wireless communication studies are initiating the investigation of the domain generalization capabilities of deep neural networks (DNNs) when applied to communication problems. They do so by adopting the out-of-distribution (OOD) performance evaluation by comparing it to the in-distribution (ID) performance as usually done within the ML community. In this paper, we demonstrate that such blind adoption can yield a misleading OOD performance analysis of DNNs unless wireless communication metrics are involved in the OOD evaluation. By analyzing the OOD performance of an end-to-end (E2E) ML communication system over additive white Gaussian noise (AWGN) channels in terms of bit error rate (BER), we show that smaller (resp. larger) BER gaps between ID and OOD performance do not necessarily translate into a high (resp. low) reconstruction accuracy. Our results suggest that the comparison between ID and OOD performances is not enough to judge whether the OOD performance is acceptable or not. The ID and OOD performances of E2E communication systems should instead be carried out based on wireless metrics.