Two-sample Testing of Discrete Distributions under Rare/Weak Perturbations

We propose a method to perform two-sample tests on discrete distributions, i.e., an unsupervised discriminator testing whether two frequency tables were each sampled from a single unspecified probability distribution. Our proposal takes feature-by-feature P-values based on a binomial allocation model, and combines the P-values using Higher Criticism. Performance on real-world data (e.g. authorship attribution challenges) shows this to be an effective unsupervised untrained discriminator even in violations of the binomial allocation model. The method has interesting theoretical properties, in the ‘rare/weak departures’ setting where, if two distributions are actually different, they differ only in relatively few features and only by relatively subtle amounts. We perform a phase diagram analysis in which the phase space quantifies how rare and how weak such departures are. Although our proposal does not require any formal specification of an alternative hypothesis, nor does it require any specification of a baseline or null hypothesis, in the limit where word counts are high, the method delivers the optimal phase diagram in the rare/weak setting: it is asymptotically fully powerful inside the region of phase space where a formally specified test would have been fully powerful. In the limit where counts are low, we derive the phase diagram as well, although the optimality of the resulting diagram is not discussed here.