Effective data screening technique for crowdsourced speech intelligibility experiments: Evaluation with IRM-based speech enhancement

It is essential to perform speech intelligibility (SI) experiments with human listeners in order to evaluate objective intelligibility measures for developing effective speech enhance-ment and noise reduction algorithms. Recently, crowdsourced remote testing has become a popular means for collecting a massive amount and variety of data at a relatively small cost and in a short time. However, careful data screening is essential for attaining reliable SI data. We performed SI experiments on enhanced speech in a well-controlled laboratory and in crowdsourced remote environments that could not be controlled directly. The target speech sounds were enhanced by two techniques, a single-channel “oracle” ideal ratio mask (IRM) and a multi-channel mask-based beamformer. We introduced simple tone pip tests, in which participants were asked to report the number of audible tone pips, to estimate their listening levels above audible thresholds. The tone pip tests were very effective for data screening to reduce the variability of crowdsourced remote results so that the laboratory results would become similar. The results also demonstrated the SI of an oracle IRM, giving us the upper limit of the mask-based single-channel speech enhancement.