Cross-Silo Federated Training in the Cloud with Diversity Scaling and Semi-Supervised Learning

Federated learning is a machine learning approach that allows a loose federation of trainers to collaboratively improve a shared model, while making minimum assumptions on central availability of data. In cross-siloed federated learning, data is partitioned into silos, each with an associated trainer. This work presents results from training an end-to-end ASR model with cross-silo federated learning system. We propose a novel aggregation algorithm that takes update diversity into account and significantly outperforms Federated Averaging (FedAvg). The system design used in this paper allows joint training with human transcribed and semi-supervised (SSL) data, yielding 7.6% relative word error rate reduction on head test set and 13.9% on tail test set, when using 20kHr of SSL data. Gains further improve to 13.8% and 20.5% respectively when SSL data is increased from 20kHr to 200kHr.