On the Impact of Soft Errors on Transformers Embeddings

Transformers are widely utilized in Natural Language Processing (NLP) and computer vision (CV) with remarkable success. For pre-trained Transformer models involving text processing, embedding representations are indispensable parameters, and occupy a large volume of memory. Soft errors on embedding vectors can lead to incorrect inputs to Transformers, which may further result in a loss of model performance. This paper takes Contrastive Language-Image Pre-training model (CLIP) for image classification as a case study to examine the impact of soft errors on embedding vectors that are represented by half-precision floating point values. Two error models are used for the injection experiments; the first model is a one-time injection with specific error rates, and the second model considers the accumulated errors with multiple injections. Simulation results show that for one-time error injection, only the first two exponent bits are critical such that errors on them would cause an obvious performance loss; the third exponent bit becomes critical for accumulated errors. These results are valuable for the efficient protection and design of a memory for the embedding vectors.