Uncovering the Interpretation of Large Language Models

Recent research has shown growing interest in the arithmetic reasoning capabilities of large language models (LLMs), especially those built on the Transformer architecture. However, our understanding of the intrinsic processes within these models for arithmetic calculations remains scant. This study leverages causal mediation analysis to offer an in-depth look at how Transformer-based LLMs approach complex arithmetic problems. We experimentally intervened on particular model activations and assessed the resulting shifts in prediction probabilities, allowing us to pinpoint which parameters are crucial for such reasoning tasks. We discovered that for complex arithmetic operations, information is channeled from mid-layer activations to the final token through enhanced attention mechanisms. Subsequently, Multi-Layer Perceptrons (MLP) modules synthesize this data, integrating it into the model's residual pathways. To validate these observations, we also evaluated the activation dynamics across different types of tasks, such as retrieving numbers from text and answering fact-based questions.