Towards Analysis and Interpretation of Large Language Models for Arithmetic Reasoning

Large Language Models (LLMs) have recently conquered the research scene, with particular regards to the Transformer architecture in the context of arithmetic reasoning. In this so-delineated scenario, this paper puts the basis for a causal mediation analysis about the approach of Transformer-based LLMs to complex arithmetic problems. In particular, we try to discover which parameters are crucial for complex reasoning tasks such as model activations. Our preliminary results state that, for complex arithmetic operations, information is channeled from mid-layer activations to the final token through enhanced attention mechanisms. Preliminary experiments are reported.