Interpretability for LLM abilities: in-context learning

Definitionp. 65

A plenty of studies attempt to interpret and disclose the inner mechanisms underlying incontext learning (ICL) (Garg et al., 2022; Kossen et al., 2023; Ren et al., 2024a; Cho et al., 2024). We discuss one interpretability method for ICL along the “feature” research line (mentioned in Section 8.1.1). When an LLM learn to solve specific tasks through context, it can be conceptualized as a computation graph, where circuits are subgraphs composed of linked features and weights that connect them. Similar to how features are concepts’ representational primitive, circuits function as task’s computational primitive (Michaud et al., 2024).

Additional Informationp. 65

Induction heads are a type of circuits within LLMs that are believed to be critical for enabling in-context learning abilities (Olsson et al., 2022). These circuits function by performing prefix matching and copying previously occurring sequences. An induction head consists of two attention heads working together: • Prefix-Matching Head: The first attention head, located in a previous layer of the model, attends to prior tokens that are followed by the current token. This means it scans the sequence for earlier instances where the current token appears immediately after certain tokens, effectively performing prefix matching. This process identifies the “attend-to” token, which is the token that follows the current token in those previous occurrences. • Copying Head (Induction Head): The second attention head, known as the induction head, takes the “attend-to” token identified by the first head and copies it, increasing its output logits. By boosting the likelihood of this token in the model’s output, the induction head extends the recognized sequence, effectively enabling the model to predict and generate sequences that mirror previously observed patterns. Through the collaboration of these two heads, induction heads enable LLMs to recognize and replicate patterns within the input sequence, which is a fundamental aspect of in-context learning. Elhage et al. (2021) find two types of circuits in the transformer: i) “query-key” (QK) circuits; ii) “output-value” (OV) circuits, which are crucial for knowledge retrieval and updating. The QK circuits play a crucial role in determining which previously learned token to copy information from. Conversely, the OV circuits determine how the current token influences the output logits.

Part of