Interpretability for LLM abilities: concepts formation and storage

Interpretability for LLM abilities: in-context learning

A plenty of studies attempt to interpret and disclose the inner mechanisms underlying incontext learning (ICL) (Garg et al., 2022; Kossen et al., 2023; Ren et al., 2024a; Cho et al., 2024). We discuss one interpretability method for ICL along the “feature” research line (mentioned in Section 8.1.1). When an LLM learn to solve specific tasks through context, it can be conceptualized as a computation graph, where circuits are subgraphs composed of linked features and weights that connect them. Similar to how features are concepts’ representational primitive, circuits function as task’s computational primitive (Michaud et al., 2024).

Interpretability for LLM abilities: generalization and emergence of abilities

Recent studies argue that LLMs exhibit emergent abilities, which are absent in smaller models but present in larger-scale models (Schaeffer et al., 2023). Existing research (Power et al., 2022; Doshi et al., 2024), by observing the dynamic training process of models, has identified two important phenomena related to generalization and emergence: grokking and memorization.

Interpretability in model safety auditing

As the application of LLMs in high-stake domains such as healthcare, finance, and law continues to increase, it is imperative to not only assess their accuracy but also scrutinize their safety and reliability (Li et al., 2023e; Liu et al., 2023b). From a societal perspective, the widespread adoption of LLMs across various domains presents potential risks. These risks could arise from a disconnect between LLM developers and users. The former often prioritize technological advancements over practical applications, while the latter may introduce LLMs into their fields without sufficient safety measures or proven success replication. Therefore, Mökander et al. (2023) have proposed that model safety should be audited by third-party entities to rapidly identify risks within LLM systems and issue safety alerts. The auditing process for model safety comprises three steps: • Governance Audit: This involves evaluating the design and dissemination of LLMs to ensure their compliance with relevant legal and ethical standards. • Model Review: This step entails a thorough examination of the LLMs themselves, including aspects such as performance, safety, and fairness. • Application Review: This involves assessing applications based on LLMs to ensure their reliability and safety in practical use.