Interpretability

Mechanistic interpretability

Techniques for understanding how models function and represent concepts internally uniquely allow for assessments of internal model cognition. These techniques could aid in the discovery of system properties or, if thorough enough, aid in constructing safety cases (Clymer, Buhl) for them (Sharkey). For example, mechanistic interpretability techniques might be able to help researchers characterise and intervene on model representations that correspond to harmful concepts such as deception or malice. Current research frontiers in mechanistic interpretability involve developing scalable techniques that beat black-box baselines for identifying and addressing flaws in systems. Mechanistic understanding of models could also help verify the success of other methods which are imperfect, such as unlearning of dangerous capabilities (see above) and analysing written ‘chains-of-thought’ (see below).

Explainability

Explainability techniques refer to methods that allow model behaviours to be attributed to specific features in their inputs. They can be useful for both diagnosing system errors and determining accountability for system failures (Gryz, Casper-B). However, current explainability tools are often unreliable (Bordt), highlighting the value of future work to improve on existing tools.

LLM chain-of-thought faithfulness and legibility

Large language model chain-ofthought reasoning does not always faithfully represent how a model arrived at its answer (Turpin). This poses challenges to safety because, without faithful reasoning, models could fool overseers by saying one thing and doing another. For example, language models have stated that they gave their answer based on a logical argument when they actually chose it based on hints that they should not have exploited (Anthropic-D, Turpin), such as seeing that the correct answer is always “B”. One potential challenge with chain-of-thought monitoring stems from how, under optimisation pressure on their reasoning, systems may learn to obfuscate their reasoning in ways that can be actively misleading (OpenAI-E, see also 2.1.1 above for an example).

Attributing model behaviours to training data

Methods for attributing model behaviours to specific examples from training data allow overseers to study how potentially harmful behaviours emerge in systems (Grosse). These tools could also help researchers identify what types of training interventions can mitigate them. For example, attributing control-subverting behaviours to specific examples from training data could help developers curate safer pretraining datasets. Research frontiers include improving the efficiency and scalability of these methods, causally studying how models develop personas and behaviours (Anthropic-F), and predicting what data is needed to learn a particular behaviour (Engstrom, Ilyas).

Studying goals in systems

Increasingly agentic AI systems are characterised by increasingly goal-oriented behaviour. As a result, studying the emergence and mechanisms behind these behaviours offers a way for researchers to study the system’s alignment with its specification (Ngo). However, it is challenging to study goals in AI systems because they cannot be inspected directly and their behaviour is sometimes but not always consistent with coherent principles (Khan, Mazeika). Directions for future work involve developing concrete definitions and measures of goals in AI systems (e.g. MacDermott) and interpreting how AI systems develop and represent goals internally (e.g. Marks).