Risk Assessment

Developing Trustworthy, Secure and Reliable Systems

AI systems that are trustworthy, reliable and secure by design give people the confidence to embrace and adopt AI innovation. Following a classic safety engineering framework, the research areas in this category involves: A. Specifying and validating the desired behaviour – This includes technical methods to address the complex challenges in specifying system behaviours in a way that accurately captures the desired intent without causing undesired side effects, for both singlestakeholder settings (e.g. reward hacking, scalable methods to discover specification loopholes) and multi-stakeholder settings (e.g. balancing competing preferences, ethical and legal alignment). B. Designing a system that meets the specification – This covers techniques for training models – both closed and open weights – that are trustworthy (e.g. reducing confabulation, increasing robustness against tampering), alternative finetuning methods to make specific precise changes to an AI system (e.g. model editing), and methods to build AI systems in a way that are guaranteed to meet their specifications (e.g. verifiable programme synthesis, world models with formal guarantees). C. Verifying that the AI system meets its specification – This entails techniques to provide high-confidence assurances that an AI system meets its specifications (e.g. formal verification), including in novel contexts (e.g. robustness testing), as well as interpretability techniques to look into the black box to understand why the AI system behaves the way it does (e.g. mechanistic interpretability).

Developing Trustworthy, Secure and Reliable Systems > Specification & Validation: Defining the system’s purpose

Specification involves defining desired system behaviour, whereas validation ensures that the specification meets the needs of the user, developer, or society – did I build the right system? In other words, specification and validation require confronting the complexity of defining objectives in a way that captures user or societal intent without omitting important constraints or causing undesired side effects, as well as dealing with disagreement and tradeoffs between diverse stakeholders.

Developing Trustworthy, Secure and Reliable Systems > Design and implementation: Building the system

This section focuses on techniques to make systems that meet their specifications. The design and implementation process involves sourcing data, pretraining models, post-training models, and integrating them into AI systems.

Developing Trustworthy, Secure and Reliable Systems > Verification: Assessing if the system works as specified

The research areas described in this section aim to assess the extent to which the built system (2.2) meets its specifications (2.1). This section discusses several broad types of techniques that can be used to provide evidence that a system is safe. In practice, the effectiveness of these methods is often limited by access and transparency, but they can play a central role in constructing AI safety cases: structured arguments for why systems pose an acceptably low level of risk (Clymer, Buhl).

Control: Monitoring & Intervention

In engineering, “control” usually refers to the process of managing a system’s behaviour to achieve a desired outcome, even when faced with disturbances or uncertainties, and often in a feedback loop. The research areas in this category involve: A. Developing monitoring and intervention mechanisms for AI systems – This includes adapting conventional methods for monitoring (e.g. hardware -enabled mechanisms, user monitoring) and intervention (e.g. off-switches, override protocols), as well as designing new techniques for controlling very powerful AI systems that may actively undermine attempts to control them (e.g. scalable oversight, containment). B. Extending monitoring mechanisms to the broader AI ecosystem to which the AI system belongs – This entails methods to support the identification and tracking of AI systems and data (e.g. logging infrastructure, data provenance, model provenance). In turn, this can facilitate accountability infrastructure and enable more informed governance. C. Societal resilience research to strengthen societal infrastructure against AI-enabled disruption and misuse – This section studies how institutions and norms (e.g. economic, security) can adapt as future AI systems come to act as autonomous entities, as well as incident response mechanisms to enable clear and rapid coordination among relevant actors to detect, respond to, and recover from accidents or misuse of advanced AI systems.

Control: Monitoring & Intervention > AI system monitoring

In emerging technical fields, new systems cannot be reasonably expected to always behave as intended and impact society as expected. Monitoring techniques play an essential role in the iterative process of identifying, understanding, and fixing problems as they emerge.