This page is still being polished. If you have thoughts, please share them via the feedback form.
Data on this page is preliminary and may change. Please do not share or cite these figures publicly.
Runtime monitoring, observability, performance tracking, and anomaly detection in production.
Also in Operations & Security
Developers must therefore implement continuous monitoring of both KRIs and KCIs to ensure that KCI thresholds are met once KRI thresholds are crossed according to the predefined "if-then" statements established in the risk analysis and evaluation phase.
Unlike in some other industries where risks primarily materialize when the final system is deployed (e.g., an aircraft’s safety risks emerge once it starts flying), AI systems can pose risks throughout their development cycle. For instance, loss of control scenarios could materialize during the training process itself, requiring continuous monitoring and risk mitigations well before deployment. This means that capability evaluation is not a one-off affair, but should be repeated regularly during training and during deployment. AI developers should establish rigorous evaluation protocols designed to produce upper bound estimations of AI systems’ capabilities in order to ensure that KRI thresholds are not crossed unnoticed. These protocols should specify the evaluation frequency in terms of both the relative variation of effective computing power used in training and fixed time intervals to account for posttraining enhancements (Anthropic, 2024).6 Evaluations must be performed sufficiently frequently. The elicitation methods used during the evaluations must be comprehensive enough to match the elicitation efforts of potential threat actors. Increased test-time computing power must be included in elicitation efforts. The evaluation environment and methodology must be documented, including specifying how post-training enhancements are factored into capability assessments. Similarly, AI developers should monitor KCIs to ensure that mitigation measures are functioning appropriately and are meeting the KCI thresholds. Independent third parties should vet evaluation protocols. These third parties should also be granted permission and resources to independently perform their evaluations, verifying the accuracy of the results. In addition, AI developers must commit to sharing the evaluation results with relevant stakeholders as appropriate.
Reasoning
Implements continuous capability evaluation protocols with predetermined thresholds throughout training and deployment lifecycle.
Risk Analysis and Evaluation
Risk analysis and evaluation is a process that starts with the definition of a risk tolerance. This risk tolerance is then operationalized into risk indicators and their corresponding mitigations required to reduce risk below the risk tolerance.
2.2.1 Risk AssessmentRisk Analysis and Evaluation > Setting a Risk Tolerance
A risk tolerance represents the aggregate level of risk that society is willing to accept from AI systems.
3 EcosystemRisk Analysis and Evaluation > Operationalizing Risk Tolerance
Risk tolerance must be operationalized into measurable criteria to be practically useful in day-to-day operations. A risk tolerance can be translated into (1) Key Risk Indicator (KRI) thresholds, which are thresholds on measurable signals that serve as proxies for risks, and (2) Key Control Indicator (KCI) thresholds, which are thresholds on measurable signals that serve as proxies for the level of mitigation achieved.
2.2.1 Risk AssessmentRisk Treatment
Risk treatment corresponds to the process of determining, implementing, and evaluating appropriate risk-reducing countermeasures
2.2 Risk & AssuranceRisk Treatment > Implementing Mitigation Measures
AI developers should operationalize their KCI thresholds into mitigation measures.
2.3 Operations & SecurityRisk Governance
Risk governance corresponds to the rules and procedures that structure the risk management system in terms of decision-making, responsibilities, authority, and accountability mechanisms
2.1.2 Roles & AccountabilityA Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Management
Campos, Simeon; Papadatos, Henry; Roger, Fabien; Touzet, Chloé; Quarks, Otter; Murray, Malcolm (2025)
The recent development of powerful AI systems has highlighted the need for robust risk management frameworks in the AI industry. Although companies have begun to implement safety frameworks, current approaches often lack the systematic rigor found in other high-risk industries. This paper presents a comprehensive risk management framework for the development of frontier AI that bridges this gap by integrating established risk management principles with emerging AI-specific practices. The framework consists of four key components: (1) risk identification (through literature review, open-ended red-teaming, and risk modeling), (2) risk analysis and evaluation using quantitative metrics and clearly defined thresholds, (3) risk treatment through mitigation measures such as containment, deployment controls, and assurance processes, and (4) risk governance establishing clear organizational structures and accountability. Drawing from best practices in mature industries such as aviation or nuclear power, while accounting for AI's unique challenges, this framework provides AI developers with actionable guidelines for implementing robust risk management. The paper details how each component should be implemented throughout the life-cycle of the AI system - from planning through deployment - and emphasizes the importance and feasibility of conducting risk management work prior to the final training run to minimize the burden associated with it.
Other (multiple stages)
Applies across multiple lifecycle stages
Developer
Entity that creates, trains, or modifies the AI system
Manage
Prioritising, responding to, and mitigating AI risks
