Risk Assessment

Demonstrating a “margin of safety” for the worst plausible system failures

Model developers can demonstrate that there is an acceptable “margin of safety” between the current version of the model and a plausible version with dangerous capabilities or potential system failures, whether these arise from the model itself or through scaffolding. This “margin of safety” can be tracked and evaluated based on the model’s performance on either component tasks or proxy tasks with varying levels of difficulty [44], and it is particularly relevant for generalpurpose models with emergent properties, where some of the risks, use cases, and model capabilities may be unknown even at the time of deployment. Margin of safety (also called “safety factor”) is a common practice in many industries - particularly in physical structures. It is common for this margin to be very conservative when feasible (e.g., 4+ in fasteners on critical structures). In situations where a high margin of safety is impractical, it may be supplemented by more frequent inspections and additional process controls. A lower safety factor can also be managed by adopting conservative assumptions regarding worst-case conditions.

Employing qualitative assessments in difficultto-measure domains

Qualitative evaluation can be used in cases when quantitative measurement is not feasible. This can give additional insights about the system which would not be available if no measurement was performed due to its difficulty [206].

Scenario analysis

Scenario analysis involves development of several plausible future scenarios, where these scenarios may be generated from varying the assumptions of a small set of driving forces. The scenarios developed can be used to take further actions to improve overall preparedness [107].

Fishbone diagram

The fishbone diagram, or a “cause-and-effect diagram” [107], can be used to show the potential causes of an undesirable event. The diagram is created by first placing a specific risk event at the “head” of the diagram, typically facing the right. Then, to the left of the risk event, the “ribs” branch off from the “backbone” to represent major causes, which further branch into sub-branches to represent root-causes, extending to as many levels as required. This is typically done via backward-reasoning, where various potential causes are explored after the risk event has been selected for analysis using this method.

Causal mapping

Causal mapping is a technique used to explore and map complex interactions between cause and effect of risks. It involves coming up with potential events related to an undesirable issue, with each event represented by a text box, then clustering similar events according to themes, and finally drawing arrows to illustrate the causal relationship between the different events. The completed causal map can then be analyzed to identify central events, clusters of events, feedback loops, and other relevant patterns [107]

Delphi technique

The Delphi technique is a multi-round forecasting process based on a structured framework on collecting and collating multiple expert judgments. It brings the benefits of anonymous and remote participation which may result in increased likelihood estimation accuracy compared to merely averaging individual estimations or simple group discussions [107]. Given a panel of experts, at each round, the experts are presented with an aggregated summary of the results from the previous round, and are then allowed to update their answers accordingly. The process ends when either a consensus is reached or the responses in later rounds no longer change significantly. This method enables elicitation of expert judgment while utilizing wisdom of the crowd in the process. A potential application of the Delphi technique is to solicit expert judgment on the likelihood of systemic risks from AI development, where crucial variables identified during each round of questionnaire can be further studied for the purpose of risk mitigation.

Cross-impact analysis

Cross impact analysis is a forecasting methodology that analyzes the likelihood of a particular issue using expert analysis (i.e., Delphi technique) in combination with analysis of events correlated with the said issue. It involves decomposing an issue into discrete and correlated events, and then collecting expert opinion on each of those events. Analysis of each event from multiple viewpoints can yield potential future scenarios [107]

Bow-tie analysis

Bow-tie analysis is a method to assess the utility of implemented controls against a particular risk event. It involves centering the unwanted risk event within a diagram. On the left, the factors that can cause the event are listed, followed by the controls that will prevent or minimize the likelihood of the event. On the right, the event is assumed to happen, and the potential effects and the relevant post-hoc controls that could minimize their impact are listed [107].

System-theoretic process analysis (STPA)

STPA is a method to assess the utility of implemented controls against a particular risk within a complex system. Unlike bow-tie analysis, STPA factors in the interactions between components as events that can cause the risk in question [107].

Risk matrices

A risk matrix is a method for risk evaluation. It is a heatmap that, for each cell, shows the severity score weighted by the likelihood score of a particular risk, usually from a scale of 1-5. Two rankings are required to construct a risk matrix: a ranking for the severity of risks, and a corresponding ranking of the likelihood of risks [107]. AI-related risks can be generated using appropriate taxonomies, and placed into the relevant cells according to their assessed likelihood and severity based on predefined criteria (e.g., likelihood level 1 corresponds to < 1% chance, and likelihood level 5 corresponds to > 90% chance; while severity level 1 corresponds to mild inconveniences to the user, and severity level 5 corresponds to a fatality or financial damage upwards of $10 million, etc.), such that particular focus can be given to mitigating risks with higher weighted scores (i.e., likelihood multiplied by severity).

Pre-allocate sufficient resources for risk management

The process of conducting thorough risk management is potentially time-consuming. Pre-allocating sufficient resources, in terms of personnel count and schedule allowances, to conduct necessary risk management activities prior to model deployment is crucial [6].