First-Order Risks

Application

"This is the risk posed by the intended application or use case. It is intuitive that some use cases will be inherently "riskier" than others (e.g., an autonomous weapons system vs. a customer service chatbot)."

Misapplication

This is the risk posed by an ideal system if used for a purpose/in a manner unintended by its creators. In many situations, negative consequences arise when the system is not used in the way or for the purpose it was intended.

Algorithm

"This is the risk of the ML algorithm, model architecture, optimization technique, or other aspects of the training process being unsuitable for the intended application.Since these are key decisions that influence the final ML system, we capture their associated risks separately from design risks, even though they are part of the design process"

Training & validation data

"This is the risk posed by the choice of data used for training and validation."

Robustness

"This is the risk of the system failing or being unable to recover upon encountering invalid, noisy, or out-of-distribution (OOD) inputs."

Design

"This is the risk of system failure due to system design choices or errors."

Implementation

"This is the risk of system failure due to code implementation choices or errors."

Control

This is the difficulty of controlling the ML system

Emergent behavior

"This is the risk resulting from novel behavior acquired through continual learning or self-organization after deployment."