Metrology for AI risk assessment

Risk Assessment

The primary goal of risk assessment is to understand the severity and likelihood of a potential harm. Risk assessments are used to prioritise risks and determine if they cross thresholds that demand specific action. Consequential development and deployment decisions are predicated on these assessments. The research areas in this category involve: A. Developing methods to measure the impact of AI systems for both current and future AI – This includes developing standardised assessments for risky behaviours of AI systems through audit techniques and benchmarks, evaluation and assessment of new capabilities, including potentially dangerous ones; and for real-world societal impact such as labour, misinformation and privacy through field tests and prospective risks analysis. B. Enhancing metrology to ensure that the measurements are precise and repeatable – This includes research in technical methods for quantitative risk assessment tailored to AI systems to reduce uncertainty and the need for large safety margins. This is an important open area of research. C. Building enablers for third-party audits to support independent validation of risk assessments – This includes developing secure infrastructure that enables thorough evaluation while protecting intellectual property, including preventing model theft.

Risk Assessment > Audit techniques and benchmarks

Techniques and benchmarks with which AI systems can be effectively and efficiently tested for harmful behaviours are highly varied and central to risk assessments (IAISR, Birhane-A).

Risk Assessment > Downstream impact assessment and forecasting

Assessing and forecasting the many societal impacts of AI systems is one of the most central goals of risk assessments.

Risk Assessment > Secure evaluation infrastructure

External auditors and oversight bodies need infrastructure and protocols that enable thorough evaluation while protecting sensitive intellectual property. Ideally, evaluation infrastructure should enable double-blindness: the evaluator’s inability to directly access the system’s parameters and developers’ inability to know what exact evaluations are run (Reuel, Bucknall-A, Casper-B). Meanwhile, the importance of mutual security will continue to grow as system capabilities and risks increase. Methods for developing secure infrastructure for auditing and oversight are known to be possible.

Risk Assessment > System safety assessment

Safety assessment is not just about individual AI systems, but also their interaction with the rest of the world. For example, when an AI company discovers concerning behaviour from their system, the resulting risks depend, in part, on having internal processes in place to escalate the issue to senior leadership and work to mitigate the risks. System safety considers both AI systems and the broader context that they are deployed in. The study of system safety focuses on the interactions between different technical components as well as processes and incentives in an organisation (IAISR, Hendrycks-B, AISES, Alaga).

Risk Assessment > Dangerous capability and propensity assessment

To assess certain hazards posed by an AI system, it is necessary to elicit and assess potentially dangerous capabilities (Phuong, Shevlane, Anthropic-B, IAISR) including dual-use cyber, chemical, biological, and nuclear knowledge, as well as capabilities for psychological manipulation, AI research and development, and autonomy which increases the risk of loss of control (see below). To assess the likelihood that these capabilities will cause harm, it is necessary to assess the system’s propensities to use them.