Expert Severity Validation
Comparing expert BAU severity assessments (how severe experts predict each risk subdomain will be) against observed incident severity (average direct harm across real-world incidents). Each dot represents one subdomain, positioned by its standardized prediction gap — both scales are z-score normalized before comparison. Positive values mean experts rate a subdomain as relatively more severe than incidents show; negative values mean incidents are relatively worse than predicted.
How to read this
Each dot is one AI risk subdomain, positioned on a horizontal axis by its standardized prediction gap (z-score of expert severity − z-score of incident severity). Dots to the right of zero mean experts predicted higher severity than incidents showed (overprediction); dots to the left of zero mean observed incidents were more severe than predicted. In Grouped view, one strip per domain; in Sorted view, all subdomains ranked from largest to smallest gap. Translucent dots have fewer than 5 qualifying incidents.
Key Takeaways
- 1.3.1 False or misleading information has the largest relative overprediction (+2.18σ) — experts rate this subdomain as relatively more severe than incidents show.
- 2.6.6 Environmental harm has the largest relative underprediction (-2.69σ) — incidents are relatively more severe than expert predictions.
- 3.12 of 21 subdomains show experts rating relatively higher severity; 9 show incidents relatively exceeding predictions.
- 4.21 of 24 subdomains have qualifying incident data. 3 excluded (no severity data) and 6 have small samples (<5 incidents).
Both scales are z-score normalized (μ=0, σ=1) before comparison to account for different measurement scales.