BackHuman choice of overreliance in critical sectors
Human choice of overreliance in critical sectors
Risk Domain
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
"Heavy reliance on AI in critical sectors like finance or healthcare can exacerbate issues related to size, speed, interconnectivity, and complexity of the system."(p. 13)
Entity: Who or what caused the harm
Intent: Whether the harm was intentional or accidental
Timing: Whether the risk is pre- or post-deployment
Other risks from Uuk et al. (2025) (60)
Control
7.1 AI pursuing its own goals in conflict with human goals or valuesAI systemIntentionalPost-deployment
Democracy
6.0 Socioeconomic & EnvironmentalOtherOtherOther
Discrimination
1.1 Unfair discrimination and misrepresentationOtherOtherPost-deployment
Economy
6.2 Increased inequality and decline in employment qualityOtherOtherPost-deployment
Environment
6.6 Environmental harmAI systemUnintentionalPost-deployment
Governance
6.5 Governance failureAI systemUnintentionalPost-deployment