Reliability issues

Sub-category

Risk Domain

7AI System Safety, Failures & Limitations

7.3Lack of capability or robustness

AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.

"Relying on general-purpose AI products that fail to fulfil their intended function can lead to harm. For example, general- purpose AI systems can make up facts (‘hallucination’), generate erroneous computer code, or provide inaccurate medical information. This can lead to physical and psychological harms to consumers and reputational, financial and legal harms to individuals and organisations."(p. 88)

Entity— Who or what caused the harm

Human

Due to a decision or action made by humans

AI system

Due to a decision or action made by an AI system

Other

Due to some other reason or is ambiguous

Intent— Whether the harm was intentional or accidental

Intentional

Due to an expected outcome from pursuing a goal

Unintentional

Due to an unexpected outcome from pursuing a goal

Other

Without clearly specifying the intentionality

Timing— Whether the risk is pre- or post-deployment

Pre-deployment

Occurring before the AI is deployed

Post-deployment

Occurring after the AI model has been trained and deployed

Other

Without a clearly specified time of occurrence

Supporting Evidence (2)

"Such reliability issues occur because of technical shortcomings or misconceptions about the capabilities and limitations of the technology. For example, reliability issues may stem from technical challenges such as hallucinations, or from users applying systems to unsuitable tasks. Existing guardrails to contain and mitigate reliability issues are not fail- proof."(p. 88)

"Type of reliability issues Examples Confabulations or hallucinations • Citing non- existent precedent in legal briefs (451) • Citing non- existent reduced fare policies for bereaved passengers (452) Common- sense reasoning failures • Failing to perform basic mathematical calculations (453*) • Failing to infer basic causal relationships (454) Contextual knowledge failures • Providing inaccurate medical information (448) • Providing outdated information about events (455)"(p. 90)