7.3 Lack of capability or robustness

This subcategory includes the broad set of risks associated with the failure of an AI system to fulfill its intended purpose. The literature identifies four main situations in which an AI may fail to perform as expected or desired.

First, the AI system can fail if it lacks the inherent capability or skill required to perform a task or if this skill is poorly developed. The consequences may be particularly harmful in situations where an AI is required to reason at a human level about important moral issues but does not possess this capability. Cultural, individual, and temporal differences in ideas of what is "right" or "ethical" compound the challenge of endowing AI with appropriate and adaptable ethical standards that are fit for all purposes.

Second, the AI system can fail when it is not robust in "out of distribution (OOD)" situations: data or conditions that were not anticipated during its training phase. These failures may occur because the training data did not confer a particular skill to the AI or because the skill was learned in a fragile way that did not permit generalization to unpredictable and complex real-world environments.

Third, the AI system can fail or become unstable when it is unfit to handle unusual changes or perturbations in input data. These unusual changes could be due to environmental noise, invalid inputs, or adversarial inputs from a malicious attacker.

Fourth, the AI system can fail as a result of oversights, undetected bugs, or errors in the design process. A common design oversight is a lack of comprehensive technical safeguards to prevent unintended downstream uses or consequences. Critical design choices about the algorithm, optimization techniques, and model architecture can also directly influence whether a system is able to consistently perform its intended function, leading to possible harm.

Excerpt from the MIT AI Risk Repository full report