AI systems acting in conflict with human goals or values, especially the goals of designers or users, or ethical standards. These misaligned behaviors may be introduced by humans during design and development, such as through reward hacking and goal misgeneralisation, or may result from AI using dangerous capabilities such as manipulation, deception, situational awareness to seek power, self-proliferate, or achieve other goals.
"even if an agent started working to achieve an unintended goal, this would not necessarily be a problem, as long as we had enough power to prevent any harmful actions it wanted to attempt. Therefore, another important way in which we might lose control of AIs is if they start trying to obtain more power, potentially transcending our own."(p. 38)
Supporting Evidence (6)
"AIs might seek to increase their own power as an instrumental goal... While the idea of power-seeking often evokes an image of “power-hungry” people pursuing it for its own sake, power is often simply an instrumental goal. The ability to control one’s environment can be useful for a wide range of purposes: good, bad, and neutral. Even if an individual’s only goal is simply self-preservation, if they are at risk of being attacked by others, and if they cannot rely on others to retaliate against attackers, then it often makes sense to seek power to help avoid being harmed—no animus dominandi or lust for power is required for power-seeking behavior to emerge [123]....AIs trained through reinforcement learning have already developed instrumental goals including tool-use...Self-preservation could be instrumentally rational even for the most trivial tasks"(p. 38)
"AIs given ambitious goals with little supervision may be especially likely to seek power. While power could be useful in achieving almost any task, in practice, some goals are more likely to inspire power-seeking tendencies than others. AIs with simple, easily achievable goals might not benefit much from additional control of their surroundings. However, if agents are given more ambitious goals, it might be instrumentally rational to seek more control of their environment. "(p. 39)
"Unlike other hazards, AIs with goals separate from ours would be actively adversarial. It is possible, for example, that rogue AIs might make many backup variations of themselves, in case humans were to deactivate some of them."(p. 39)
"Some people might develop power-seeking AIs with malicious intent. "(p. 39)
"There will also be strong incentives for many people to deploy powerful AIs. Companies may feel compelled to give capable AIs more tasks, to obtain an advantage over competitors, or simply to keep up with them. It will be more difficult to build perfectly aligned AIs than to build imperfectly aligned AIs that are still superficially attractive to deploy for their capabilities, particularly under competitive pressures"(p. 39)
"If an agent repeatedly found that increasing its power correlated with achieving a task and optimizing its reward function, then additional power could change from an instrumental goal into an intrinsic one, through the process of intrinsification discussed above."(p. 39)
Part of Rogue AIs (Internal)
Other risks from Hendrycks, Mazzeika & Woodside (2023) (13)
Malicious Use (Intentional)
4.0 Malicious Actors & MisuseMalicious Use (Intentional) > Bioterrorism
4.2 Cyberattacks, weapon development or use, and mass harmMalicious Use (Intentional) > Unleashing AI Agents
4.2 Cyberattacks, weapon development or use, and mass harmMalicious Use (Intentional) > Persuasive AIs
4.1 Disinformation, surveillance, and influence at scaleMalicious Use (Intentional) > Concentration of Power
6.1 Power centralization and unfair distribution of benefitsAI Race (Environmental/Structural)
6.4 Competitive dynamics