Rigidity and Mistaken Commitments

"On the other hand, such commitments may only be credible to the extent that a human cannot intervene, increasing the incentive for delegation to AI agents. This could be worsened if other, potentially incompatible commitments can be made by other actors, leading to a ‘commitment race’ (Kokotajlo, 2019) or potential conflict. In complex networks (see Section 3.2), commitments triggered by a small number of agents could – without careful planning – cascade through the network and have a far more damaging effect (Xia & Conitzer, 2010)."(p. 35)

"During the Cold War, the Soviet Union developed the the automated Perimeter system – of- ten called ‘Dead Hand’ – to guarantee a nuclear launch if its leadership were incapacitated, thus ensuring a credible commitment of retaliation (Hoffman, 2009). While this mechanism was intended as a deterrent, its automatic and largely irrevocable nature exemplifies how credible commitments can become dangerously dual-use: once triggered, there would be little chance to override or de-escalate. In a similar vein, during Operation Iraqi Freedom in 2003 an automated US missile defence system shot down a British plane, killing both occupants (Borg et al., 2024; Talbot, 2005). While the system’s operators had one minute to override the system (even in its autonomous mode), they decided to trust its judgment, resulting in a tragic outcome. In more general AI contexts, similarly inflexible commitments could offer short-term advantages or trust but risk uncontrolled escalation, lock-in, and catastrophic outcomes if not carefully designed with appropriate fail-safes and oversight."(p. 35)