2.2 AI system security vulnerabilities and attacks

AI systems, like other software systems, face a range of security threats. These issues may arise from inherent weaknesses in the design of AI algorithms, the data used to train the models, or the operational context. Specific examples include:

Toolchain and dependency vulnerabilities that arise unintentionally through the use of automated code-generation tools (e.g., Github Copilot, Python language, OpenCV), deep learning frameworks (e.g., Tensorflow, PyTorch), or as a result of complex interdependencies in the development environment.

External tool and API integration into AI system applications can compromise the trustworthiness and privacy of systems due to their potential unreliability or susceptibility to adversarial control.

Security vulnerabilities in physical and network infrastructure, such as vulnerabilities in graphics processing units, or GPUs, or to sophisticated attacks like side-channel and rowhammer attacks, can lead to unauthorized access or manipulation of model parameters when used during training of AI systems. The use of distributed network systems for training AI systems such as LLMs exposes them to network-specific threats like pulsating attacks or congestion.

Direct manipulation of AI systems such as adversarial attacks and instruction-based attacks. Adversarial attacks focus on altering the model's learning process or extracting its data. They include perturbations designed to deceive models into incorrect outputs, extraction attacks to steal model insights, and poisoning attacks to alter model behavior. Instruction-based attacks manipulate the way the model handles and responds to inputs. Attackers deliberately craft prompts to induce models to produce biased or unsafe outputs (a.k.a. 'jailbreaking'). This manipulation directly targets the operational aspects of AI systems with the intent to cause harm.

Excerpt from the MIT AI Risk Repository full report