Internal protection

Definitionp. 41

Internal protection involves modifying an LLM during the process of training or tuning to improve its ability to defend against various attacks, reduce risks, and ensure safe behaviors in real-world applications.

Additional Informationp. 41

Fu et al. (2024) propose single- and mixed-task losses for instruction tuning and demonstrate that LLMs can significantly boost safe management of risky content via appropriate instruction tuning, thus defending for attacks involving malicious long documents. Han et al. (2023) inject three types of defense functions into the different stages of Federated LLMs in federal learning aggregation to support the defense mechanism for adversarial attacks. Hasan et al. (2024) demonstrate that employing moderate WANDA pruning can bolster LLM’s defense against jailbreak attacks while obviating the need for fine-tuning. The WANDA pruning (Sun et al., 2024c) involves removing a subset of network weights, with the goal of preserving performance. In addition, a considerable number of studies choose fine-tuning (Touvron et al., 2023) or instruction-tuning (Deng et al., 2023b) to strengthen LLM’s defense against prompt attacks. Liu et al. (2024c) introduce a two-stage adversarial tuning framework, which enhances the ability of LLMs to resist unknown jailbreak attacks through iterative improvement of confrontational prompts. Touvron et al. (2023) collect adversarial prompts and their safety demonstrations, subsequently integrating these samples into the general supervised fine-tuning pipeline. Correspondingly, Yi et al. (2023) apply adversarial training to the self-supervised fine-tuning stage of LLMs, teaching them to ignore instructions embedded in external content, thus enhancing their robustness to indirect prompt injection attacks. Despite effectiveness, internal protection could increase the complexity of LLMs, and reduce their interpretability and maintainability. It is worth mentioning that current defense strategies focus too much on defending attacks and ignore the reduction of effectiveness. Varshney et al. (2024) propose that the ideal defense strategy should make LLMs safe against “unsafe prompts” rather than over-defense on “safe prompt”. They also propose an evaluation benchmark termed Safety and OverDefensiveness Eval (SODE), and their experiment results lead to important findings. For example, self-checking does improve the safety of inputs, but at the cost of extremely excessive defense.

Part of