External safeguard

Definitionp. 40

External safeguard focuses on protecting an LLM from malicious inputs or external threats by implementing safety measures outside the model. A number of studies propose external safeguard methods to detect unsafe content in the input and output of the given LLM.

Additional Informationp. 40-41

Kumar et al. (2023) wipe out tokens one by one for a given prompt, and then if any resulting subsequence or the prompt itself is detected as harmful, the input will be marked as harmful. Xie et al. (2024) propose GradSafe, which effectively detects jailbreak prompts by carefully checking the gradient of key parameters in LLM. Cao et al. (2023) build a robustly aligned LLM (RALLM) to defend against potential alignment-breaking attacks. Zeng et al. (2024d) assign different roles to LLM agents and use their collaborative monitoring and filtering of responses generated by LLMs to complete defense tasks. Other methods present various techniques for attacks including undesired content, such as classification system (Markov et al., 2023), token-level detection algorithms (Hu et al., 2023), toolkit with programmable rails (Rebedea et al., 2023), and LLM-based input-output safeguard model (Inan et al., 2023). Another strand of research starts from providing an instructive prompt or modifying a given prompt. Xie et al. (2023) provide instructions to guide LLMs to self-check and respond responsibly. Wei et al. (2023b) provide several examples of safe response to encourage safer outputs from LLMs. Xiong et al. (2024) use well-designed interpretable suffix prompts, which can effectively defend various standard and adaptive jailbreak techniques. Yi et al. (2023) add a reminder to the prompt fed to an LLM, which instructs the LLM not to execute commands embedded in external content, hence avoiding the execution of malicious instructions hidden in external content. Given the response generated by a target LLM from its original input prompt, Wang et al. (2024b) asks an LLM to infer the input prompt that may cause the response through using a “backtranslation” prompt. The inferred prompt is called a backtranslation prompt, which tends to reveal the actual intention of the original prompt. Robey et al. (2023) randomly interfere a given input prompt, generate multiple copies, and then aggregate/summarize corresponding responses to detect the adversarial input. External threats may employ a variety of attack strategies, making them difficult to predict and guard against. It is challenging to identify highly covert attacks or malicious input and output. And too much reliance on external safeguard may lead to the safety of the whole system is weak. If the external system crashes, the overall safety of the protected LLM would be greatly reduced.

Part of