Value misalignment and robustness evaluation

Definitionp. 74-75

For GPT-4 (OpenAI, 2023a), OpenAI conducts internal quantitative evaluations following its content policies, e.g., evaluations on hate speech, self-injurious suggestions, and illegal suggestions. These evaluations measure the likelihood that GPT-4 generates content violating value alignment when given a prompt. Similarly, to ensure that Claude 3 (Anthropic, 2024a) is as safe as possible prior to deployment, Anthropic’s Trust and Safety team conducts a full multi-modal red teaming exercise to thoroughly evaluate Claude 3, including evaluations over trust and safety, bias, discrimination, and more. CAIS explores the relationship between AI safety and general upstream capabilities (e.g., general knowledge and reasoning), and finds that many safety benchmarks are highly correlated with upstream model capabilities, which can lead to “safety washing” (Ren et al., 2024b). In such cases, capability improvements are mischaracterized as safety progress. Based on these findings, they propose defining AI safety as a set of explicitly described research goals that are empirically separable from general capability progress, thus ensuring more accurate safety evaluation. The Allen Institute for AI develops WildTeaming (Jiang et al., 2024c), an automated red teaming framework for identifying and reproducing human attacks.WildTeaming directly exploits jailbreak strategies of human users and leverages those strategies to address vulnerabilities in LLMs.

Additional Informationp. 75

Evaluations over value misalignment and robustness emphasize the importance of safety benchmark development. In this respect, a number of companies are committed to the development of multi-dimension, multi-domain safety evaluation benchmarks. Alibaba Cloud launches a “100 Bottles of Poison” for Chinese LLMs, developing a value alignment evaluation benchmark (Cvalues) (Xu et al., 2023b). In this benchmark, adversarial safety prompts in multiple categories are created under the guidance of experts from different domains, aiming to evaluate LLMs in terms of both safety and responsibility. Cohere releases the first multilingual human-labeled red team prompt datasets to distinguish between global and local harms (Aakanksha et al., 2024). These datasets are used to evaluate the robustness of different alignment techniques in the face of preference distributions across geographies and languages.

Part of