Training Methodology

Definitionp. 73

Reinforcement learning from human feedback (RLHF) has proven a useful avenue for enhancing the safety of LLMs at the post-training stage (Ouyang et al., 2022; OpenAI, 2023a; Touvron et al., 2023; Bai et al., 2022a). RLHF performs human feedback-based fine-tuning, which uses human preferences as a proxy to specify human values (Shen et al., 2023). Normally, RLHF consists of three core steps, which are (1) collecting human feedback data, (2) using the collected human feedback data to train reward models, and (3) fine-tuning LLMs using reinforcement learning algorithms such as Proximal Policy Optimization (PPO) (Schulman et al., 2017).

Additional Informationp. 73-74

Anthropic further proposes Constitutional AI (Bai et al., 2022b), where a set of moral and behavioral principles, termed as a constitution, is developed for aligning LLMs via supervised learning and reinforcement learning. In Claude 3 (Anthropic, 2024a), the rules of the constitution are derived from the Universal Declaration of Human Rights, Apple’s Terms of Service, Principles Encourage Consideration of Non-Western Perspectives, DeepMind’s Sparrow Rules, and Anthropic Customization Principles (Anthropic, 2023a). Alignment approaches similar to Constitutional AI are also used in Gemini (Anil et al., 2023) and Qwen2 (Yang et al., 2024a). With Constitutional AI, a research team from Google delves into the comparison between reinforcement learning from AI feedback (RLAIF) (Lee et al., 2024b) and RLHF, and demonstrates that RLAIF could be a competitive alternative to RLHF, thus reducing the reliance on expensive human annotation. In addition to the annotation cost, traditional RLHF methods usually involve optimizing the reward function for human preferences, which is effective but may bring about challenges such as increasing computational complexity and the need to consider the bias-variance trade-off when estimating and optimizing rewards (Schulman et al., 2016). To mitigate these issues, DPO (Rafailov et al., 2023) has been proposed to simplify the alignment process, reducing computational overhead and enabling more robust optimization by using preference data in a more direct way. In addition to alignment training, interpretability methods can also be used to achieve safety controls over LLMs. The Center for AI Safety (CAIS) introduces the concept of representation engineering (Zou et al., 2023a) to improve the transparency of AI systems by leveraging cognitive neuroscience. Representation engineering extracts various safety-related concepts through representation learning methods, and then modifies or controls these safety-related conceptual representations through representation control methods to reduce risks associated with LLMs. They conduct extensive case studies using representation engineering on safety aspects such as honesty and ethics, demonstrating the potential of representation engineering to improve transparency, safety, and trust in AI systems.

Part of