3.1 Current Alignment Methods

Definitionp. 9, 10

Pre-trained models possess broad capabilities but lack built-in safety protocols, potentially generating harmful outputs or failing to follow instructions. “Post-training” processes steer model behavior to achieve instruction adherence, policy compliance, and other desirable response properties. Common post-training methods include: ● Supervised Fine-Tuning (SFT): Developers curate datasets of desired model behaviors and fine-tune models to match these examples. Datasets include refusal examples (such as declining to provide bomb-making instructions) and helpful yet harmless responses. SFT directly teaches models specific behavioral patterns through imitation learning. ● Reinforcement Learning from Human Feedback (RLHF): Developers use human preferences between different model outputs to questions as a reward signal. They then use reinforcement learning to optimize models for these reward signals. ● Reinforcement Learning from AI-assisted Feedback (RLAIF): Methods like Anthropic's Constitutional AI, OpenAI’s deliberative alignment, and the broader category of AI-assisted feedback, including RLAIF, use AI systems to generate training feedback based on predefined principles or constitutions (such as OpenAI’s model spec), and have gained traction as a scalable alternative to purely human-generated feedback. Modern alignment training pipelines typically combine these methods. These approaches can also steer the model towards prioritizing certain types of requests over others (such as system-level safety specifications over others). Reward signals for agentic AI systems could also consist of more complex and real-world tasks, such as writing safe software code.

Additional Informationp. 10

However, behavioral alignment approaches have several limitations, open questions, and possible challenges: 1\. Robustness Challenges: Current safety training methods modify surface-level behaviors without altering underlying model capabilities. Research shows that harmful fine-tuning can rapidly undo alignment post-training with surprisingly few examples – sometimes just dozens of harmful input-output pairs. Adversarial prompts (“jailbreaks”) can bypass alignment post-training – models trained to refuse direct harmful requests may still comply when those requests are adversarially rephrased, decomposed into steps, or embedded in different contexts. Developers can continuously patch new vulnerabilities; however, in attempting to correct for this, a different problem can occur with “over-refusal,” where models become excessively cautious and decline legitimate requests like medical questions. 2\. Training Signal Learning and/or Design Challenges: Behavioral alignment requires translating human values into training objectives, but this translation introduces challenges. Reward misspecification and/or “reward hacking” could occur, with models exploiting flaws in reward signals, such as generating unnecessarily verbose responses that score well but provide little value, or appealing to evaluator biases rather than producing useful outputs. Models might also learn different objectives than intended, even when performing well on a well-designed training signal, and could potentially then “fake alignment” to avoid further modification. These different objectives could potentially emerge from the model learning proxy goals that happen to achieve good training performance but generalize in undesired ways (also known as “goal misgeneralization”). 3\. Understanding and Measurement Gaps: Although there is some early work in the area, the field currently lacks reliable methods to assess alignment effectiveness, making it difficult to determine how well current techniques will scale to more advanced systems. These limitations highlight the need for continued research into the significance of these challenges and methods that can scale reliably with advancing model capabilities.

Part of