RLHF

RL/PbRL/IRL/Imitation Learning

RLxF

Building on the RLHF paradigm, we introduce RLxF as a fundamental framework for scalable oversight, aiming to enhance feedback efficiency and quality and expand human feedback for more complex tasks. This enhances RLHF by incorporating AI components (Fernandes et al., 2023). The x in RLxF signifies a blend of AI and humans.

Iterated Distillation and Amplification IDA

Iterated Distillation and Amplification (IDA) introduces a framework for constructing scalable oversight through iterative collaboration between humans and AIs (Christiano et al., 2018). The process commences with an initial agent, denoted as A[0], which mirrors the decision-making of a human, H. A[0] undergoes training using a potent technique that equips it with near-human-level proficiency (the distillation step); Then, collaborative interaction between H and multiple A[0] instances leads to the creation of an enhanced agent, A[1] (the amplification step).

Recursive Reward Modeling RRM

Recursive Reward Modeling (RRM) (Leike et al., 2018) seeks to broaden the application of reward modeling to much more intricate tasks. The central insight of RRM is the recursive use of already trained agents At−1 to provide feedback by performing reward learning on an amplified version of itself for the training of successive agents At on more complex tasks. The A0 is trained via fundamental reward modeling (learned from pure human feedback). This approach is not only influenced by human feedback but also by the model’s own assessments of what constitutes a rewarding outcome.

Debate

Debate involves two agents presenting answers and statements to assist human judges in their decision-making (Irving et al., 2018), as delineated in Algorithm 3. This is a zero-sum debate game where agents try to identify each other’s shortcomings while striving to gain higher trust from human judges, and it can be a potential approach to constructing scalable oversight.

Cooperative Inverse Reinforcement CIRL

The framework of Cooperative Inverse Reinforcement Learning (CIRL), however, unifies control and learning from feedback and models human feedback providers as fellow agents in the same environment. It approaches the scalable oversight problem not by strengthening oversight but by trying to eliminate the incentives for AI systems to game oversight, putting humans giving feedback and the AI system in cooperative rather than adversarial positions (Shah et al., 2020). In the CIRL paradigm, the AI system collaborates with humans to achieve the human’s true goal rather than unilaterally optimizing for human preferences.