AI model-assisted oversight of AI systems

Definitionp. 40

AI model-assisted oversight can help monitor and supervise the training of increasingly capable GPAI systems, which may become difficult to oversee at scale by human supervisors during training or testing. Monitoring and supervision may become especially difficult in cases where increasingly advanced GPAIs perform near or above human level in some specialized domains, where supervision quality might fail to keep pace with capabilities improvement. The training signal may include labeled data, reward function, and user feedback on produced outputs.

Additional Informationp. 40-41

Currently, there are two broad approaches to provide scalable training signals to such systems: 1. Scalable oversight: Improving of the supervisor’s capabilities to supervise, such that they can provide accurate training signals quickly and at scale [31]. For example, a debate format can be used between two GPAI systems (two instances of the same GPAI, or similarly capable systems). A Human supervisor judges the debate, making it easier to assess correct responses in domains which might otherwise require significant time investment of domain specific expertise [102]. 2. Weak-to-strong generalization: Enhance the training signals while ensuring that the enhanced signal remains faithful to the intentions of the original human-provided signal [37]. For example, a hierarchical (“bootstrapping”) oversight approach can be implemented: A series of GPAI models with increasing capabilities are used, where each model in the hierarchy provides oversight for the next more capable model. The least capable model at the base of the hierarchy is the only one directly overseen by human supervisors, as it is easier to oversee than the more capable models.

Part of