This page is still being polished. If you have thoughts, please share them via the feedback form.
Data on this page is preliminary and may change. Please do not share or cite these figures publicly.
Technical mechanisms and engineering interventions that directly modify how an AI system processes inputs, generates outputs, or operates, including changes to models, training procedures, runtime behaviors, and supporting hardware.
Reasoning
Mitigation name lacks concrete mechanism; placeholder text prevents focal activity identification.
Value Misalignment
Methods for mitigating LLM amorality: methods without training
Many researchers employ in-context learning methods either to induce LLMs to generate unethical content and subsequently correct them (Duan et al., 2024b) or to guide LLMs in recognizing unethical instructions within the prompts (Roy et al., 2023; Phute et al., 2024; Ganguli et al., 2023), thereby preventing the generation of unethical outputs.
1.2.1 Guardrails & FilteringMethods for mitigating LLM amorality: methods within training
Numerous studies utilize SFT to align LLMs with human values, though this approach is not strictly limited to ethical and moral considerations (Sun et al., 2023; Liu et al., 2024e). During this phase, constructing or filtering SFT datasets related to moral principles to train LLMs is widly explored (Solaiman & Dennison, 2021; Hendrycks et al., 2021; Zhao et al., 2024b). During the reinforcement learning phase of post-training, Ouyang et al. (2022) employ manually annotated data to align LLMs with human values. However, this approach requires substantial computational resources. In response, Bai et al. (2022a) have explored an iterative online training framework, which involves updating the preference model and RL policies weekly, based on newly acquired human feedback. Simultaneously, to reduce the cost associated with manually annotated data, synthetic data have proven effective as well (Kim et al., 2023; Bai et al., 2022b). Although few RLHF-related studies specifically focus on ethical and moral aspects, their application can undoubtedly enhance the safety of LLMs in echtics and morality.
1.1.2 Learning ObjectivesValue Misalignment
99.9 OtherValue Misalignment > Mitigating social bias
1 AI SystemValue Misalignment > Privacy protection
1 AI SystemValue Misalignment > Methods for mitigating toxicity
1 AI SystemRobustness to attack
1 AI SystemRobustness to attack > Red teaming
Red teaming is widely used in LLMs to explore their safety vulnerabilities prior to the deployment of them. Red teaming can be broadly categorized into two distinct types: manual red teaming and automated red teaming
2.2.2 Testing & EvaluationLarge Language Model Safety: A Holistic Survey
Shi, Dan; Shen, Tianhao; Huang, Yufei; Li, Zhigen; Leng, Yongqi; Jin, Renren; Liu, Chuang; Wu, Xinwei; Guo, Zishan; Yu, Linhao; Shi, Ling; Jiang, Bojian; Xiong, Deyi (2024)
The rapid development and deployment of large language models (LLMs) have introduced a new frontier in artificial intelligence, marked by unprecedented capabilities in natural language understanding and generation. However, the increasing integration of these models into critical applications raises substantial safety concerns, necessitating a thorough examination of their potential risks and associated mitigation strategies. This survey provides a comprehensive overview of the current landscape of LLM safety, covering four major categories: value misalignment, robustness to adversarial attacks, misuse, and autonomous AI risks. In addition to the comprehensive review of the mitigation methodologies and evaluation resources on these four aspects, we further explore four topics related to LLM safety: the safety implications of LLM agents, the role of interpretability in enhancing LLM safety, the technology roadmaps proposed and abided by a list of AI companies and institutes for LLM safety, and AI governance aimed at LLM safety with discussions on international cooperation, policy proposals, and prospective regulatory directions. Our findings underscore the necessity for a proactive, multifaceted approach to LLM safety, emphasizing the integration of technical solutions, ethical considerations, and robust governance frameworks. This survey is intended to serve as a foundational resource for academy researchers, industry practitioners, and policymakers, offering insights into the challenges and opportunities associated with the safe integration of LLMs into society. Ultimately, it seeks to contribute to the safe and beneficial development of LLMs, aligning with the overarching goal of harnessing AI for societal advancement and well-being. A curated list of related papers has been publicly available at https://github.com/tjunlp-lab/Awesome-LLM-Safety-Papers.
Build and Use Model
Training, fine-tuning, and integrating the AI model
Developer
Entity that creates, trains, or modifies the AI system
Manage
Prioritising, responding to, and mitigating AI risks
Other
