This page is still being polished. If you have thoughts, please share them via the feedback form.
Data on this page is preliminary and may change. Please do not share or cite these figures publicly.
Implementation standards, guidelines, and documented best practices for AI development.
Also in Shared Infrastructure
Describe the frontier AI development lifecycle and identify risk management activities that the organization must perform at each phase. This helps integrate safety and security into all stages of development, deployment, and monitoring. In cybersecurity, it has helped advance a “shift left” approach, i.e., designing safety into systems during development and tackling issues early in the software development lifecycle. While some AI development lifecycle frameworks exist, they need additional work to adapt to a frontier AI context and map appropriate risk management activities at each stage. We recommend that the FMF develop a consensus model that captures these key activities for developers, and that AI developers, philanthropists, and government funders pursue research supporting a “shift left” for frontier AI safety and security.
Purpose: Provides a holistic view of safety/security activities in software development, deployment, and operations. Value proposition: Promotes “shift left” and “security by design”; calls attention to important deployment decisions and need for continuous monitoring. Limitations: Can omit cross-cutting categories of activities; does not focus on other actors’ activities. Existing infrastructure that users can adopt for frontier AI risk management: Limited. Various models exist, but no consensus and limited detail on specific activities. Most suitable parties to conduct further research: NIST, Frontier Model Forum
Reasoning
Foundational research describing frontier AI development lifecycle stages and processes.
Establish a detailed lifecycle framework for frontier AI that describes safety and security activities at each stage.
Establish a detailed lifecycle framework for frontier AI that describes safety and security activities at each stage. (4.3.2 | Proposed lifecycle framework) This framework can build on work by the OECD while incorporating details from frontier AI developers, and should map activities to the NIST AI RMF where possible. It should ensure all phases are appropriately covered, which could include a “shift left” (see recommendation 4), and a stage for post-deployment monitoring and response.
3.2.2 Technical StandardsPursue research that supports a “shift left”
Pursue research that supports a “shift left” for frontier AI by emphasizing safety and security activities earlier in the development cycle. (4.3.3.1 | “Shifting left” on AI safety and security; 6.2.2 | Lifecycle) Potential research areas could include: software requirement specification techniques borrowed from safety-critical domains, dataset curation techniques, and foundational research to build safer and more secure AI systems.
2.4.1 Research & FoundationsFunctional: Identify essential categories of safety and security activities (“functions”)
Identify essential categories of safety and security activities (“functions”) that an organization must perform, and map these to a specified set of outcomes. This helps organizations to organize their risk management activities at a high level, and to assess if these activities are achieving the necessary outcomes. A functional approach is particularly helpful for identifying cross-cutting categories (e.g., organizational governance or insider security) that provide resilience against multiple known and unknown risks. It is also the most ready-to-adopt, based on the National Institute of Standards and Technology (NIST) AI Risk Management Framework (RMF) and supplementary guidance from other researchers that begins to adapt this framework to cover catastrophic risks from frontier AI. We recommend that NIST or the Frontier Model Forum (FMF) establish consensus on the highest-priority categories of activities for frontier AI developers and develop a detailed catalog of measures (“controls”) for these activities.
3.2.2 Technical StandardsFunctional: Identify essential categories of safety and security activities (“functions”) > Establish consensus on which categories of activities in the NIST AI RMF are the highest priority for frontier AI developers.
Establish consensus on which categories of activities in the NIST AI RMF are the highest priority for frontier AI developers. (3.3.1 | The NIST AI RMF) NIST and/or the FMF, with researcher input, should identify high-priority categories for frontier AI safety and security. To ensure defense-in-depth, frontier AI developers should implement multiple independent measures for these categories.
3.3.1 Industry CoordinationFunctional: Identify essential categories of safety and security activities (“functions”) > Develop a detailed catalog of measures (“controls”) that are important for frontier AI safety and security.
Develop a detailed catalog of measures (“controls”) that are important for frontier AI safety and security. (3.3.3 | Providing detailed controls) For instance, NIST SP 800-53 lists 1,000 detailed controls for cybersecurity across 20 “families.” No current equivalent exists for AI, and it would be useful for frontier AI developers to have a similar catalog focused on frontier AI safety and security.
3.2.2 Technical StandardsThreat-based: Compile and describe the tactics, techniques, and procedures (TTPs) that threat actors use
Compile and describe the tactics, techniques, and procedures (TTPs) that threat actors use, based on real-world evidence and what research suggests is possible. This approach typically focuses on TTPs to attack AI models (“effect on model”), but we suggest expanding it to cover TTPs using AI models (“effect on world”), given concerns about malicious use of AI. While MITRE ATLAS provides a solid starting point for “effect on model” TTPs, more work would be needed to build out both “effect on model” and “effect on world” approaches into comprehensive databases of TTPs and mitigations. We recommend that MITRE, or the FMF, expand MITRE ATLAS into such a resource for frontier AI experts. We also recommend the US Cybersecurity and Infrastructure Agency (CISA) assess the effects of frontier AI systems on the top ten most vulnerable National Critical Functions. Database owners should strongly consider limiting public access, due to the risk of facilitating attacks by malicious actors.
3.2.3 Research ResourcesThreat-based: Compile and describe the tactics, techniques, and procedures (TTPs) that threat actors use > Restructure and expand MITRE ATLAS to further address attacks on frontier AI.
Restructure and expand MITRE ATLAS to further address attacks on frontier AI. (5.3.2.1 | An “effect on model” approach) MITRE ATLAS is a knowledge base of tactics, techniques, and procedures (TTPs) that malicious actors can use to attack AI systems. The high-level categories (“tactics”) are closely adapted from the equivalent cybersecurity knowledge base. We suggest restructuring these high-level tactics to reflect an AI-specific taxonomy (e.g., to include tactics like compromising training pipelines), and expanding on techniques and procedures that could enable misuse such as bypassing model guardrails.
3.2.2 Technical StandardsThreat-based: Compile and describe the tactics, techniques, and procedures (TTPs) that threat actors use > Develop a common taxonomy of TTPs describing malicious use of frontier models to impact other actors and systems.
Develop a common taxonomy of TTPs describing malicious use of frontier models to impact other actors and systems. (5.3.2.2 | An “effect on world” approach) The knowledge base should combine real-world evidence and what research suggests is possible. Database owners should strongly consider limiting public access, due to the risk of facilitating attacks by malicious actors.
3.2.2 Technical StandardsAdapting cybersecurity frameworks to manage frontier AI risks: A defense-in-depth approach
Ee, Shaun; O'Brien, Joe; Williams, Zoe; El-Dakhakhni, Amanda; Aird, Michael; Lintz, Alex (2024)
The complex and evolving threat landscape of frontier AI development requires a multi-layered approach to risk management ("defense-in-depth"). By reviewing cybersecurity and AI frameworks, we outline three approaches that can help identify gaps in the management of AI-related risks. First, a functional approach identifies essential categories of activities ("functions") that a risk management approach should cover, as in the NIST Cybersecurity Framework (CSF) and AI Risk Management Framework (AI RMF). Second, a lifecycle approach instead assigns safety and security activities across the model development lifecycle, as in DevSecOps and the OECD AI lifecycle framework. Third, a threat-based approach identifies tactics, techniques, and procedures (TTPs) used by malicious actors, as in the MITRE ATT&CK and MITRE ATLAS databases. We recommend that frontier AI developers and policymakers begin by adopting the functional approach, given the existence of the NIST AI RMF and other supplementary guides, but also establish a detailed frontier AI lifecycle model and threat-based TTP databases for future use.
Other (stage not listed)
Applies to a lifecycle stage not captured by the standard categories
Governance Actor
Regulator, standards body, or oversight entity shaping AI policy
Govern
Policies, processes, and accountability structures for AI risk management
