AI ecosystem monitoring

Tracing usage patterns

One key, high-level lens into how AI systems impact the world is through usage monitoring (e.g. Anthropic-A). By collecting and monitoring data on how users access, download, and/or interact with frontier systems, AI service providers can gather insights about potential impacts and risks. However, key challenges with tracking usage include privacy preservation, infrastructure for sharing insights, and effective tools for identifying potential risks.

Data provenance

Various techniques can help to identify AI-generated content and are a principal defence against AI deepfakes and misinformation. Methods include developing reliable classifiers of AI-generated content, watermarking AI-generated data (images, video, audio, and text) (Cao), and tagging AI-generated data with metadata to indicate its origin. These techniques are inherently imperfect – they can be undone by tampering with data. However, in forensic science, similar techniques like fingerprinting are also circumventable but useful nonetheless. Further progress on these methods will involve both more reliable methods for data provenance and their integration into AI products and services.

Model provenance

Tools for model provenance help to identify and track AI models – especially open-weight ones. Most notably, these tools help researchers study the origins and lifecycle of harmful models in the ecosystem. Methods for model provenance involve techniques to help users and AI providers ascertain the identity and origin of a model. This can include black-box methods, such as identification backdoors (Cheng), identifiable biases in text generation (Kirchenbauer), and white-box methods, such as model weight watermarks. Much like data provenance methods, model provenance methods can be circumvented, but they can be informative in many cases nonetheless. Research frontiers include studying how stable these techniques are under fine-tuning and other modifications to a model’s weights. Engineering efforts may also be needed to integrate these techniques into model development and platform infrastructure

Agent authentication

Some protocols can allow for the verification of AI agent identities while they are using web services. As AI systems become increasingly capable and agentic, methods to authenticate AI agents when they use web services are increasingly important from a security and monitoring standpoint. Key challenges toward effective agent authentication lie in the development and standardisation of protocols (e.g. South).

Compute and hardware tracking

Researching techniques and gathering intelligence to monitor the distribution of AI hardware, both legal and illegal, enables the assessment of risks of malicious and irresponsible use and the allocation of resources to promote beneficial use (Sastry).

Logging infrastructure

Monitoring and saving information about what AI systems are doing allows for informed scrutiny when harmful or unexpected events happen. As highly autonomous AI systems grow in their capability and influence, there will be a rise in harmful and unintended incidents from these systems’ actions. Having effective infrastructure to capture and save information about what these systems do will be key for improving awareness and accountability in the age of advanced AI agents (Chan). Logged incidents and the necessary infrastructure constitute another example of a potential area of mutual interest. Just as competing aircraft manufacturers voluntarily share data about aircraft accidents, companies or countries may find it in their interest to share and jointly collect information about serious AI incidents. Establishing shared incident reporting systems allows the field to collectively learn from serious failures and risks, ensuring safety and security to foster public trust in AI’s opportunities.

Assessing risk-management frameworks

Technical tools for risk management are only effective inasmuch as they are meaningfully integrated into safety frameworks. Just as it is key to evaluate and monitor AI systems, it is also necessary to evaluate and monitor risk management protocols for their effectiveness and robustness to single points of failure (e.g. human error). Currently, researchers’ ability to assess risk management frameworks is limited by the degree of transparency into how AI developers manage risks. However, monitoring the successes and failures of safety frameworks is key for risk management over time (Alaga).It is also an area of mutual interest due to the value of sharing insights on best practices and potential failures of risk management frameworks. Frontiers for future work include refining assessment frameworks and developing reporting infrastructure.