Post-deployment practices

Pop-up interventions in LLMs

Supplementary information can be shown to the user in specific query topics where factual accuracy is critical. This intervention can effectively divert users from potential inaccuracies generated by AI models in sensitive contexts. For example, during electoral processes, where model hallucinations can be particularly costly or have a negative impact on society, LLMs can offer their users the option of being redirected to accurate and up-to-date information sources [11]. Since pop-up interventions can be intrusive to workflows, they are best used in situations where the benefits of the information outweigh the cost of distraction

AI identification

AI identifiers can be used to indicate that an AI is involved in a process or an interaction [40]. For AI systems that interact directly with users, a visible output may be used, e.g., a displayed text message saying “I am an AI language model”, accompanied with the appropriate warnings and caveats relevant to the user, [40]. Whereas, for AI systems that interact with other systems or applications, other forms of watermark or unique identifiers can be used. In either situation, agent cards can serve as an identifier, where further details about the underlying AI system, the specific instance of the AI agent, and other information relevant to the development of the agent, can be included.

AI output watermaking

Output produced by or with AI assistance can be marked to clearly identify its origin. Verification of the watermark can involve the use of statistical tests or having the mark immediately visible to a human inspector. Ideally, the watermark does not significantly alter the utility of the output, and is robust against digital and physical manipulation that results in data degradation [216, 145].

AI output metadata

Output produced or whose production is aided by AI can contain metadata to record its origin and the transformations it has undergone. Metadata is evidence that subsequent versions or its derivatives come from this original version. The metadata can include the original AI model source, along with ownership, and its subsequent edits [169].