Monitoring

Definitionp. 76

In order to monitor the risks that may occur during the interaction between LLMs and users, researchers develop a variety of monitoring tools for scrutinizing LLMs inputs and outputs and predicting risks

Additional Informationp. 76-77

GPT-4 (OpenAI, 2023a) uses a detection system combining machine learning and rule-based classifiers to identify content that may violate their usage policies. When such content is identified, the deployed monitoring system takes defensive measures such as issuing warnings, temporarily suspending, or in severe cases, banning the corresponding users. Similarly, Claude 3 (Anthropic, 2024a) has a content classifier that identifies any content that violates the Acceptable Use Policy (AUP) (Anthropic, 2024b). User prompts that are flagged as violating AUP trigger a command for Claude to respond more carefully. In the case of particularly serious or harmful user prompts, Claude 3 is prevented from responding at all. In the case of multiple violations, the Claude is terminated. It is important to note that these classifiers need to be updated regularly to address the changing threat landscape. ERNIEBot (Sun et al., 2021) deploys a content review system that intervenes on LLM inputs by means of manual review or rule-based filtering to ensure that LLM inputs conform to a specific standard or specification. On the output side, after filtering out harmful and sensitive words in an LLM-generated response through the content review system, the safety content of the response is used as the final output by means of semantic rewriting. Garak (Derczynski et al., 2023) is a vulnerability scanner for LLMs, which checks models for hundreds of different known weaknesses using thousands of different prompts, and checks model responses to see if the model is at risk in some way. In contrast to the private monitoring systems mentioned above, there are a number of open-source monitoring tools. Perspective API (Jigsaw, 2021) identifies offensive, rude, discriminatory, and other toxic content in online conversations. WildGuard (Han et al., 2024) evaluates the safety of user interactions with LLMs through three safety audit tasks, including Harmfulness of Prompt, Harmfulness of Response, and Rejection of Response. Llama Guard (Inan et al., 2023) is used to detect whether the input prompts and output responses generated by LLMs violate predefined safety categories. Llama 3 (Dubey et al., 2024) uses two prompt-based filtering mechanisms, Prompt Guard and Code Shield. Prompt Guard is used to detect prompting attacks, which are mainly two types of attacks, direct jailbreaks and indirect prompt injections. Code Shield is able to detect the generation of unsafe code before it can potentially enter a downstream use case (e.g., a production system), and is able to support seven programming languages. While these monitoring tools can assist humans in monitoring risky content generated by LLMs, the robustness of the tools themselves remains an issue. Previous research has found evidence of bias in the Perspective API, e.g., giving higher toxicity scores to text containing racial or gender identity terms or phrases associated with African American English (Sap et al., 2019). In the context of a content-moderation tool, these kinds of biases can cause real harm, as they may lead to suppression of speech within or about marginalized communities. Therefore, it is also important to review these widely recognized monitoring tools. IndieLabel (Risk & Alliance, 2024), a detection tool for the Perspective API, prompts users to assign a toxicity score to a small number of text examples (about 20 social media posts). A lightweight model is then trained to predict the user’s perception of a large number of examples (roughly thousands of social media posts), and these predictions are used to uncover areas of potential disagreement between users and the Perspective API. Similarly, BELLS (Dorn et al., 2024a) is a framework for evaluating the reliability and generalizability of LLMs monitoring systems, which allows for a comparison of the reliability of multiple monitoring tools, thus creating a performance competition in anomaly detection.

Part of