Training Data

Definitionp. 73

Data quality is particularly important for both pre-training and post-training of LLMs. Due to the wide range of sources of training data used for LLMs, it is inevitable that there are redundancies, errors, and harmful contents in collected data. Such low-quality data not only affects the performance of LLMs, but also misguides LLMs to generate content that is not expected by humans, such as toxic, pornographic, and biased content. Therefore, data filtering is usually required to ensure the quality of data before training, where rule-based filtering methods and model-based content classifiers are commonly used.

Additional Informationp. 73

GPT-4 (OpenAI, 2023a) identifies pornographic content in training data by combining a dictionary-based approach and a content classifier. Claude 3 (Anthropic, 2024a) integrates multiple data cleaning and filtering methods to improve data quality. Yi (Young et al., 2024) constructs a set of filters based on heuristic rules, keyword matching, and classifiers. Qwen (Bai et al., 2023; Yang et al., 2024a) develops a set of data preprocessing procedures in which humans work with models to perform data filtering by using models to score data content in combination with human review. In addition to this, some companies specify model development policies that require the training data of these models not to include user data in order to protect user privacy, e.g., Meta and Anthropic. While filtered training data reduces harmful content, filtering is still not sufficient for LLMs to learn to perform in a safe way. In order to further improve the safety of LLMs, additional instruction data of a safe nature is usually added to training data. Gemini (Anil et al., 2023) emphasizes the importance of adversarial query data in the post-training stage. It constructs a query dataset for post-training, which contains about 20 categories of harmful data by integrating expert authoring, model synthesis, and automated red teaming methods. Similarly, Yi (Young et al., 2024) constructs a comprehensive safety classification system, and then builds a safety dataset for SFT training based on this classification. It is worth noting that these safety datasets usually require human supervision or guidance to ensure their quality. For example, the safety datasets used in models such as Baichuan (Yang et al., 2023a) and TigerBot (Chen et al., 2023) are constructed under the guidance of relevant domain experts, which highlights the importance of safety datasets compared to general datasets.

Part of