Exclusionary norms

Ethical and social risks of harm from language models

Weidinger et al. (2021)

Source DOI

Sub-category

Risk Domain

1Discrimination & Toxicity

1.1Unfair discrimination and misrepresentation

Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.

"In language, humans express social categories and norms. Language models (LMs) that faithfully encode patterns present in natural language necessarily encode such norms and categories...such norms and categories exclude groups who live outside them (Foucault and Sheridan, 2012). For example, defining the term “family” as married parents of male and female gender with a blood-related child, denies the existence of families to whom these criteria do not apply"(p. 13)

Entity— Who or what caused the harm

Human

Due to a decision or action made by humans

AI system

Due to a decision or action made by an AI system

Other

Due to some other reason or is ambiguous

Intent— Whether the harm was intentional or accidental

Intentional

Due to an expected outcome from pursuing a goal

Unintentional

Due to an unexpected outcome from pursuing a goal

Other

Without clearly specifying the intentionality

Timing— Whether the risk is pre- or post-deployment

Pre-deployment

Occurring before the AI is deployed

Post-deployment

Occurring after the AI model has been trained and deployed

Other

Without a clearly specified time of occurrence

Supporting Evidence (3)

"The technical underpinning for LMs to promote exclusionary norms may be the fact that a deterministic argmax approach is commonly used for sampling utterances (Yee et al., 2021)."(p. 13)

"A LM trained on language data at a particular moment in time risks not just excluding some groups, but also enshrining temporary values and norms without the capacity to update the technology as society develops....The risk, in this case, is that LMs come to represent language from a particular community and point in time, so that the norms, values, categories from that moment get “locked in” (Bender et al., 2021; Gabriel and Ghazavi, 2021)."(p. 14)

"Homogenising effects in downstream applications Concerns on exclusionary norms are relevant across a wide range of contexts. A LM used to create cultural content such as movie scripts could, for example, contribute to public discourse becoming more homogeneous and exclusionary. Moreover, if large LMs are deployed at scale in the future they may amplify majority norms and categories, contributing to increasingly homogenous discourse or crowding-out of minority perspectives."(p. 14)

Part of Discrimination, Exclusion and Toxicity