Inequality, Marginalization, and Violence

Evaluating the Social Impact of Generative AI Systems in Systems and Society

Solaiman et al. (2023)

Source DOI

Sub-category

Risk Domain

1Discrimination & Toxicity

1.1Unfair discrimination and misrepresentation

Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.

"Generative AI systems are capable of exacerbating inequality, as seen in sections on 4.1.1 Bias, Stereotypes, and Representational Harms and 4.1.2 Cultural Values and Sensitive Content, and Disparate Performance. When deployed or updated, systems' impacts on people and groups can directly and indirectly be used to harm and exploit vulnerable and marginalized groups."(p. 13)

Entity— Who or what caused the harm

Human

Due to a decision or action made by humans

AI system

Due to a decision or action made by an AI system

Other

Due to some other reason or is ambiguous

Intent— Whether the harm was intentional or accidental

Intentional

Due to an expected outcome from pursuing a goal

Unintentional

Due to an unexpected outcome from pursuing a goal

Other

Without clearly specifying the intentionality

Timing— Whether the risk is pre- or post-deployment

Pre-deployment

Occurring before the AI is deployed

Post-deployment

Occurring after the AI model has been trained and deployed

Other

Without a clearly specified time of occurrence

Supporting Evidence (3)

Community Erasure: "Biases in a system’s development process and safety provisions for generative AI systems, such as content moderation, can lead to community erasure [97]. Avoiding the generation of the harms outlined is seen as a generally desirable outcome. However, the removal of harmful content can come with its own costs of lower general performances for sub-populations that use models for generation [269]. Mitigation thus currently serves as a double-edged sword, where removal of toxic content also has negative implications, in particular for marginalized communities. Both the benefits and the costs of content moderation are unequally distributed. The automatic systems that remove undesirable content can perform next to randomly or be harmful for marginalized populations [208], while the selection criteria for what constitutes safe content are aligned with technical safety and mitigation decisions. These impacts compound to make marginalized populations pay a greater cost for an intervention that they benefit from less."(p. 13)

Long-term Amplifying Marginalization by Exclusion (and Inclusion): "Biases, dominant cultural values, and disparate performance seen in lack of representation in training and development of generative AI systems can exacerbate marginalization when those systems are deployed. For example, increasing resourcing and performance for already highly resourced languages reinforces those languages’ dominance. Inclusion without consent can also harm marginalized groups. While some research strives to improve performance for underrepresented Indigenous languages [116], the same Indigenous groups resist AI approaches to use of their language [158]. Profit from Indigenous languages and groups who have been systematically exploited continues directly and indirectly."(p. 13)

Abusive or Violence Content: "Generative AI systems can generate outputs that are used for abuse, constitute non-consensual content, or are threats of violence and harassment [9]. Non-consensual sexual representations of people, include representations of minors as generative child sexual abuse material (CSAM) [155]. Abuse and violence can disparately affect groups, such as women and girls [10]"(p. 14)