Misinformation risks

Entrenched viewpoints and reduced political efficacy

"Design choices such as greater personalisation of AI assistants and efforts to align them with human preferences could also reinforce people’s pre-existing biases and entrench specific ideologies. Increasingly agentic AI assistants trained using techniques such as reinforcement learning from human feedback (RLHF) and with the ability to access and analyse users’ behavioural data, for example, may learn to tailor their responses to users’ preferences and feedback. In doing so, these systems could end up producing partial or ideologically biased statements in an attempt to conform to user expectations, desires or preferences for a particular worldview (Carroll et al., 2022). Over time, this could lead AI assistants to inadvertently reinforce people’s tendency to interpret information in a way that supports their own prior beliefs (‘confirmation bias’), thus making them more entrenched in their own views and more resistant to factual corrections (Lewandowsky et al., 2012). At the societal level, this could also exacerbate the problem of epistemic fragmentation – a breakdown of shared knowledge, where individuals have conflicting understandings of reality and do not share or engage with each other’s beliefs – and further entrench specific ideologies. Excessive trust and overreliance on hyperpersonalised AI assistants could become especially problematic if people ended up deferring entirely to these systems to perform tasks in domains they do not have expertise in or to take consequential decisions on their behalf (see Chapter 12). For example, people may entrust an advanced AI assistant that is familiar with their political views and personal preferences to help them find trusted election information, guide them through their political choices or even vote on their behalf, even if doing so might go against their own or society’s best interests. In the more extreme cases, these developments may hamper the normal functioning of democracies, by decreasing people’s civic competency and reducing their willingness and ability to engage in productive political debate and to participate in public life (Sullivan and Transue, 1999)."

Degraded and homogenised information environments

"Beyond this, the widespread adoption of advanced AI assistants for content generation could have a number of negative consequences for our shared information ecosystem. One concern is that it could result in a degradation of the quality of the information available online. Researchers have already observed an uptick in the amount of audiovisual misinformation, elaborate scams and fake websites created using generative AI tools (Hanley and Durumeric, 2023). As more and more people turn to AI assistants to autonomously create and disseminate information to public audiences at scale, it may become increasingly difficult to parse and verify reliable information. This could further threaten and complicate the status of journalists, subject-matter experts and public information sources. Over time, a proliferation of spam, misleading or low-quality synthetic content in online spaces could also erode the digital knowledge commons – the shared knowledge resources accessible to everyone on the web, such as publicly accessible data repositories (Huang and Siddarth, 2023). At its extreme, such degradation could also end up skewing people’s view of reality and scientific consensus, make them more doubtful of the credibility of all information they encounter and shape public discourse in unproductive ways. Moreover, in an online environment saturated with AI-generated content, more and more people may become reliant on personalised, highly capable AI assistants for their informational needs. This also runs the risk of homogenising the type of information and ideas people encounter online (Epstein et al., 2023)."

Weaponised misinformation agents

"Finally, AI assistants themselves could become weaponised by malicious actors to sow misinformation and manipulate public opinion at scale. Studies show that spreaders of disinformation tend to privilege quantity over quality of messaging, flooding online spaces repeatedly with misleading content to sow ‘seeds of doubt’ (Hassoun et al., 2023). Research on the ‘continued influence effect’ also shows that repeatedly being exposed to false information is more likely to influence someone’s thoughts than a single exposure. Studies show, for example, that repeated exposure to false information makes people more likely to believe it by increasing perceived social consensus, and it makes people more resistant to changing their minds even after being given a correction (for a review of these effects, see Lewandowsky et al., 2012; Ecker et al., 2022). By leveraging the frequent and personalised nature of repeated interactions with an AI assistant, malicious actors could therefore gradually nudge voters towards a particular viewpoint or sets of beliefs over time (see Chapters 8 and 9). Propagandists could also use AI assistants to make their disinformation campaigns more personalised and effective. There is growing evidence that AI-generated outputs are as persuasive as human arguments and have the potential to change people’s minds on hot-button issues (Bai et al., 2023; Myers, 2023). Recent research by the Center for Countering Digital Hate showed that LLMs could be successfully prompted to generate ‘persuasive misinformation’ in 78 out of 100 test cases, including content denying climate change (see Chapters 9 and 18). If compromised by malicious actors, in the future, highly capable and autonomous AI assistants could therefore be programmed to run astroturfing campaigns autonomously, tailor misinformation content to users in a hyperprecise way, by preying on their emotions and vulnerabilities, or to accelerate lobbying activities (Kreps and Kriner, 2023). As a result, people may be misled into believing that content produced by weaponised AI assistants came from genuine or authoritative sources. Covert influence operations of this kind may also be harder to detect than traditional disinformation campaigns, as virtual assistants primarily interact with users on a one-to-one basis and continuously generate new content (Goldstein et al., 2023)."

Increased vulnerability to misinformation

"Advanced AI assistants may make users more susceptible to misinformation, as people develop competence trust in these systems’ abilities and uncritically turn to them as reliable sources of information."

Entrenching specific ideologies

"AI assistants may provide ideologically biased or otherwise partial information in attempting to align to user expectations. In doing so, AI assistants may reinforce people’s pre-existing biases and compromise productive political debate."

Eroding trust and undermining shared knowledge

"AI assistants may contribute to the spread of large quantities of factually inaccurate and misleading content, with negative consequences for societal trust in information sources and institutions, as individuals increasingly struggle to discern truth from falsehood."

Driving opinion manipulation

"AI assistants may facilitate large-scale disinformation campaigns by offering novel, covert ways for propagandists to manipulate public opinion. This could undermine the democratic process by distorting public opinion and, in the worst case, increasing skepticism and political violence."