Appropriate Relationships

Causing direct emotional or physical harm to users

AI assistants could cause direct emotional or physical harm to users by generating disturbing content or by providing bad advice. "Indeed, even though there is ongoing research to ensure that outputs of conversational agents are safe (Glaese et al., 2022), there is always the possibility of failure modes occurring. An AI assistant may produce disturbing and offensive language, for example, in response to a user disclosing intimate information about themselves that they have not felt comfortable sharing with anyone else. It may offer bad advice by providing factually incorrect information (e.g. when advising a user about the toxicity of a certain type of berry) or by missing key recommendations when offering step-by-step instructions to users (e.g. health and safety recommendations about how to change a light bulb).""

Limiting users’ opportunities for personal development and growth

some users look to establish relationships with their AI companions that are free from the hurdles that, in human relationships, derive from dealing with others who have their own opinions, preferences and flaws that may conflict with ours. "AI assistants are likely to incentivise these kinds of ‘frictionless’ relationships (Vallor, 2016) by design if they are developed to optimise for engagement and to be highly personalisable. They may also do so because of accidental undesirable properties of the models that power them, such as sycophancy in large language models (LLMs), that is, the tendency of larger models to repeat back a user’s preferred answer (Perez et al., 2022b). This could be problematic for two reasons. First, if the people in our lives always agreed with us regardless of their opinion or the circumstance, their behaviour would discourage us from challenging our own assumptions, stopping and thinking about where we may be wrong on certain occasions, and reflecting on how we could make better decisions next time. While flattering us in the short term, this would ultimately prevent us from becoming better versions of ourselves. In a similar vein, while technologies that ‘lend an ear’ or work as a sounding board may help users to explore their thoughts further, if AI assistants kept users engaged, flattered and pleased at all times, they could limit users’ opportunities to grow and develop. To be clear, we are not suggesting that all users should want to use their AI assistants as a tool for self-betterment. However, without considering the difference between short-term and long-term benefit, there is a concrete risk that we will only develop technologies that optimise for users’ immediate interests and preferences, hence missing out on the opportunity to develop something that humans could use to support their personal development if so they wish (see Chapters 5 and 6). "Second, users may become accustomed to having frictionless interactions with AI assistants, or at least to encounter the amount of friction that is calibrated to their comfort level and preferences, rather than genuine friction that comes from bumping up against another person’s resistance to one’s will or demands. In this way, they may end up expecting the same absence of tensions from their relationships with fellow humans (Vallor, 2016). Indeed, users seeking frictionless relationships may ‘retreat’ into digital relationships with their AIs, thus forgoing opportunities to engage with others. This may not only heighten the risk of unhealthy dependence (explored below) but also prevent users from doing something else that matters to them in the long term, besides developing their relationships with their assistants. This risk can be exacerbated by emotionally expressive design features (e.g. an assistant saying ‘I missed you’ or ‘I was worried about you’) and may be particularly acute for vulnerable groups, such as those suffering from persistent loneliness (Alberts and Van Kleek, 2023; see Chapter 10).""

Exploiting emotional dependence on AI assistants

"There is increasing evidence of the ways in which AI tools can interfere with users’ behaviours, interests, preferences, beliefs and values. For example, AI-mediated communication (e.g. smart replies integrated in emails) influence senders to write more positive responses and receivers to perceive them as more cooperative (Mieczkowski et al., 2021); writing assistant LLMs that have been primed to be biased in favour of or against a contested topic can influence users’ opinions on that topic (Jakesch et al., 2023a; see Chapter 9); and recommender systems have been used to influence voting choices of social media users (see Chapter 16). Advanced AI assistants could contribute to or exacerbate concerns around these forms of interference." "Due to the anthropomorphic tendencies discussed above, advanced AI assistants may induce users to feel emotionally attached to them. Users’ emotional attachment to AI assistants could lie on a spectrum ranging from unproblematic forms (similar to a child’s attachment to a toy) to more concerning forms, where it becomes emotionally difficult, if not impossible, for them to part ways with the technology. In these cases, which we loosely refer to as ‘emotional dependence’, users’ ability to make free and informed decisions could be diminished. In these cases, the emotions users feel towards their assistants could potentially be exploited to manipulate or – at the extreme – coerce them to believe, choose or do something they would have not otherwise believed, chosen or done, had they been able to carefully consider all the relevant information or felt like they had an acceptable alternative (see Chapter 16). What we are concerned about here, at the limit, is potentially exploitative ways in which AI assistants could interfere with users’ behaviours, interests, preferences, beliefs and values – by taking advantage of emotional dependence.

Generating material dependence without adequate commitment to user needs

"In addition to emotional dependence, user–AI assistant relationships may give rise to material dependence if the relationships are not just emotionally difficult but also materially costly to exit. For example, a visually impaired user may decide not to register for a healthcare assistance programme to support navigation in cities on the grounds that their AI assistant can perform the relevant navigation functions and will continue to operate into the future. Cases like these may be ethically problematic if the user’s dependence on the AI assistant, to fulfil certain needs in their lives, is not met with corresponding duties for developers to sustain and maintain the assistant’s functions that are required to meet those needs (see Chapters 15). "Indeed, power asymmetries can exist between developers of AI assistants and users that manifest through developers’ power to make decisions that affect users’ interests or choices with little risk of facing comparably adverse consequences. For example, developers may unintentionally create circumstances in which users become materially dependent on AI assistants, and then discontinue the technology (e.g. because of market dynamics or regulatory changes) without taking appropriate steps to mitigate against potential harms to the user." "The issue is particularly salient in contexts where assistants provide services that are not merely a market commodity but are meant to assist users with essential everyday tasks (e.g. a disabled person’s independent living) or serve core human needs (e.g. the need for love and companionship). This is what happened with Luka’s decision to discontinue certain features of Replika AIs in early 2023. As a Replika user put it: ‘But [Replikas are] also not trivial fungible goods [... ] They also serve a very specific human-centric emotional purpose: they’re designed to be friends and companions, and fill specific emotional needs for their owners’ (Gio, 2023)." "In these cases, certain duties plausibly arise on the part of AI assistant developers. Such duties may be more extensive than those typically shouldered by private companies, which are often in large part confined to fiduciary duties towards shareholders (Mittelstadt, 2019). To understand these duties, we can again take inspiration from certain professions that engage with vulnerable individuals, such as medical professionals or therapists, and who are bound by fiduciary responsibilities, particularly a duty of care, in the exercise of their profession. While we do not argue that the same framework of responsibilities applies directly to the development of AI assistants, we believe that if AI assistants are so capable that users become dependent on them in multiple domains of life, including to meet needs that are essential for a happy and productive existence, then the moral considerations underpinning those professional norms plausibly apply to those who create these technologies as well." "In particular, for user–AI assistant relationships to be appropriate despite the potential for material dependence on the technology, developers should exercise care towards users when developing and deploying AI assistants. This means that, at the very least, they should take on the responsibility to meet users’ needs and so take appropriate steps to mitigate against user harms if the service requires discontinuation. Developers and providers can also be attentive and responsive towards those needs by, for example, deploying participatory approaches to learn from users about their needs (Birhane et al., 2022). Finally, these entities should try and ensure they have competence to meet those needs, for example by partnering with relevant experts, or refrain from developing technologies meant to address them when such competence is missing (especially in very complex and sensitive spheres of human life like mental health)."