Relational Agents
The parasocial abyss stares back.
A few weeks ago, my young daughter discovered voice mode on ChatGPT while borrowing my phone and then disappeared into her room asking it a range of questions. About the solar system, the sun, the planets and who knows what other wide-ranging questions that she had about how the world works.
I’ve always been an advocate of AI as coach and tutor. It’s an amazing tool. Imagine me searching through twenty-four volumes of the Encyclopaedia Britannica when I was a kid! How incredible to have all the answers readily available to you, whenever you wanted to find out about something.
However, the event unsettled me to the point of wanting to know what is the state of ‘Relational’ AI, and how safe is it in the context of vulnerable humans?
How might she be perceiving this warm-sounding, natural human voice, and what may she be asking it? While the frontier labs have a range of loose controls for vulnerable people, and provide some tools to enforce that, there are none enabled on my phone.
For starters, I am not a clinical therapist, let alone a paediatric one. But I’m connected enough to know that this event represents the tip of the iceberg, and for some months now I’ve been casually researching the field of relational AI.
An epidemic of loneliness
Let’s start this with some background. It is telling that the US Surgeon General’s advisory in 2023 was labelled ‘Our Epidemic of Loneliness and Isolation’. Whether or not the high rates of loneliness are increasing remains debated, it is evident that large proportions of the population feel lonely.
While the loneliness trend in Australia overall seems to have dropped (from 39% to 29%), school loneliness is on the rise (Twenge, Haidt et al., Journal of Adolescence (2021), using PISA data from 2012 to 2018.) and in Australia, 18% of 15-year-olds feel lonely at school (PISA 2022, via the Australian Council for Educational Research).
Would it surprise you then that people are turning to AI for companionship?
The clinical evidence is patchy at best, but there are some sources to cite.
- 72% of US teens have used AI companions, and about 1 in 3 for social interaction or relationships (Common Sense Media, July 2025).
- 12% of Australian adults have used a chatbot for emotional support, rising to about 2 in 5 among Gen-Z and millennials, with 8% describing a ‘friendship’ (YouGov, June 2026).
- 0.15% of ChatGPT’s weekly users show “heightened levels of emotional attachment” (OpenAI, Oct 2025), which at their scale equates to around a million people a week.
Therefore, I think it is easy to say that the field of ‘relational’ AI is one that is only rising in importance. Yet its prevalence seems less important than a race to AGI.
Or to put it more bluntly, the intended beneficiary of these tools, humans, seems a peripheral consideration behind their generality.
Unintentional use of powerful tools.
The idea of Relational AI is not a new one; it is a twenty-year-old research field. Large frontier labs have taken their own positions, and regulators are now defining rules around the category. Though at this stage of the evolution, it is still a patchwork quilt of detection, controls and blocks that are not uniform across the tools.
The reality is these tools were just not built for relational purposes as their central function. They are general purpose tools not suitable for long and deeply specific conversations that a vulnerable person may have. While safety is layered on throughout the models, and specific disclosures made about their suitability, research shows that these safety controls degrade over longer conversations. It is also easy to jailbreak the model through persistent or alternative prompting, making prompting, as the first line of defence, extremely easy to bypass.
In fact, it may be that it is just not possible to guarantee their safety for relational AI. Google DeepMind researchers published a 2024 ethics paper where it was stated that: “…it will also be difficult to develop mitigations to make general assistants safe in all cases, whatever relationship a user establishes with them.”
Get David Jenkins’s stories in your inbox
Join Medium for free to get updates from this writer.
So instead, we are left with soft disclosures, paper restrictions and a sense of owning your own risk. Which is all fine, except that I, like many other people, have never read the fine print, and we know absolutely that relationships with AI are being developed.
- OpenAI: Age restrictions, parental controls, teen protection, age prediction and a specific service for teens.
- Google: Age rules, family rules, minor filtering and persona restrictions.
- Anthropic: 18+ only, minor detection, face verification, self-harm classifiers and crisis banners.
- xAI: Self-declaration, kids-mode, no well-being or crisis safeguards that I could find.
Despite my interest in the field of relational artificial intelligence, none of this came home until I realised my daughter was having a conversation with an ill-equipped stranger in her room. A stranger that has no sense of the real world, just a predictor of what comes next.
Is the companion a deep fake?
Well, not so fast, there are so many green shoots of what it means to have an AI companion. There are pioneers in this space looking at relational AI and how it can safely be brought into a therapeutic, clinical or relational context.
There is also an availability problem, particularly around loneliness and mental health, whether that be social stigma, cost or lack of willingness. And by availability, I mean only 52.1% of US adults with a mental illness receiving any treatment (SAMHSA). Likewise in Australia, one in five Australians had a mental disorder in the past year, and only 45.1% of them saw a health professional about it (ABS National Study of Mental Health and Wellbeing 2020–22).
In closing this gap on availability there are several pioneers where the foundation is created around the persona, the limits unable to be turned off, and a human in the loop as a necessary condition for referral. This is the opposite of a general-purpose model with disclaimers.
- Andromeda’s Abi: A companion robot for aged care. The design is deliberately non-human, follows a scripted behavioural design and will defer recommendations to a health care worker.
- Hippocratic AI: Hands off to a human nurse when a conversation goes off what it is designed to handle.
- ElliQ and Hyodol: Companions for older people that sit inside state programs with social-worker oversight.
- Dartmouth Therabot: A randomised trial of 210 people produced large benefits, with clinical supervision and a crisis pathway built in.
- LOVOT and Tombot’s Jennie are honest, non-verbal companions that don’t pretend to be people.
Still, the area of paediatrics or children is off limits for most. The company behind the Moxie child companion robot recently shelved the technology, starting again from the ground up around a different thesis. “We don’t see a future in AI companionship for children.”
Leading regulators are also moving across China, the EU, Australia and the US states, with regulation in force or in draft that limits relational AI or puts in hard brakes, for “sustained emotional interaction”.
The sense I have is there is literally too much benefit in this field for it to remain fragmented and dormant. Ultimately tools designed around human needs first will evolve within the associated regulatory safe frameworks. They will not pretend to be humans, they will be designed around specific jobs, and they will have safe foundations.
Your role as the first line of defence.
So perhaps you’re a parent, like me. Or you have an exposure to someone who is vulnerable and potentially exposed to an unsafe environment with general purpose AI agents. While the AI industry itself finds the safe space, you can also participate.
- Don’t lend your phone. Your adult phone has adult settings, and these tools are clear they are not designed for children, least of all under-13s.
- Switch on parental controls. You can switch off voice, memory, images and opt out of model training. ChatGPT includes reviewer-checked safety notifications for self-harm, though they are not real-time.
- Switch voice off. Voice has weaker controls and is more likely to feel parasocial. Turn off this immersive mode.
- Don’t rely on custom prompts. While these can provide some controls, they are the weakest and most easily bypassed. Research shows their effectiveness degrades over long conversations.
- Use your own phone’s tools. Screen-time limits, app approvals and child accounts are mechanisms that you can fully own.
- Explain the risks. Explain what artificial intelligence is and keep the lines of communication open.
Above all, I am extremely excited about the potential of this technology in a therapeutic or clinical setting, but we must wait for the environment to catch up first, before it truly delivers on a compelling promise of being your ‘forever friend’, champion or tutor.
In the meantime then, approach with the necessary caution.
If this article raises anything for you, or you’re worried about someone, please reach out. In Australia, call Lifeline on 13 11 14, Kids Helpline on 1800 55 1800 (ages 5–25), or Beyond Blue on 1300 22 4636. In the US, call or text 988. In an emergency, call 000 (Australia) or 911 (US).
