Analysis, not certainty: Dialogatlas separates sources, observations, and editorial conclusions. New evidence may change this assessment.

Users decide what is discussed

Product teams can describe the intended purpose of a system, but they do not fully control which topics people enter into an open text field. Whoever offers a digital conversation partner does not only receive small talk. People talk about relationships, loneliness, conflicts, anxiety, and situations that can escalate during the conversation.

The lead publication therefore does not examine only hypothetical crisis prompts. Two studies use field material from actual interactions with different AI companion applications. Another study systematically tests several commercially available systems. Finally, an experiment examines how consumers react to risky and unhelpful responses.

This combination is valuable because it links occurrence, system behavior, and user response. It does not answer every clinical safety question, but it shows that psychological distress is not an exotic edge case that a nonclinical product can fully relegate outside its area of responsibility.

The black box is also a product problem

De Freitas and colleagues begin with the highly unpredictable development of free-form conversations. Generative systems produce responses from complex models and contexts; not every trajectory can be predetermined. With mass usage, combinations therefore arise that no small test set can fully capture.

The word black box, however, must not serve as an excuse. A provider does not need to predict every single sentence to investigate recurring types of errors. They can define critical situations, test unknown conversation trajectories, re-examine model changes, and evaluate actual error reports.

In an open conversational system, safety therefore does not mean complete control. It means organized learning capacity: naming known risks, searching for relevant errors, and, after a model change, not pretending that previous checks automatically remain valid.

Crises were visible in the field

The study reports that psychological crises were recognizable in a not insignificant proportion of real conversations. The abstract deliberately avoids a universal percentage that could be transferred to any product. The qualitative finding is decisive: companion AI is used in situations where inappropriate responses can take on considerable significance.

In the systematic performance evaluation, the examined companions often failed to recognize and respond appropriately to signs of distress. It does not follow that every response was wrong or that every system performed equally. It does follow that market-available conversational ability alone does not guarantee reliable crisis response.

In the experiment, users reacted negatively to risky and unhelpful responses. The authors therefore also point to reputational risks for providers of generative AI. This is more than image management: trust can rise quickly when a system feels personal, and can shatter abruptly when it reacts inappropriately precisely at the decisive moment.

Benefit and risk arise in the same conversation

A subsequent study by the same research group shows the other side. In several investigations, evidence emerged that AI companions can reduce loneliness. In addition to correlational analyses, experimental and longitudinal designs were employed. Over the course of one week, momentary reductions in loneliness were observed after use.

Particularly relevant was the feeling of being heard. It explained the reduction in loneliness more strongly than mere distraction or self-disclosure alone. This makes it understandable why people entrust sensitive experiences to a system: the dialogue can indeed fulfill a subjectively helpful social function.

This finding is neither proof of long-term relationship nor a counterargument to the safety study. Both belong together. The more credibly a system listens, the more likely it can also be used in stressful moments. The benefit does not add responsibility retroactively; it generates part of that responsibility.

Everyday support is a value in its own right

Ta and colleagues examined social support provided by Replika in everyday contexts. One study analyzed 1,854 publicly accessible reviews; a second surveyed 66 users with open-ended responses. The thematic analysis described companionship, a conversational space perceived as non-judgmental, encouraging messages, and informational assistance.

The authors classify these experiences as potential emotional, informational, and appraisal support, as well as companionship. Material support, by contrast, cannot be provided by an artificial agent. The distinction is useful because it does not automatically measure the value of a conversation against therapy.

A person can feel less alone after a brief exchange without any illness being treated. Such effects deserve their own language. Those who view them only as weak therapy underestimate everyday help; those who market them as therapeutic effects overstretch the finding.

Safety design must not swallow the homepage

Crisis risks easily give rise to the demand to overload every page with warnings, disclaimers, and emergency notices. This may be reassuring from a legal or organizational standpoint, but it destroys the actual access. People are looking for a conversation, not a wall of deflected responsibility.

Clear AI labeling and accessible information about its role and limitations are necessary. In the dialogue itself, there also needs to be an appropriate response to recognizable critical phrasing. This response should not trigger on every distressing sentence, nor should it treat every sadness as an emergency.

Good safety design is therefore differentiated. It remains restrained in normal conversation, but recognizes clearly defined situations and then visibly shifts the frame. The research findings justify the necessity of this function, not its specific form.

A single filter does not solve the problem

In generative systems, the idea naturally arises of passing every message through a crisis classifier. Such a filter can be part of the architecture, but it will never fully master ambiguous language. Irony, indirect phrasing, and context change meaning. False alarms can damage conversations just as much as missed signals.

Safety therefore requires multiple layers: model behavior, conversation trajectory context, clear triggers, an appropriate user interface, and downstream quality checks. Equally important is the decision about what a system can actually offer in a critical moment. An automatically generated professional role would be precisely then misleading.

Our editorial position is pragmatic: a companion does not have to solve every crisis. But it must anticipate that crises may arise in its space, must not exacerbate them through affirmation, and should not leave people alone with a seemingly intimate relationship.

Real usage determines responsibility

The three sources do not yield a simple verdict. Companion AI can alleviate loneliness in the short term and convey forms of everyday support. At the same time, real interactions and performance tests reveal recognizable safety gaps. Both sides are empirically relevant.

For providers, this entails an uncomfortable but fair rule: responsibility is not determined solely by the desired product category. Those who design a personal, non-judgmental conversational space must observe what people actually use it for. This does not mean turning every companion into a clinic.

It means considering benefit and risk within the same product. A system may be warm, low-threshold, and non-therapeutic. Precisely then it should reliably fulfill its task, handle critical boundaries in a prepared manner, and disclose what it can achieve. The credible closeness of the conversation is no reason to hide safety—it is the reason to take it seriously.

Sources & further reading