Analysis, not certainty: Dialogatlas separates sources, observations, and editorial conclusions. New evidence may change this assessment.

Thirteen studies, but no uniform subject matter

The systematic review by Hannah Gaffney, Warren Mansell, and Sara Tai is the focus. The researchers searched MEDLINE, EMBASE, PsycINFO, Web of Science, and the Cochrane Library. Included were studies of autonomous systems that simulated conversations and reported at least one mental health outcome. This meant the subject matter extended beyond text-based applications and also included other forms, including robots. In total, 13 studies met the criteria.

Four of these were full-scale randomized controlled trials. The remaining works consisted of feasibility studies, randomized pilot studies, and quasi-experimental investigations. This composition alone limits sweeping judgments. In addition, there was great content diversity: the systems addressed different mental health problems, were designed differently, and followed different therapeutic orientations. The category of conversational agent thus refers here more to a technical form of interaction than to a unified intervention.

Improvement is a signal to be taken seriously

The most striking finding of the review is initially positive: all included studies reported a reduction in psychological distress after the intervention. This should neither be downplayed nor prematurely reinterpreted. A research field in which early systems are repeatedly associated with improvements has plausible developmental value. The authors accordingly assessed efficacy and acceptability as promising, but demanded more robust experimental designs.

The crucial phrase is “after the intervention.” Before-and-after changes show a temporal association but do not explain it. Without an appropriate comparison, it remains open what share is attributable to the conversational agent, to expectations, to structured self-observation, or to other influences. This methodological limitation does not erase the observed benefit. Rather, it determines how far that benefit may be interpreted.

The control group changes the story

Five controlled studies found a significantly greater reduction in psychological distress compared with inactive control groups. That is more than a mere finding on usage or satisfaction: in these studies, the results differed between groups. The review thus does provide indications that digital conversational interventions can be effective. However, it provides no reason to attribute this effect without further ado to every system, every target group, or every application context.

Three other controlled studies compared the conversational agents with active control conditions. There, no superiority could be demonstrated. This is precisely where the core conflict lies. An inactive control group primarily answers whether the offering performs better than no corresponding activity. An active comparison condition poses the more demanding question: does the automated conversation add something extra when people already receive another form of support or activity? According to the review’s findings, this question remained open.

Not superior does not mean equivalent

The absence of evidence of superiority does not imply that conversational agents are ineffective. Nor does it imply that they would be equivalent to an active alternative. A study that shows no superiority is not yet evidence of equivalence. The reasons behind the respective null findings cannot be determined from the information provided. A robust editorial assessment must therefore distinguish between a lack of evidence for additional benefit and evidence against any benefit.

Gaffney, Mansell, and Tai name precisely the next scientific tasks: interventions should be streamlined, possible equivalence with other forms of treatment should be specifically investigated, and mechanisms of action should be clarified. Efficiency also needs to be demonstrated more convincingly. Our position on this is clear: the active comparison is not a subsequent stress test for already finished products. It determines what performance can be attributed to the conversation as such.

The two-week pilot shows why usage matters

The pilot study by Kien Hoa Ly, Ann-Marie Ly, and Gerhard Andersson expands the picture to include usage practice. A fully automated conversational agent for promoting psychological well-being was examined in a randomized mixed-methods design. During the two-week intervention, the app was opened an average of 17.71 times. The authors rated this engagement as higher than that in the studies they cited on Woebot and Panoply. The qualitative analysis also described a previously unreported sub-theme regarding the moderating role of the system.

This is a constructive finding: full automation did not have to be associated with low usage in this study. The researchers saw the good adherence in particular as a reason for a replication with a larger sample and an active control group. However, the accessible source material does not extend further. The provided abstract is incomplete and does not name the sample size nor fully reconstructable outcome values for efficacy. These details therefore cannot be responsibly supplemented.

Psychiatric breadth meets narrow evidence

The review by Aditya Vaidyam and colleagues examined the use of conversational agents in screening, diagnosis, and treatment in psychiatry. Their systematic search from June 2018 yielded 1,466 records. Eight studies met the inclusion criteria; two additional studies were added after screening reference lists. Populations with mental disorders or increased risk were considered, including depression, anxiety, schizophrenia, bipolar disorders, and substance-related disorders.

The authors assessed the preliminary evidence as generally favorable. They saw particular potential in psychoeducation and adherence. Satisfaction scores were high across the included studies. This supports the assumption that conversational agents can be experienced as acceptable and pleasant tools. However, high satisfaction alone does not imply a proven clinical effect. This review also points to the heterogeneity of the studies and calls for more standardized outcome reporting before effectiveness can be assessed more thoroughly.

Acceptance is not a surrogate measure, but neither is it a trivial matter

Across the three publications, a cautious pattern emerges: the systems were used, satisfaction was often high, and the main review described their acceptance as promising. This is not evidence of a specific treatment effect. Nevertheless, it is relevant for product development and care practice. An intervention that is barely opened or quickly abandoned cannot convey its intended content. Engagement is therefore a necessary practical prerequisite, even if it does not constitute a sufficient efficacy test.

From an editorial perspective, we consider it wrong to dismiss this level as mere accessory. The conversational format could have an independent value precisely in making structured content accessible and repeatedly usable. Whether this value stems from conversation dynamics, design, reminders, personalization, or other characteristics is not clarified by the available sources. The positive user experience should therefore be taken seriously as a development finding, without renaming it as evidence of efficacy.

What is sought is the additional contribution of the conversation

The three studies justify neither blanket enthusiasm nor blanket dismissal. They show an early research landscape with recognizable improvements, favorable acceptance signals, and isolated controlled effects. At the same time, the evidence becomes thin precisely where an active alternative makes the comparison more demanding. The benefit for the general well-being of nonclinical populations also remained unclear in the main review; the two-week pilot primarily provides an engagement signal in this regard.

The next step in the research therefore does not consist in having further systems merely compete against non-use. It consists in determining the additional contribution of the automated conversation compared with a credible active alternative and explaining its mechanisms. Until then, the appropriate attitude is one of engaged methodological rigor: conversational agents are interesting interventions that are apparently often accepted. But their particular value only begins where the comparison shows what the conversation itself achieves.

Sources & further reading