Authors:	Jungmann, Stefanie Klan, Timo Kuhn, Sebastian Jungmann, Florian
Title:	Accuracy of a chatbot (Ada) in the diagnosis of mental disorders : comparative case study with lay and expert users
Online publication date:	27-Nov-2019
Year of first publication:	2019
Language:	english
Abstract:	Background: Health apps for the screening and diagnosis of mental disorders have emerged in recent years on various levels (eg, patients, practitioners, and public health system). However, the diagnostic quality of these apps has not been (sufficiently) tested so far. Objective: The objective of this pilot study was to investigate the diagnostic quality of a health app for a broad spectrum of mental disorders and its dependency on expert knowledge. Methods: Two psychotherapists, two psychology students, and two laypersons each read 20 case vignettes with a broad spectrum of mental disorders. They used a health app (Ada—Your Health Guide) to get a diagnosis by entering the symptoms. Interrater reliabilities were computed between the diagnoses of the case vignettes and the results of the app for each user group. Results: Overall, there was a moderate diagnostic agreement (kappa=0.64) between the results of the app and the case vignettes for mental disorders in adulthood and a low diagnostic agreement (kappa=0.40) for mental disorders in childhood and adolescence. When psychotherapists applied the app, there was a good diagnostic agreement (kappa=0.78) regarding mental disorders in adulthood. The diagnostic agreement was moderate (kappa=0.55/0.60) for students and laypersons. For mental disorders in childhood and adolescence, a moderate diagnostic quality was found when psychotherapists (kappa=0.53) and students (kappa=0.41) used the app, whereas the quality was low for laypersons (kappa=0.29). On average, the app required 34 questions to be answered and 7 min to complete. Conclusions: The health app investigated here can represent an efficient diagnostic screening or help function for mental disorders in adulthood and has the potential to support especially diagnosticians in their work in various ways. The results of this pilot study provide a first indication that the diagnostic accuracy is user dependent and improvements in the app are needed especially for mental disorders in childhood and adolescence.
DDC:	150 Psychologie 150 Psychology
Institution:	Johannes Gutenberg-Universität Mainz
Department:	FB 02 Sozialwiss., Medien u. Sport FB 04 Medizin
Place:	Mainz
ROR:	https://ror.org/023b0x485
DOI:	http://doi.org/10.25358/openscience-235
URN:	urn:nbn:de:hebis:77-publ-594315
Version:	Published version
Publication type:	Zeitschriftenaufsatz
License:	CC BY
Information on rights of use:	https://creativecommons.org/licenses/by/4.0/
Journal:	JMIR formative research 3 4
Pages or article number:	e13863
Publisher:	JMIR Publications
Publisher place:	Toronto
Issue date:	2019
ISSN:	2561-326X
Publisher URL:	http://dx.doi.org/10.2196/13863
Publisher DOI:	10.2196/13863
Appears in collections:	JGU-Publikationen