Comparative evaluation of ambient digital scribe systems in clinical documentation
Department
Health Informatics
Document Type
Article
Publication Title
Journal of the American Medical Informatics Association
Abstract
OBJECTIVES: To demonstrate a replicable in-house framework for evaluating ambient digital scribe (ADS) systems and, applying it, to assess 3 commercial systems and examine alignment between clinician Likert ratings and a manual transcript-level error audit.
MATERIALS AND METHODS: Two standardized clinical encounters-a brief problem-focused visit and a comprehensive annual visit-were simulated and audio-recorded. Three ADS vendors generated notes from identical recordings. A multidisciplinary panel of 56 physicians rated the notes on 7-point Likert scales across 4 domains: relevance, accuracy, quality, and provider confidence. In parallel, physician reviewers cross-referenced each note against the source recording, enumerating hallucinations, omissions, and inaccuracies using a pre-specified rubric.
RESULTS: In this exploratory sample, Vendor A received the highest mean Likert ratings across all 4 domains (Friedman χ²(2) = 15.529, P < .001 short visit; χ²(2) = 10.753, P = .005 long visit). After Bonferroni correction, only Vendor A vs Vendor B on the short visit remained significant (P = .0098). The audit identified fewest errors in Vendor B's notes (n = 3), followed by Vendor A (n = 8) and Vendor C (n = 9). Physicians flagged hallucinations more often than the audit confirmed, a pattern we tentatively label "pseudo-hallucinations."
DISCUSSION: In this sample, clinician preference did not track audit-identified error counts, raising the hypothesis that note-style features influence ratings alongside documentation correctness; larger samples are required.
CONCLUSION: Pairing clinician Likert assessments with transcript-level error audits may be a useful adjunct in evaluating ADS and other generative AI clinical tools; confirmation in larger studies is needed before this framework drives procurement decisions.
First Page
2177
Last Page
2185
DOI
10.1093/jamia/ocag157
Volume
33
Issue
11
Publication Date
11-1-2026
Publisher
Hanley & Belfus
Medical Subject Headings
Humans; Documentation; Electronic Health Records
PubMed ID
42776572
Recommended Citation
Weiss, E., Zsenits, B., Nasar, F., Phillips, M., Mudgundi, V., Salhab Altamimi, T., Muth, A., Zhu, K., Kolosko, J., Hansen, S., Bui, Q. N., Yaseen, A., Balboul, M., Lee, J., & Núñez-Issac, O. A. (2026). Comparative evaluation of ambient digital scribe systems in clinical documentation. Journal of the American Medical Informatics Association, 33 (11), 2177-2185. https://doi.org/10.1093/jamia/ocag157