What happens when automated clinical advice is wrong?
This reviewed research on clinician responses to incorrect automated advice. Including ECG interpretation, rules-based clinical decision support, image-based AI, selective prediction, and LLMs/GenAI.
46 articles were included.
Shared under an open access licence.
PS. Check out my YouTube: https://youtube.com/@safe_as_pod?si=V2dbdmW–WbnhzA9
Shout a coffee: https://buymeacoffee.com/benhutchinson
In all, they argue that:
“Human oversight is therefore not intrinsically protective and should be designed, tested, and monitored as a conditional error-interception capability within each technology and clinical context“.
Background:
· “Artificial intelligence is increasingly evaluated and deployed as a clinical collaborator rather than a standalone diagnostic competitor” but “comparisons between an algorithm and clinicians in isolation do not establish whether the combined human–AI system will be safe in practice”
· “The phrase “human in the loop” describes an organizational arrangement, not a demonstrated safety property”
· “A clinician may formally retain final authority yet still function as a weak safety barrier when the automated output is plausible, fluent, difficult to verify”
· Automation bias is an “inappropriate reliance on automated cues”, leading to commission errors (users follow incorrect advice) or omission (false alerts suppress an expected action)
· “Erroneous automated advice can overturn otherwise correct clinical judgments”
Findings:
· “Across three adjusted cohorts, clinician correctness was substantially higher with correct than incorrect automated advice”
· “In four heterogeneous real-world computerized ECG studies, 26.6% of erroneous interpretations were not corrected or recognized”
· “Contemporary AI and large language model studies also documented harmful reliance but were too heterogeneous for pooling”
· “Human oversight should be evaluated as an error-interception capability rather than assumed to ensure safety”
· “Across 357 disagreement opportunities, the system therefore corrected 36 errors but created 50 new errors”
· “explanations and generic warnings were not dependable safeguards”
· “…a warning that “ChatGPT can make mistakes” did not reduce diagnostic switching”
Ref: Zeng, D., Lei, J., & Li, Z. (2026). Human oversight of erroneous clinical AI and decision support a systematic review and meta analysis. npj Digital Medicine.
#HumanFactors #PatientSafety #ClinicalAI #AutomationBias #Healthcare #SafetyScience #DigitalHealth #AI #LLM
