A brief read covering the concept of noise, pertaining to judgements. This is based on the work from Kahneman, Sibony and Sunstein.
From the article:
· While biases in judgements have captured a lot of attention, “it has been suggested that ‘noise’ (defined as an undesirable variability in human judgements) is a highly important, yet under-recognised source of error in human decision-making”
· Examples of noise are things like variability in sentencing decisions for two matters which are close to identical, or variability in clinical decisions between different practitioners for something which may warrant similar clinical decisions
· In the legal sentencing example, Kahneman et al. identified how hot ambient weather influenced tougher sentencing, or whether the judge’s sports team lost the night before – these things introduce noise (variability) where it is undesirable or unwarranted
· In the safety context, noise could be variability in risk assessment scores between different practitioners or groups for the same matter
· In contrast to biases and heuristics, which are difficult and not always beneficial to try to counter, noise (as undesirable variability) is seen to be easier to tackle
· Kahneman et al. defined bias as “predictable, systematic errors skewed in the same direction” and “noise [as] randomly scattered error due to judgement variability”
· Image 1 visualises bias vs noise
· It’s argued that “noise remains an under-recognised but highly prevalent issue (to the extent that ‘wherever there is judgment, there is noise”
· For medicine, the paper says there’s “significant geographical/regional heterogeneity in terms of offered clinical interventions for common conditions, which are generally reflective of differences in local physician training and practices” and more
· “besides inter-physician practice differences, studies have also found intra-physician variability in prescription habits (e.g. opioids, antibiotics) attributed to environmental factors such as time of the day, where common patient-requested, but clinically less appropriate medications tend to be prescribed more liberally towards the end of the clinic session or work day when there is greater physical/decisional fatigue”
· Image 2 shows a comprehensive list of factors related to noise in clinical decision making
· They provide some suggestions on reducing noise, for one routine ‘noise’ audits which measure the extent and source of noise in decisions, which can be done via different means like surveys, scenario training, workshops, document review and more
· [* Again in the safety world, this could include plotting scores from risk assessments and seeing how much variability there is, or which controls are implemented for similar work tasks, or financial decisions, or if you have metarules/rules for departing rules, then tracking how they are used]
- Noise is normal and everywhere, so the goal of the noise audit is to determine how widespread it is and establish some boundaries on what “what is considered an acceptable/healthy level of practice variability”
- They provide a bunch of healthcare specific ways to tackle variability – I’ve skipped most of this but includes clinical reasoning skill enhancement, a Bayesian-centric testing-and-treatment threshold model, consensus guidelines, AI support systems, “choice architecture with appropriate behavioural nudges (e.g. pre-orders/default options to optimise prescribing habits”
- Other ideas include collating aggregate judgements of experts, challenging assumptions with differential explanations
- Other ideas include heuristics-based decision hygiene templates, e.g. a pivot-and-cluster strategy or three-step clinical reasoning routine, and more
- Then they cover individual factors like stress etc. and improving the work environment like duty hours, work intensity, interruptions, appropriate support systems and more
Ref: Ng, I. K., Morgan, D. J., & Tow Keang, L. (2025). Why do doctors make poor decisions? Spotlighting ‘noise’as an under-recognised source of error in clinical practice. Journal of the Royal Society of Medicine, 01410768251324822.
Study link: https://doi.org/10.1177/01410768251324822
SafetyInsights.org 