This experimental studied whether checklists can lead to the concealment of unlisted items in the use of checklists. That is, in some cases people using a checklist may miss *more* unlisted things compared to people using a checklist, or as the authors put it, leave people “selectively vulnerable to unlisted sources of error” (p1527).
While checklists may be effective in many contexts because they’re able to help people rapidly and efficiently focus on a few key issues – it’s said that this benefit comes at a cost: because checklists help selectively focus attention on certain listed things they may also seek to conceal unlisted issues that aren’t on the checklist.
In experiment 1, 18 adults at an aviation school were included in a scenario using a checklist. In trying to eliminate the effects of a lifetime of exposure to checklists (shopping lists etc.) that most adults would have, in experiment 2 the authors had 48 children undertake a checklist scenario (who had no prior experience to checklists).
Finally, they studied whether the unintended consequences of checklists can be eliminated by simply telling the participants that the checklist may be incomplete. Note, the findings relate mostly to learning/teaching settings but I think are interesting enough to consider for application in safety-critical contexts.
Results
As shown in the image below, participants in the checklist condition were less likely to identify unlisted issues compared to the non-checklist condition. Predictably, however, they were also more likely to detect more listed items compared to not (since that’s kind of a key purpose of a checklist…)
A possibility remained that participants in the checklist condition may have noticed the unlisted issues but didn’t list them because they thought they were only supposed to focus on the items listed on the checklist. Nevertheless, the findings suggest that this isn’t the case. They explain that 50% of the participants in the checklist condition listed at least one issue that wasn’t on the checklist; indicating an intention from people to try and list non-indicated issues.
In summarising experiment 1, the authors state that “although checklists confer an advantage relative to baseline for listed items, they impair participants’ ability to detect unlisted items” (p1529).
For experiment 2, as discussed above this sought to eliminate the effect that prior experience with checklists have on people, where it may influence their efforts towards only the listed items. Regarding children with no history of checklist use, found was that “prior experience with checklists is not necessary for learners to infer that only the listed items are important” (p1530), and this finding said to be robust.
Furthermore, even very young children “are susceptible to an inductive bias in which the presence of a checklist impairs their detection of unlisted items relative to baseline” (p1530). Telling children that there may be more items to identify & fix that aren’t on the checklist didn’t ameliorate the unintended consequences of checklists.
It’s said that the positive aspects of checklists are often promoted but rarely the unintended consequences. The things that make checklists effective at what they do may also create blindspots.
The authors nicely state that “Checklist designers are not omniscient, and our results suggest that checklists may leave learners especially vulnerable to unanticipated errors” (p1531). Further, the inferential biases “induced by checklists manifest at an early age, even before children have much experience with checklists or other kinds of formal instruction” (p1531).
Given the existing research highlighting the positive aspects of checklists, such as almost two-fold decrease in surgical mortality and other benefits, the authors aren’t proposing that pilots or surgical teams should abandon checklists.
In contexts where “checklists are normally used (where the environment is familiar, tasks are well rehearsed, the users are themselves experts, the listed items may otherwise be missed, and non-checklist errors are rare)” (p1531) then checklist use may be advantageous. However, in contexts outside of these assumptions and where unexpected sources of variability are present, then checklist use may magnify blindspots on certain issues.
In trying to counter the possible limitation of checklists, simply telling people that the checklist may be incomplete was not successful in this study.; although this was in young children and the authors admit that this intervention may be more successful in real contexts where it counts.
As another idea, the authors suggest that in safety-critical contexts perhaps an optimal compromise could be two independent observers: one with a checklist and one without. The checklist focuses on the listed items and the non-checklist user focuses on unanticipated issues.
Authors: EA Stave, PJ Muentener, LE Schulz, 2014, Proceedings of the Annual Meeting of the Cognitive Science Society
Study link: https://escholarship.org/content/qt5sp731v3/qt5sp731v3.pdf
Link to the LinkedIn article: https://www.linkedin.com/pulse/unintended-consequences-checklists-ben-hutchinson
SafetyInsights.org 