SafetyInsights.org

Happy to share our latest study on audit failures. It is full open access.

This study focused on the mechanisms of how safety audits fail, according to accident investigations. After screening thousands of accident reports, we were left with a sample of 44 reports for evaluation (primarily from the US and the CSB).

We were particularly interested in what the investigators expected of audits prior to accidents and hence, their counterfactual expectations (e.g. should, could, would have etc.)

We grouped the core findings into nine counterfactual themes under four categories. Audit failure mechanisms involved:

1) A failure to understand (by misinterpreting the saliency of critical risks)

2) A failure to act (on known issues or prevent their deterioration)

3) A failure to manage (by not implementing effective auditing schedules or audit scopes)

4) A failure to focus (by lacking focus on critical items, focusing on surface compliance, or failing silently)

Some particularly interesting findings, in my view, were that:

1) Few accident reports discuss the context or relevancy of auditing. That is, based on major accident reports, we have little indication of whether or not audits had a contributory role (But Kudos to the US CSB, who far more regularly evaluate the role of audits).

2) Audits have important limitations as indicators of weak signals

3) Some audits can “fail silently”. This suggests that audits can fail to achieve their critical goals, but a) provide little intel that they are failing, or b) provide glowing reports of system effectiveness while masking substantial deficiencies.

We’ve also tabulated all of the specific audit failure types identified by the investigators in a supplementary appendix; this isn’t included as part of the main manuscript so you’ll need to follow the link (see comments).

Study link: https://doi.org/10.1002/prs.12579

Supplementary data: https://aiche.onlinelibrary.wiley.com/action/downloadSupplement?doi=10.1002%2Fprs.12579&file=prs12579-sup-0001-TableS1.docx

Link to the LinkedIn article: https://www.linkedin.com/posts/benhutchinson2_happy-to-share-our-latest-study-on-audit-activity-7157143270492459008-LbDy?utm_source=share&utm_medium=member_desktop