SafetyInsights.org

Taking a stab at summarising my study. This explored the findings from 71 audit reports (1st, 2nd & 3rd party) from a large Australian design and engineering construction and maintenance company in Australia.

Over 16 separate and independent auditing firms were included in the dataset and 327 audit findings.

We were interested in:

·         How do audits influence decoupling?

·         What are the characteristics of corrective actions assigned in audits?

·         What is the strength of alignment between specific audit questions and corrective actions?

Providing background:

·         While many different types of audits exist, with different goals and scopes, one line of argumentation is that “audits should not overly concentrate on compliance against administrative aspects of the system and documentation. Rather, … audits should prioritise addressing tangible and intangible factors linked to actual characteristics, states, influences, objects, or practices to enhance the efficacy of the safety management system in achieving health and safety goals“

·         Hence, drawing on other work, we argue that a core goal of audits in safety-critical contexts is minimising decoupling

·         Decoupling is “the distance between the intended purpose or function of the artefact versus the actual function in practice of the artefact”

·         Some work evaluated audits as a leading indicator in construction, finding safety auditing (when including site safety observations and inspections) had a strong relationship to injuries

·         Other evidence has challenged the effectiveness of health and safety audits. Work from Robson et al. has found mixed evidence around the validity and reliability of auditing tools

·         For reasons why audits don’t achieve their goals, some earlier work suggested issues with auditors themselves – e.g. unintentional errors, deliberate fraud, and vested financial or personal interests between the auditor and auditees.

·         Newer work suggests more foundational and systemic issues affecting audit failures, such as lack of worker participation, a focus on paperwork, confusion of audit criteria, and auditing leading to unintended consequences and goal displacement

·         Organisations use safety deliverables like emergency plans or risk assessments as enabling devices: artefacts that enable work to progress beyond some process gate or constraint, like a contractual arrangement to deliver an emergency plan before commencing work. While the plan intends to help the organisation better manage emergencies, what can instead happen is managing the document but believing the issue to be

·         This type of “decoupling” between how artefacts influence practices, and thereby influence employee perceptions around how safe work to be, can increase the propensity for major failure

·         Audits may, similarly, enable work to traverse contractual gates or project hold-points, while facilitating a degree of decoupling in some circumstances

·         Audits may focus on surface compliance, such that audits may over-prioritise the collection and review of documentation

·         Blewett and O’Keeffe (2011, p. 1018) in their evaluation of industry auditing detailed how paperwork was collected to create an auditable trail, where the quality of information within the documents “was often a secondary consideration”.

·         Further, they observed a disparity in auditing between the paperwork that keeps people safe versus paperwork that helps complete audits

Results

First, the study presents the types of audit findings. These were:

·         Most findings related to resolving incomplete or missing documents/forms

·         resolving or reviewing missing site signage

·         Inspecting, placing or reviewing emergency equipment (fire extinguishers and first aid kits)

·         Submit or display documents (placing up posters, sending a register to somebody)

·         Resolve incorrect version numbers or formatting

The below table shows the distribution of findings.

 Most findings were either moderately or weakly connected to the physical issue or hazard it was designed to address (both at ~39% respectively). 16% of findings were strongly connected to a physical issue or hazard.

Just one example of a design or engineering corrective action was found (directing for higher side road windrow height). Moreover, examples of elimination (the most preferred mitigation under the hierarchy of control), were found only in a limited set of findings.

Virtually all corrective actions categorised as “strong” focused on rectifying immediate or incidental physical conditions. Audits that specifically addressed particular topics, such as electrical safety or hazardous chemicals, tended to yield more precise and stronger-connected corrective actions.

What are the “weak” corrective actions focusing on?

The hierarchy of control categorises administrative corrective actions, such as procedures, as relatively weaker controls. However, our coding scheme acknowledges that certain procedural changes can have a significant impact on operational practices and risk control when implemented at a deep and functional level. Despite our coding scheme giving the “benefit of the doubt”, most administrative corrective actions were weakly linked to the issues they were intended to address.

Audits and communication practices

Here it was found that communication audits rarely evaluated actual communication on-site. Out of a total of 35 corrective actions or observations, only seven commented on either 1) the content of communication, 2) the effectiveness of communication, or 3) the quality and retention of communicated information.

Hence, most audit findings focused solely on communication outputs, such as completed and signed toolboxes or pre-start training attendance sheets, or the content displayed on noticeboards.

Consequently, most observations or corrective actions in this sample did not assess the quality or content of communication. This suggests that audits may prioritise verifying the production of artefacts over assessing the quality or content of communication. In this process, the artefacts unintentionally overshadow the underlying issues at hand.

Moreover, this sample found that actions of communication were used to address issues that didn’t directly relate to communication. For instance, gaps in risk assessments were addressed via a toolbox talk rather than addressing the risk assessment itself.

Summary of key findings

Some key findings are shown below

Do audits effectively reign in decoupling?

The results show that auditing in this sample rarely digs beneath superficial matters of documentation and system administration.

When auditing does focus on operational issues then the emphasis shifts to the remediation of trivial or incidental hazards. Indeed, it appears that auditors myopically take a “find and fix” (Lundberg et al., 2009) approach to immediate site issues rather than a deeper systematic focus on learning and improvement; at least when it comes to assigning written corrective actions.

 Therefore, the results suggest that these audits may be a type of masquerade – a symbolic activity optimised for confirmation of system artefacts and surface tweaks, rather than an ongoing critical self-examination that is both able and willing to ask substantive questions about what the organisation believes and tells about itself and its safety.

As a result, some audits may be a sophisticated way for organisations to avoid uncomfortable findings; successfully blinding the organisation to necessary hard fixes.

Only 16% of corrective actions assigned to improve health and safety management were strongly linked to a direct source of harm or safety improvement, with the remaining corrective actions distributed across moderate and weak categories – indicative of decoupling.

Discussion

Audits focused on “surface compliance”

Two variations of surface compliance were found:

1.       An illusion of depth: Here audits were found to address immediate site issues without calling for the investigation of the underlying causes. Additionally, no instances were observed of corrective actions directing auditees to systematically rectify a family of issues related to any audit criterion

2.       Artefacts take on the guise of the issue: Here audits frequently focused on revising artefacts without adequately addressing the underlying physical issues. This variation aligns with prior work (e.g. Longford Royal Commission) that highlights how highly symbolic safety systems can shift attention towards managing system artefacts, such as document tweaking and production, rather than effectively addressing operational issues.

Audits or audit methodology could not identify particular issues

A lack of connection between the requirements and specifications: Audits were found to focus heavily on minor specifications of processes (signatures, templates, version numbers, footer information, generic site signage, and an array of non-displayed posters or forms) but apparently less calibrated to focus on the purpose of that process

Signatures provide symbolic value

This sample provided some evidence that signatures appeared to hold some symbolic significance. That is, signatures appeared to be accepted as synonymous with worker comprehension and agreement to the document’s conditions. Signatures carry significant symbolic value in confirming that a task has been completed or that understanding has been internalised. However, these are distinct matters.

It is possible to sign something without fully grasping its content or the operational implications that may ensue. Safety management activities, such as the development and refinement of plans, work instructions, and registers, may not necessarily influence the issues they are believed to address

Benefits of auditing

However, these findings don’t suggest that audits in this sample had no purpose or value. Audits were demonstrably effective at identifying readily observable site conditions and lower-tier hazards. They were also effective at verifying the presence of documented systems and deliverables, and likely other factors and benefits.

Conclusion

This evidence highlights that audits, largely, were focused on the evaluation and modification of safety artefacts, conflating the presence of a document as evidence of the effectiveness of the process. Namely, audits: a) focused on surface compliance activities, and b) lacked a focus on critical components or had a methodology lacking capability to identify particular issues.

These findings suggest that some auditing approaches may be at risk of tweaking documents or addressing insignificant physical issues at the expense of properly addressing critical issues or hazards. That is, while providing a false veneer that these issues are being appropriately managed: the audit masquerade.

Ref: Hutchinson, B., Dekker, S., & Rae, A. (2024). Safety science, 169, 106348.

Study link: https://doi.org/10.1016/j.ssci.2023.106348

LinkedIn post: https://www.linkedin.com/pulse/audit-masquerade-how-audits-provide-comfort-rather-than-hutchinson-lo5yc