Safety indicators: questioning the quantitative dominance

This paper challenges the dominance of quantitative safety indicators in construction and argues for the addition of qualitative indicators, to better inform quantitative.

First, the matter of safety vs unsafety is touched upon. It’s said that in one view safety has been defined as an absence of undesirable occurrences. This “makes it problematic to measure, as unlike many other key performance indicators that rely on action to occur, safety can be deemed successful, without action” (p11).

The above is more akin to measuring “unsafety”, rather than the necessary conditions, resources, states etc. to ensure safe and reliable performance.

It’s discussed that “precise measurements” and quantitative indicators, including leading, were largely borrowed from economics and this is partially responsible for the intensification in precision measurement. This may be ideal for economics, but their relevance for large complex construction projects is challenged in certain ways.

The author covered some known limitations of quantitative indicators (lagging & leading). This includes manipulation, focusing on the easily measurable (rather than important), measuring unsafety more than safety, and measuring things that were going to be done anyway. Existing quantitative indicators are seen as “golden numbers”, which are given great importance but may not be properly calibrated to the complexities of factors that influence performance.

That is, there has been historically a stronger focus on measuring the “clearly measurable” things, resulting in less focus on other indicators which involve more ambiguity and complexities of real work.

Some issues with indicators, like injury and incident statistics, is statistical validity, variability and sample size. Quoting the paper “As Hopkins (2009) notes, recordable injuries and incidents have a low probability of occurring over short time frames, and when measured at a single construction project, they are usually neither stable nor valid. Hence, accident frequency rates (and other lagging indicators) should be treated with caution as some are potentially unreliable” (p12).

Further, for leading indicators, it is simply assumed (without necessarily strong evidence) that counting the number of safety walks, toolbox talks etc. is correlated with quality and desirable performance outcomes. Instead, “the quality, effectiveness or content of these activities is not considered, meaning twenty ‘tick-box’ toolbox talks appear far superior than two empowering and engaging toolbox talks” (p14).

There are other considerations with the leading / lagging dichotomy. One is that, as shown elsewhere, these definitions aren’t necessarily stable. Lagging and leading can come to swap directions, where lagging lead and leading lag. In other cases, leading and lagging can be considered one or the other depending on how they’re used.

The author highlights the addition of qualitative indicators. Quantitative are seen as more “factual” (counting stuff), whereas qualitative is more subjective and provides feedback on experiences and perspectives. For instance, does an increase in near miss reports indicate higher risk exposures or better reporting? qualitative can provide necessary context around this question, whereas too much focus on quantitative indicators may “risk sacrificing quality” (p14) and potentially omits important information explaining the quantitative number.

Much has been written elsewhere about the manipulation, gaming or “safe-washing” of statistics, incident definitions etc. The author touches upon this in this paper. For instance indicators can be manipulated to achieve targets, like suitable duties, early return to work and other ways.

Further, with the manipulation of indicators – instead of seeing this as a statistical limitation, it’s suggested to be an opportunity to better learn about the nature of the organization via qualitative exploration and indicators. One study found that worker-submitted safety observations were believed to have been vetted by management; with this perception of vetting ripe for qualitative indicators to explore.

The author suggests that for every quantitative indicator – there should be a corresponding qualitative indicator to explain the quantitative and provide rich context. Qualitative indicators can provide feedback on the quality of interventions, compared to quantitative.

Likewise, combining qualitative and quantitative is likely to be “complementary … [where] qualitative indicators can provide information about if, and how, organizational conditions support safety” (p14).

It’s recognized that some qualitative indicators may already be collected (audits, surveys, discussions) but aren’t currently used to assess this type of performance.

In all, it’s argued that if you want indicators to inform on how and why a particular quantitative indicator is what it is, then this can “only be captured qualitatively” (p16). Thus, indicators should leverage the strengths of both quantitative and qualitative to capture the realities of work.

The author states “On their own, traditional statistical lagging indicators have not provided enough insight into effectively avoiding future accidents … This is perhaps because we do not currently have the qualitative tools necessary to understand safety comprehensively” (p13).

Author: Oswald, D. (2020). Construction Management and Economics, 38(1), 11-17.