The mixed blessing of risk defences and redundancy: James Reason

A few random extracts from James Reason’s timelessly awesome Managing the Risks of Organizational accidents.

(Note: This isn’t an endorsement of the somewhat linearity of defences-in-depth, since we have evidence that emergent behaviour can playout in reality and with equifinality etc)

There’s hundreds of things I could extract (and maybe will in time), but here’s just a few from the start:

Reason outlines a number of defence types, like for creating understanding & awareness of hazards, providing guidance on how to operate within safe limits, provide alarms/warnings, restore the system to a safe state, interpose safety barriers between hazards and their loss targets, contain and eliminate hazards along a trajectory and provide a means of escape/rescue

However, “Defences-in-depth are a mixed blessing”, they can “make systems more complex, and hence more opaque, to the people who manage and operate them”

The role of human controllers in complex systems have “become increasingly remote, both physically and intellectually, from the productive systems which they nominally control”

Compared to more simplistic systems of yesteryear, “the human elements of modern technologies are often distanced from the day-to-day hazards of their respective operations”

“To anticipate and forestall disasters is to understand regularities in the ways small events can combine to have disproportionately large effects”

“Errors and violations committed by those at the sharp end are common enough in organizational accidents, but they are neither necessary nor sufficient causes … Latent conditions, however, are always present in complex systems”

“we also have a fairly good idea of the kinds of latent conditions that are most likely to constitute a threat to the safety of the system. There is no mystery to this. They relate to basic organizational processes: designing, constructing, operating, maintaining, communicating, selecting, training, supervising and managing”

“the quality of both production and protection is dependent upon the same underlying organizational processes. Safety is not a separate issue”

“we cannot prevent latent conditions from being seeded into the system since they are an inevitable product of strategic decisions. All we can usefully do is to make them visible to those who manage and operate the organization so that the worst of them, at anyone time, can be corrected”

“Redundant defensive back-ups increase the interactive complexity of high-technology organizations and thus increase the likelihood of unforeseeable common-mode failures”

“Adding redundancy makes the system more opaque to the people who nominally control and manage it”

“As a consequence of this dangerous concealment, and because of their obvious engineering sophistication, redundant defences can cause system operators and managers to forget to be afraid”

“This false sense of security prompts them to strive for even higher levels of production …

“As Perrow put it: ‘Fixes, including safety devices, often merely allow those in charge to run the system faster, or in worse weather, or with bigger explosives.”