Risks of automation and removing people

Lisanne Bainbridge’s 1982 paper on the ironies of automation is a classic; I’ve posted extracts before (as have many others – seriously, the paper is worth your time).

James Reason (source below, pg. 43) provided a nice summary of the paper’s key points in the attached image. But, briefly, efforts to automate systems and remove humans can introduce other unexpected and sometimes catastrophic failure modes.

Relating to aviation, Reason notes another irony of automation being that while computerised systems can remove some degrees of local performance variability (via influencing performance shaping factors), it can also “increase the probability of higher-level mistakes with the capacity to cause the destruction of the entire aircraft and its occupants” (p45); such as reducing cognitive burdens of horizontal navigation but sometimes increasing the burden of more complex and safety-critical vertical navigation tasks.

Drawing on research from David Woods and colleagues, it’s noted that automation can: (pp45-46)

Increase user memory demands
Increase uncertainty as to where users should focus attention
Increase difficulty for team members to share a distribution awareness
Impair mental models of the system and increase workload during high-demand periods
Limit a user’s ability to develop effective strategies for coping with task demands

Further, Woods & Sarter identify other properties of automated systems which promote these problems: (p.46)

They can hide interesting events, changes and anomalies [* automation transparency comes to mind]
They force serial access to highly related information
They offer a ‘keyhole’ view of a limitless virtual space
They contain complex and arbitrary sequences of operations and modes
They can suppress important information about the activities of other team members

Source: Reason (1997). Managing the risks of organizational accidents. Ashgate Publishing