A cool paper from Richard Cook about internet/software failures and complex systems.
I’m not a software person – but still found it pretty interesting.
** Parts 2 & 3 in comments **
Just a few extracts:
- “Problems that require intervention crop up regularly” and “Publicly, companies usually describe these events as sporadic and minor— systemically equivalent to a cold or flu that is easily treated at home”
- “Even a cursory look inside, however, shows a situation more like an intensive care unit: continuous monitoring, elaborate struggles to manage related resources, and many interventions by teams of around-the-clock”
- “Far from being hale and hearty, these are brittle and often quite fragile assemblies that totter along only because they are surrounded by people who understand how they work”
- “The people who intervene in failure are often the same people who built the stuff in the first place … They participated in the intricacies, dependencies, and assumptions that produced and arranged these artifacts”
- Humans have been “indissolubly [linked]” into machines in a “continuously changing, nondeterministic, fully distributed system”
- Richard talks of lines of representation for an internet-facing system. The horizontal line is all the representations available to people working above that line – display screens, keyboards and below the line are the technical artefacts – code libraries, compilers, stacks and more
- The line represents, in part, the delineation between what can be directly manipulated and that which cannot, “Computing is detectable only via representations synthesized to show its passing. Similarly, it can be manipulated only via representations”
- “Above the line of representation are the people, organizations, and processes that shape, direct, and restore the technical artifacts that lie below that line”
- “People who work above the line routinely describe what is below the line using concrete, realistic language”
- “remarkably, nothing below the line can be seen or acted upon directly. The displays, keyboards [etc] that constitute the line of representation are the only tangible evidence that anything at all lies below the line”
- “All understandings of what lies below the line are constructed” built from “inferences made fromrepresentations that appear on thescreens and displays”
- “These inferences draw on our mental models—those that have been developed and refined over years”
- “It will be immediately apparent that no individual mental model can ever be comprehensive” and “any complete model will be stale and that any fresh model will be incomplete”
- This is covered by Woods’ theorem: “As the complexity of a system increases, the accuracy of any single agent’s own model of that system decreases rapidly”
- “This is a complex system; it is always changing. The composition and arrangement of the components are such that the system’s behavior is nondeterministic”
- “All models of the system are approximations. It is impossible to anticipate all the ways that it might break down or defend against all eventualities”
- “As the complexity below the line has increased, so too has the complexity above the line”
- “The job of bringing expertise to bear and coordinating the application of that expertise is nontrivial and often undertaken under severe time and consequence pressure”
- “Similar faults and failures can occur above and below the line. The reverberation across the line of representation tends to shape the structure below the line (particularly the functional boundaries)”
- “Because structure and function above the line parallel structure and function below, parallels can be expected in the forms of dysfunction that can occur”
- “It’s one system, not two. The line of representation appears to be a convenient boundary separating two “systems,” a technical one below the line and a human one above it”
- However, “People are constantly interacting with technologies below the line; they build, modify, direct, and respond to them”
- “These interactions weld what is above the line to what is below it. There are not two systems separated by a representational barrier; there is only one system”.
Ref: Cook, R. I. (2020). Above the line, below the line. Communications of the ACM, 63(3), 43-46.
Study link: https://dl.acm.org/doi/pdf/10.1145/3379510
SafetyInsights.org 