“Are You Sure Your Software Will Not Kill Anyone?” (Nancy Leveson on complex software)

Not another CrowdStrike post – but this reminded me of some comments from Nancy Leveson about the interactively complex and sometimes “impossible” task of thoroughly understanding software failure modes.

Some cherrypicked examples from several of Nancy’s sources:

· “Complexity has many facets, most of which are increasing in the systems we are building, particularly interactive complexity”

· “We are designing systems with potential interactions among the components that cannot be thoroughly planned, understood, anticipated, or guarded against”

· “The operation of some systems is so complex that it defies the understanding of all but a few experts, and sometimes even they have incomplete information about its potential behavior.”

· “Software is an important factor here—it has allowed us to implement more integrated, multi-loop control in systems containing large numbers of dynamically interacting components where tight coupling allows disruptions or dysfunctional interactions in one part of the system to have far-ranging rippling effects”

· “The problem is that we are attempting to build systems that are beyond our ability to intellectually manage: Increased interactive complexity and coupling make it difficult for the designers to consider all the potential system states or for operators to handle all normal and abnormal situations and disturbances safely and effectively”

· Use of software testing and simulation can never be comprehensive or complete in complex software systems, moreover “simulation can show only that we have handled the things we thought of, not the ones we did not think about, assumed were impossible, or unintentionally left out of the simulation environment”

· Regarding, formal verification “Virtually all accidents involving software stem from unsafe requirements, not implementation errors… When I look at accidents where it is claimed the implemented software logic has led to the loss, I always find the software logic flaws stem from a lack of adequate requirements”

· “Formal verification (or even formal validation) can show only the consistency of two formal models. Complete discrete mathematical models do not exist of complex physical systems”

· “Software allows us to build systems of such high interactive complexity that it is difficult, and sometimes impossible, to plan, understand, anticipate, and guard against potential unintended interactions”