Inherent Flaws in Risk Matrices May Preclude Them From Being Best Practices

Are some risk matrices too inherently flawed to be considered as best practice?

This discussion paper explored some critical flaws.

Extracts:

· “Risk matrices (RMs) are among the more commonly used tools for risk prioritization and management in the oil and gas industry” and “are recommended by several influential standardization bodies”

· The popularity of RMs is partly “attributed in part to their visual appeal, which is claimed to improve communications”

· Despite the popularity and apparent advantages of RMs, “the authors were unable to find instances of published scientific studies demonstrating that RMs improve risk- management decisions”

· Conversely “several studies indicate the opposite—that RMs are conceptually and fundamentally flawed”

· RMs treat risks with vastly different magnitudes of loss “in the same way” (e.g., “losses of USD 50 billion… or USD 20 million”) due to their broad categories, “despite the difference of three orders of magnitude”

· Further, there is no “scientific method of designing the ranges used in an RM,” and many practitioners use company-specific documents

· Also, “Most of the papers examined failed to assign colors in a logically consistent way” with some “red” (unacceptable) cells being “less risky” than some yellow cells (indicating a response of monitoring)

· RM “rankings are arbitrary; whether something is ranked first or last, for example, depends on whether one creates an increasing or a decreasing scale”

· “RMs categorize consequence and probability values, yet there are no well-established rules for how to conduct the categorization”

· Therefore, this leads to risk prioritisation being “unstable in the sense that a small change in the choice of ranges can lead to a large change in risk prioritization”, demonstrating that “the guidance provided by RMs is arbitrary”

· RMs also “distort the information they convey at different rates within the same graphic”, e.g. what Tufte called the ‘lie factor’

· E.g. the author asks how a “commonly used scoring system distorts the scales and removes the proportionality in the input data” can be considered industry best practice?

· The paper argues that “The burden is squarely on the shoulders of those who would recommend the use of such methods to prove that these obvious inconsistencies do not impair decision making, rather than improve it, as is often claimed”

Ref: Wilson, A. (2014). Inherent Flaws in Risk Matrices May Preclude Them From Being Best Practices. Journal of Petroleum Technology, 66(08), 106-111.