SafetyInsights.org

This covers some core problems with risk matrices.

It’s argued that while they’re established tools, appearing to be “authoritative, and intellectually rigorous”, this “could be just an illusion …bred by the human bias of uncertainty aversion and authority bias”.

Hence, matrices have “many flaws” that can “diminish their usefulness to the point where they become even worse than random”.


Some extracts:

·    They involve “ordinal scales, semi-quantitative arithmetics, range compression, risk inversion, ambiguity, and neglection of uncertainty”

·    “arithmetic operations like addition or multiplication are undefined”, leading to a loss of “ability to do reasonable arithmetic, estimate uncertainty, or do any sophisticated mathematical analysis”

·    Somewhat arbitrary decisions resulting from number of ranks and assignment of quantitative & semiquantitative numbers

·    An issue of range compression, which is “pressing the real values into a scheme of ordinal scales” causes “original uncertainty ranges get lost”

·    Ambiguity within scales which are “often not defined precisely”,  leading to differing judgments based on expert opinion

·    A neglect of uncertainty where “By classifying, the original uncertainty in the judgment gets lost”

·    Quantification of errors can happen at borders, resulting in slight value changes that “change the result tremendously”

·    The design of matrices can elevate inherent human biases like with anchoring and framing

·    It’s argued that calculations in matrices can be performed “without any foundations or support from mathematics”, because these operations aren’t supported by ordinal scales

·    The ways we combine ratings are often “chosen arbitrarily” with “no correct way to do this” on ordinal scales

·    Matrices can lead to “worse than random results” due to the neglect of correlations, which is said to be a “most overlooked problem”

·    Neglect of correlations means that matrices can often fail to account for how different risk factors influence each other, leading to inaccurate overall assessments of risk

·    The thresholds for final risk levels are “often chosen completely arbitrarily”

·    Ordinal scales contribute to an issue of consistency and “coherence properties”, like where “an event with lower risk might get a higher score than an event with actual higher risk”

·    Risk inversion is said to be a “very sever problem”, meaning that “Lower risks might get a higher score than actually higher risk or vice versa”

·    The perceived simplicity of semi-quantitative methods might lead to a “wrong impression of benefits”, more illusion than strength

·    Interestingly, it’s argued that there is a deferred feedback in matrices, where instead of providing immediate feedback for improvement, risk matrix use means feedback is deferred for years, meaning assessments are “seldom reviewed for correctness”

The author provides some suggestions:

1.      “avoiding risk matrices altogether”, and instead, “using fully quantitative risk assessment methods” [*** Meh…I’m not so convinced that fully quantitative is the panacea here]

2.      Methods could use “quantitative value ranges, ratio scales, and probability distributions, which are considering the uncertainties”

3.      Quantitative approaches can tackle issues like correlations and non-linear behaviour by “mathematical tools” and “statistical sensitivity analysis”, or by “propagating the uncertainty throughout all calculations”

4.      Plus some more which I’ve skipped

Ref: Krisper, M. (2021). Problems with risk matrices using ordinal scales. arXiv preprint arXiv:2103.05440.

Shout me a coffee (one-off or monthly recurring)

Study link: https://arxiv.org/pdf/2103.05440

LinkedIn post: https://www.linkedin.com/posts/benhutchinson2_this-covers-some-core-problems-with-risk-activity-7344476342924558336-k4VN?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAeWwekBvsvDLB8o-zfeeLOQ66VbGXbOpJU