This explored the use (and limitations) of a risk matrix for assessing risks related to a public health issue when the frequency and severity of risks are negatively correlated.
They constructed a risk assessment matrix using tainted blood transfusion infection risk, which had a negative correlation between frequency and consequence. They used experiential data to compare with estimations from the risk matrix.
As outlined by other research (Cox Jr), serious limitations with risk matrices include (p9577):
- Poor resolution
- Ambiguous inputs and outputs
- Sub-optimal allocation of resources based on inaccurate risk estimation
- Errors in assigning higher rankings to quantitatively lower risks
On the last point, Cox noted that the use of a risk matrix involving an issue with inversely correlated frequency and consequence could be “worse than useless” and worse than a random guess. This was tested in the paper.
Results
I’ve skipped over all of the argumentation and data discussing the application of the risk matrix. However, Cox’s proposition was confirmed where the risk assessment matrix “provides a risk categorization (color code) that is incorrect in relation to an empirical calculation of the risk” (p9584).
Here, it’s said that “Given the wide-spread and apparently growing popularity of risk matrices for risk assessment purposes, the prospect of obtaining results that are worse than random is clearly a serious concern” (p9584, emphasis added).
In discussing the findings, the author draws on other research, where it’s said that “When used to assess a set of hazards with a negative correlation between frequency and consequence, risk matrices are often uninformative and occasionally misleading” (p9585).
Other researchers conclude that risk matrices are limited in their ability to rank risks correctly and “should not be used as they often are, that is, as proxies for risk management decisions by the simple device of overlaying them with colors associated with risk management priorities” (p9585). One reason is because optimally allocating resources goes far beyond a function of simple two dimensions of likelihood and consequence, “upon which the matrix rests” (p9585).
The limitations of using risk matrices, such in the case of this public health risk scenario with inversely correlated frequency & consequence, highlights that decision makers need to carefully reconsider the use of the risk matrix.
Importantly, it’s stated that “There is no straightforward solution to address the concerns raised about risk assessment matrices”, nor do they suggest an alternative for mapping risks that lack prior knowledge on harm frequency and severity. They don’t, however, suggesting a complete abandonment of risk matrices per se, but note that if people choose to use them then they must be aware of the limitations and have a healthy scepticism.
This may be problematic if, they explain, “risk matrix users see this technique as a simple tool and ignore the embedded complexity involved” (p9586). Moreover, the “primary danger” revealed from this analysis is that over-simplified risk analyses tend to become the risk management decision itself rather than something purely as a guide.
Link in comments.
Authors: Shabnam Vatanpour, Steve E. Hrudey, Irina Dinu, 2015, Int. J. Environ. Res. Public Health
Study link: https://doi.org/10.3390/ijerph120809575
Link to the LinkedIn article: https://www.linkedin.com/posts/benhutchinson2_this-explored-the-use-and-limitations-of-activity-6841466986267049984-HcUf
SafetyInsights.org 