This study explored the reliability and utility of risk matrices for ranking hazards relating to public leisure activities.
A driving factor for this study is previous research identifying “serious mathematical defects and inconsistencies” in risk matrices. Many of these issues aren’t just user-related but actually inherent and structural to the matrix itself.
The study was undertaken in two parts with international students. Students in the first part rated hazards on a 5×5 matrix, with attention being drawn to a specific hazard involving a fall from height. A subset of students at a later date were asked to review, individually, the risk ratings generated by different groups.
Way too many findings to discuss, so I can only provide a few points.
Results
Key findings (quoting page 2068):
- different risk assessors may assign vastly different ratings to the same hazard
- even following lengthy reflection and learning [risk rating] scatter remains high
- the underlying drivers of disparate ratings relate to fundamentally different worldviews, beliefs, and a panoply of psychosocial factors that are seldom explicitly acknowledged.
- risk matrices when used in this context may be creating no more than an artificial and even untrustworthy picture of the relative importance of hazards, which may be of little or no benefit to those trying to manage risk effectively and rationally
The authors then discuss reasons why such a large scatter of risk scores were present.
Matrix Technical Issues
This reason relates to primarily technical factors inherent to the matrix. The first issue is around assigning likelihood – often via ratings like high, medium or low. This simplification “glosses over” whether the rating refers to each individual exposure to the hazard or averaged over some defined period (like a year), or whether the rating is a collective exposure of all of those exposed during an interval.
Another issue is how matrices encourage users to multiply likelihood and consequences to calculate a “risk rating”. As others have pointed out, this is inappropriate because the axes of the matrix are ordinal (the order or ranking of numbers, e.g. 1st place, 2nd place) and not cardinal (counting numbers indicating the quantity).
Hence, summing these values has been said to be a “gross abuse,” which may disengage the brain.
A third issue is the qualitative nature of the scales. Defining the scales with enough accuracy is problematic and therefore will contribute to significant variation between raters. These data support that assertion, finding participants had significant trouble anchoring their valuations in “objectivity”.
It’s noted that perhaps the only way to better objectify the evaluations is to base them in actuarial data (based on accidents and the like). Such data is unlikely to be readily available to most people using matrices (perhaps outside of expert risk analysts, working groups etc.).
Information issues
Scatter of risk evaluations are also influenced by information on hazards and accident rates. Participants were forced to rely on their own subjective opinions to rate the hazard exposures.
In the second follow-up study, participants had some time to research and reflect on the hazards. While research and reflection did alter risk evaluations for 2 of the 3 hazards, scatter was still just as wide.
Psychosocial influences
A whole host of factors are likely at play in this category; only some are covered here.
One set of factors include individual and collective risk appetites and heuristics in identifying and evaluating risk. Things that concern people are influenced heavily by perception and affect.
For instance the riskiness of hazards is influenced by whether “exposure is voluntary, whether the hazard is new or old, whether it is in some way dreaded” (p2074; the latter is called dread risk and helps to explain the peculiar but normal fear people have for rare shark attacks or plane crashes but normalisation of driving/cycling risks and the like).
Further to psychometric factors is the “deeper realm of social and cultural influences” (p2074). One example is how ratings are influenced by affect (the underlying emotion or mood people experience to something). That is, as argued by Paul Slovic, how we *feel* about things influences and comes before how we *think* about something.
Therefore our underlying feelings of goodness or badness towards a hazard influences how we perceive it. If we see something as providing positive benefits (walking beside a cliffside could be technically dangerous but if the environment is beautiful and historical, we perceive the risk as lower because of how we feel about it).
Worldviews also influence ratings of hazards. I’ve skipped this section largely.
Mental processing
Another area influencing risk evaluations are cognitive and decision-making processes. They discuss the system 1 / system 2 modes of thinking. Again, I’ve skipped most of this as you can find books and hundreds of articles written about it (and the critical work pointing out issues with the dichotomy).
Some heuristics influencing evaluations are substitution, confirmation, availability and base rate neglect. An example is how people (including expert risk assessors) “frequently substitute a target question with a heuristic question …“How likely is an accident to happen on this medieval bridge?” becomes “Can I imagine a bad event happening on this bridge” (p2075).
Further they note “Associative memory builds on representations of the world that it considers “normal” to create rules about how things “should be.”. This results in abnormal events or things considered “bad” (affect) to create more visceral reactions and assessments of risk.
These mechanisms introduce confirmation bias where people look for things that align with their existing beliefs over looking for things that disconfirm or argue against their position.
Knowledge and Beliefs
This section looked at how the individual knowledge and beliefs of assessors informs their focus; leading to different risk trade-offs.
Some prioritise risk elimination or minimisation whereas others optimisation in balancing cost and reduction. Reconciling these disparate goals “is not straightforward”.
I’ve skipped heaps here but for interest the authors talk about fungibility in risk.
Third-party factors
Finally they discuss a host of other factors influencing risk assessments. For instance how the risk unfairly impacts particular groups like the young or elderly or how the duty holder was capable of readily addressing the issue.
Conclusion
In considering these findings and that of the study from Tony Cox Jr, the authors state that (pp2076-2077):
- “Risk matrices are limited in their ability to rank risks correctly and should not be used as they often are”
- “categorizing risk may require inherently subjective judgments that are prey to so many sources of inconsistency that there may even be “no objectively correct way to fill out a risk matrix.”
- The authors do not believe that “there is a reasonable prospect of “calibrating” or otherwise training risk assessors to increase their precision the benefits they associate with a place or an activity”
- Accuracy is also an issue and “there simply may be no right way to fill in these matrices” and given the host of factors influencing risk calculation scores, “there is simply no way of incorporating them in this two-dimensional schema”
- They argue against the idea that risk matrices are simple and transparent – stating this to be false
- “The risk matrix is not a simple tool as made out to be and, in fact, its apparent simplicity may even make it a dangerous tool that militates against deeper thinking” (emphasis added)
- “Perhaps it can be seen as an attempt to cram emotional-affective thought processes into an apparently rational thinking process and thereby present a multidimensional issue as a purely two-dimensional one, that is, one totally lacking in fungibility”
The Safety of Work podcast also covered this paper a while back – link below.
Authors: Ball, D. J., & Watt, J. (2013). Risk analysis, 33(11), 2068-2078.
Study link: https://doi.org/10.1111/risa.12057
Safety of Work podcast link: https://safetyofwork.com/episodes/ep8-do-risk-matrices-help-us-make-better-decisions-_LdvZqom
Link to the LinkedIn article: https://www.linkedin.com/pulse/further-thoughts-utility-risk-matrices-ben-hutchinson
SafetyInsights.org 