SafetyInsights.org

This study explored different risk matrix designs on user comprehension and use of matrices.

I’ve posted a lot of research dunking on risk matrices (RMs), so here’s another olive leaf focusing on designing something potentially more useful.

[** I still stand by the critical orientation, since we have little good evidence that RMs improve risk management or decision-making, and carry some substantial limitations and biasing effects…but, whatever.]

The design changes were a geometric transformed matrix in a log configuration, a square matrix, a text-only control and the ‘standard’ matrix – see below.

I’ve skipped HEAPS, so check out the full paper if this interests you.

Background:

·         “There are extensive theoretical critiques of risk matrices”, most aimed at mathematical or logical issues with quantitative RMs

·         In comparison, there’s less research around the “merits and demerits” of qualitative and semi-qualitative matrices

·         They note that “There is some qualitative experimental evidence that people may struggle to use semiqualitative risk matrices to make rational decisions”, but otherwise there is little evidence on the perception, use and comprehension of RMs

·         The use of logarithmically increasing RM categories is said to be common and sometimes recommended, where log scales “facilitate comparison between risks that span several orders of magnitude, and can aid interpretation of data with logarithmic/exponential trends”

·         Nevertheless, a lot of people, including experts, struggle with comprehending log scales compared to linear scales

·         They also discuss other matters of RM design, colour, shape etc., which I’ve skipped (check out my site though, as I’ve covered several papers exploring these facets)

·         One researcher recommended using untransformed values as scale labels, but the authors note that “the probabilities in risk matrices are sometimes so small that completely untransformed values may be difficult to interpret”, e.g. negative exponents like 10-4 on RM axes

·         An alternative might be more familiar numbers that increase non-linearly, like 1, 5, 25, 125, 625, as this might communicate difference in scale/size better than 1, 2, 3, 4, 5

·         Back to log RMs, they argue that the increasing size of the cells as one moves to the top right may work as a simple visual cue that conveys magnitude and priming effects

·         They cover a range of other data on RM design and use, again I’ve skipped a lot; one example is the preference people have for reducing impact over likelihood

Results

Key insights were:

·         “risk matrices are not always superior to text for the presentation of risk information”

·         “there are changes to the standard format of qualitative and semiqualitative risk matrices (rectangular cells, linear scale labeling, use of a key) that may help them to communicate risk more effectively”

·         “a nonlinear/geometric labelling scheme helps matrix comprehension (when the likelihood/impact scales are nonlinear)”

·         “To a lesser degree, results suggested that changing the shape of the matrix so that cells increase in size nonlinearly facilitates comprehension as compared to text alone, and that comprehension might be enhanced by integrating further details about the likelihood and impact onto the axes of the matrix rather than putting them in a separate key”

·         The “geometric scale labeling (1, 5, 25, 125, 625) substantially improved participants’ ability to answer risk comparison questions, producing the largest effect size”

·         “the primary recommendation for an improved risk presentation format is the use of ordinal, explicitly nonlinear scale labels for matrices with an exponential or otherwise nonlinear increase in likelihood and/or impact along the axes”

·         Hence, overall, there may be a possible, but small, benefit of a log format with increasing spacing between lines, but this effect may not be present for people already well-familiar using matrices

They also found there to be a small dip in performance for log RMs compared to standard, but this dip was more than compensated for in other ways.

Text alone generally underperformed against one of the matrices in all test conditions, but there may be a benefit to combining them in real-world examples (rather than experimental conditions).

For why log performed better, they suggest that, perhaps, “the “logarithmic” cell design works with (rather than against) cognitive heuristics such as magnitude priming”. And while these effects may bias human judgements, the biases can result in more “rational” judgements. Hence, the log format provides a visual cue about magnitude and the non-linear nature of the data.

Below highlights the best performing combination of changes:

Finally, they briefly discuss the “ethical use of risk matrices”, highlighting:

·         RMs omit many important factors relevant in risk and decisions and “there are ethical problems with relying upon them in an overly narrow fashion”

·         And, all decisions “necessarily have positive and negative externalities that are not included in the measure of “impact” used by the risk assessment, which must also be taken into account”

·         The use of numbers may also be misleading, where numbers can be misconstrued with reality

·         And finally, “It is crucial that risk matrices not be used as a replacement for critical thinking, but rather as one tool in a toolbox for ethical decision making”

Ref: Sutherland, H., Recchia, G., Dryhurst, S., & Freeman, A. L. (2022). How people understand risk matrices, and how matrix design can improve their use: Findings from randomized controlled studies. Risk Analysis, 42(5), 1023-1041.

Study link: https://onlinelibrary.wiley.com/doi/pdf/10.1111/risa.13822

My site with more reviews: https://safety177496371.wordpress.com

LinkedIn post: https://www.linkedin.com/pulse/how-people-understand-risk-matrices-matrix-design-can-ben-hutchinson-vleuc