Another healthcare study that explored what incident investigation approaches are used to generate recommendations, how recommendations are generated, what types of recommendations are generated and how the quality is assessed.
11 studies met inclusion criteria for review.
Providing background:
- Prior evaluation of healthcare investigations found that 80% of recommendations generated from investigations were ‘weak’, being unlikely to result in significant improvements in safety or risk reduction
- Another study based in Australia found only a small number of recommendations were rated as strong, and the most common types were reviewing or enhancing documents, and training and education
- Despite how widespread the use of incident investigations are, there is “a surprising lack of understanding about what actually happens in local health care settings with respect to this important activity”, nor is there much empirical focus “about recommendation generation by people conducting investigations”.
Results
Overall, this study found that:
- Approaches that went by “root cause analysis” monikers were the dominant investigation approaches, but there was marked differences in the approaches
2. As seen below, training and education was the most frequent category of actions. This was followed by a new procedure/memorandum, change of process of policy, and adjustment to policy or guideline.
3. 14% of all actions were too vague to be categorised.
Of the studies which provided data on the recommendation quality, there was a range of different indices. Some made judgements of actions via strength, terms like much better/better/same/worse, effectiveness, implementation, aimed at system level improvements, likelihood of preventing incident recurrence, quality, sustainability and more.
They found a paradoxical situation where “despite the ubiquity of recommendation generation, very little is known about it in practice”. Moreover, “although RCA dominates as the approach to investigation, there are no specific tools or approaches used to generate recommendations”.
They found that recommendations largely focused on staff knowledge and skills, rather than engineering and design, and workplace improvement.
There is also little agreement in the literature on how to assess the effectiveness of recommendations, meaning that “there is very little understanding of what makes a “good” recommendation”.
A number of challenges were levied towards investigation techniques. This includes: time constraints, lack of resources, and unwilling colleagues, investigations used to “support existing agendas rather than to generate new findings” (emphasis added), and investigations used to “manage scrutiny and maintain reputations”. On the latter, investigations failed to “appreciate the complex organizational agendas as well as social and political influences on recommendation generation”; factors likely to hamper improvements in safety.
Thus, they argue that this research “suggest that the generation of recommendations is likely to be a highly complex sociopolitical process with many stages and influences”.
As noted, <7% of all findings were rated as ‘strong’, or system-focused, like standardising equipment, architectural changes or simplifying processes. Hence, most investigation and recommendation generation efforts is tipped towards “weaker” recommendations that focus on improving individuals’ behavior and practice, rather than the wider system deficiencies that contribute to incidents”.
These findings are supported by other research in healthcare and across other sectors.
They also highlight how some investigation frameworks have been “identified as narrowing the view of causation or giving greater attention to causative factors relating to individuals”.
Few (none?) of the evaluated investigation frameworks also provided much guidance around how to assess or generate more effective improvements.
They also briefly discussed weaknesses in other safety approaches, like incident report systems, which are used to gauge effectiveness of investigation recommendations. Here they argued that “incident reporting systems … only detect a minority of incidents that actually occur, and this number may be even lower for incidents resulting in harm”.
Therefore “Incident recurrence may be a poor marker of investigation success, if reporting remains unreliable” and may not be a good proxy for gauging the effectiveness of investigations and actions.
Finally, they argue that “The studies included within this review provide no evidence that carrying out investigations and generating recommendations improve the quality or safety of care”.
Further, there seems to be “little consideration of the potential negative consequences of recommendations themselves”.
Authors: Lea, W., Lawton, R., Vincent, C., & O’Hara, J. (2023). Journal of Patient Safety, 10-1097.
LinkedIn post: https://www.linkedin.com/pulse/exploring-black-box-recommendation-generation-local-care-hutchinson-cg02e
SafetyInsights.org 
2 thoughts on “Exploring the “Black Box” of Recommendation Generation in Local Health Care Incident Investigations: A Scoping Review”