What happens when a high-authority automated system operates with an incorrect model of reality?
This question is explored via Dekker & Woods’ (2024) article on automated systems that are Strong, Silent, and Wrong.
Check it out on my YouTube – link in comments.
We explore the breakdown of human-automation coordination, mode confusion, and automation surprises, using the Boeing 737 MAX MCAS accidents as examples.
Discussed is how complex system design must prioritise observability, redirectability, and graceful failure.
This also opens up the question about when automation gets it wrong, can people recognise the situation and take back control?
Automated systems can act correctly according to its internal model, but dangerously in the real world.
#safetyscience #aviation #automation #complexity #humanfactors #systemsafety
