Normal accidents, artificial life, and meaningful human control
“Lines are blurring between natural and artificial life, and we’re facing hard questions about maintaining meaningful human control (MHC) in an increasingly complex and risky environment.”
RISKY BUSINESS: Ever heard of a “normal accident?” Stanford University and University of Milan researchers found that complex systems with tightly coupled components suffer from major accidents if they run long enough. At a certain level of complexity, these accidents – which arise from multiple trivial causes – are unavoidable.
Complex and loose coupling: Universities are complex but have loose coupling; for example, they can replace professors or add new departments.
Complex and tight coupling: Modern AI systems and their underlying models are complex – common LLMs range from billions to trillions of parameters. But they haven’t been as coupled with other systems.
…until recently, when OpenAI began providing support for plugins in ChatGPT / GPT-4.
What can we do: Require frequent check-ins with human operators, such as “Would you like me to perform operation X?” Breaking up the AI’s workflow into smaller, permissioned pieces – in effect, decoupling them– helps better manage the risk and enhance the possibility of meaningful human control.
VISUALIZING ACCIDENTS: AI accidents can be hard to visualize. Imagine instead that we are building a nuclear plant - would we accept one that works 80% of the time? That means it will fail one in every five instances! The research team writes that it depends on what happens when it fails. After Three Mile Island, Chernobyl, and Fukushima, error tolerances and public discourse changed. Global public opinion shifted against nuclear energy, and governments delayed, heavily regulated, or banned it entirely.
Black Swans: The key to good risk management, writes risk analyst Nassim Nicholas Taleb and his colleagues, is preparing for extreme outcomes - not averages, typically known as Black Swan events.
“Extremes are rare by definition,” they write, “and when they manifest themselves, it is often too late to intervene. Sufficient – and solid – evidence, in particular for risk management purposes, is already available in the tail properties themselves. An existential risk needs to be killed in the egg, when it is still cheap to do so.”
MEANINGFUL HUMAN CONTROL: Like hospitals in triage, organizations are constantly making decisions under time pressure and limited resources, a surefire recipe for errors, inadvertent and otherwise. When working with AI, meaningful human control (MHC) is the bare minimum standard. According to NATO researchers, MHC requires “humans to be able to make informed choices in sufficient time” to influence or prevent AI impacts.
Tricky elements: informed choice and sufficient time are both critical but tricky.
Informed choices: Humans and AIs need to have accurate mental models of each other so that humans can recognize AIs’ shortcomings and seize control as necessary.
Thinking at regular speed: Humans cannot operate at machine speed. Instead, we must rely on machine-produced explanations for their behavior via confidence estimates or feature attributions.
Designing human-machine hybrids: When designing combined human-AI teams, leaders should know who does what, when tasks will be reallocated, and how team members will communicate.
Gut check: Leaders can tell whether their teams meet MHC by seeing if their behavior complies with ethical guidelines. They can also ask if teams feel in control.
BE NICE TO ARTIFICIAL LIFE: Researchers studying artificial life say agents’ complexity “brings them so close to living beings that they can be cataloged as artificial creatures” (emphasis theirs). They designed an experimental world populated with insect-like digital creatures to explore 1) how humans would behave towards them, 2) why they would behave that way, and 3) whether this behavior would alter the artificial creatures’ evolution.
High fidelity: The experimental world was designed to be interactive, adaptable, and somewhat realistic – avoiding the uncanny valley while allowing humans to feed or kill creatures. Over time, the creatures evolved: they got better at acquiring food and living longer.
Human preferences: The study results showed that no humans planned to kill the creatures, but a few (<0.3) did anyway due to accident, curiosity, or dislike.
Bottom line: artificial worlds and creatures can evolve from external influences – but there is a lot to learn about how humans can make an impact and why we decide to be cruel or kind.
WHAT’S NEXT: Emergent threat models require new tools to map and neutralize them. Studying complex systems, human-AI team dynamics, and artificial life can provide insights that improve our mental models of AIs, interaction systems, and ourselves. These emerging fields challenge us to consider our decisions' second- and third-order effects and flash a yellow danger sign when we should pull back and demand more control over our tools.
GO DEEPER: Here are more resources to understand how complexity influences risk – and how collective behavior can teach us about working with AIs.
Perrow, C. (1999). Normal accidents: Living with high risk technologies. Princeton university press.
Taleb, N. N., Bar-Yam, Y., & Cirillo, P. (2022). On single point forecasts for fat-tailed variables. International Journal of Forecasting, 38(2), 413-422.
Ashby, W. R. (1960). Design For A Brain: The origin of adaptive behavior+. Chapman & Hall Ltd, New York, second edition.
Hasbach, J. D. and Witte, T. E. (2021). Human-Machine Intelligence: Frigates are Intelligent Organisms. In 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC).
Hollnagel, E. and Woods, D. D. (2005). Joint Cognitive Systems. Foundations of Cognitive Systems Engineering. CRC Press, Boca Raton, FL.
