AI agents and robots are joining human teams, and most of the worry is about the machines. But the people in those teams can be the hazard too. The safety driver in Uber’s 2018 test vehicle was watching her phone despite a company ban. A Société Générale trader used his knowledge of the bank’s own controls to hide positions that cost it €4.9 billion. NASA engineers accepted known anomalies as normal until they caused the Challenger and Columbia disasters. None of these people were untrained. Training reduces honest error, but it does not remove misbehavior. Experts break rules for reasons that seem sensible at the time, teams drift toward the edge of safe operation, and automation can hide the drift.
Engineers have long designed for fallible, and sometimes deliberate, human actors. They use separation of duties, interlocks, graded authority, organizational design and just reporting cultures. These old lessons still apply when the team includes AI agents wand robots, if their assumptions about people are made explicit.
Why Read the Paper
Old Lessons for a New Team: Engineering Against Human Misbehavior in Human–AI–Robot Teams follows Old Lessons for a New Machine. It classifies human misbehavior, from honest error to deliberate harm, and asks of each kind whether a system must tolerate it, detect it or prevent it by making it impossible. A worked robot-cell example shows how production pressure and override authority create the shortest paths to harm.
Team decision theory adds a warning. A team that leans on its most reliable member is most exposed when that member fails unnoticed. In an illustrative three-checker team, the expert’s undetected defection lets 41% of unsafe actions go through, against 0.6% when everyone behaves. Robust design can limit that exposure, and the paper shows what it costs. It also covers AI teammates that agree when they should challenge, and why human teams cannot be relied on to adjust by themselves. It ends with recommendations, each with an acceptance test. It is written for engineers, program leaders and decision makers who build and run hybrid teams.
Why QSI
QSI builds tools for systems engineering, analysis and optimization that are used across the aerospace and defense industries. Our TEAMS software helps engineers model systems, analyze reliability and testability, and detect and isolate faults before they become failures. It also supports adaptive training, readiness assessment, modernization planning and lifecycle decision support. Designing systems that keep working when a part fails, whether that part is a sensor, an AI agent or a person, is what we do every day.
You can read the full essay here.
