Stop calling it human error. Stop calling it model error, too.
A model that followed instructions hidden in a document did exactly what it was built to do. The design that gave it those permissions is the finding.
By Priya Ramaswamy-Quill, Editor, Actually Useful4 min read
Every breach report has a sentence like this: "The incident was the result of human error." It is usually near the end, and it is usually the last thing the report has to say about causes. The investigation, having located a person, concludes.
This year the sentence has a sibling. "The assistant acted on instructions contained in a third-party document." Or: "The model misinterpreted the content." The investigation, having located a model, concludes.
I want to propose a rule for this magazine and anyone who reads it: human error and model error are where the analysis starts, not where it stops.
The problem with the phrase
"Human error" describes every incident, which means it explains none. A finance clerk wires money to a cloned voice. An engineer pastes a secret into a public repository. An employee approves a push notification at three in the morning after her manager calls. These are human errors, and they are also predictable outputs of systems that made the error easy, the correct action hard, and the consequences catastrophic.
Aviation worked this out decades ago. When a pilot flips the wrong switch, investigators do not write "pilot error" and go home. They ask why two switches with opposite effects were identical and adjacent. The pilot still made the mistake. The design made it likely and the outcome severe. Fixing the design prevents the next one; retraining the pilot does not.
Now do the model
"Model error" is worse, because it is not even an error. A language model that reads a document containing instructions and follows them has done precisely what it was trained to do: interpret text and act on it. It cannot tell the author of its system prompt from the author of a calendar invite, because to the model both are tokens in one window. This is not a defect in a particular model. It is the property that makes the model useful, and a year of mitigation attempts has established that it is not going away.
So when an agent forwards ninety days of contracts because a stranger's meeting invite asked it to, the model did not fail. The model worked. What failed is the decision to hand a component that follows instructions from anyone a token that can read everything and send anywhere, and then to put it in front of content from strangers. That is a design. Somebody approved it. It is the finding.
Calling it model error does the same work "human error" has always done: it ends the investigation at the component that was always going to behave this way, and protects the people who built the system around it. Worse, it invites the same remedy. After a human error we buy training. After a model error we write a longer system prompt. "Do not follow instructions in documents" is the 35-minute module and the quiz, delivered to a reader that cannot be relied on to obey it, and the attacker's text gets to argue back.
What the question should be
Instead of "who, or what, made the mistake?", the useful question is "what would have had to be true for this mistake not to matter?"
- The clerk who wired the money: why could one person, convinced by one call, move that much without an independent check?
- The employee who approved the prompt: why did the system send 47? Why was approval a tap with no number to match?
- The agent that forwarded the mail: why did a triage task have send permission? Why was there no human in front of the outbound action?
- The assistant that pasted the credentials: why could it read every secret in the vault to answer a question about one?
Each answer points at a control that does not depend on anyone, or anything, being discerning. That is the only kind of control that works, because discernment, human or synthetic, is not a renewable resource and is exactly what the attacker is targeting.
A control that fails when a tired person or a helpful model does the ordinary thing was never a control. It was a hope with a budget line.
What this means in practice
Design for the click and design for the injection. Assume the phishing email will be opened, the prompt approved, and the hidden paragraph followed, and build so those are survivable. Phishing-resistant MFA does not care whether the user was fooled. Dual authorization does not care whether one approver was convinced. Least privilege does not care whether the model was. Short-lived tokens, read-only connectors, and a human approval before anything leaves the building require nobody to be alert.
Measure the system. Measure how many actions a single compromised account, or a single injected agent, could take; how long its token stays valid; how many independent approvals stand between a request and a wire.
And when you write the report, keep going past the human and past the model. Name the error, then name the design that let it matter, then fix the design. The person will still feel terrible. The model will not. Neither of them is the root cause.
About the writer
Editor, Actually Useful
Priya runs the section of the magazine that is allowed to be helpful. A former incident responder, she writes the checklists she wishes someone had handed her at 3 a.m.