The first hour of an AI incident, in order
A checklist for the moment an agent or model-backed app has done something it should not have, written for whoever is in the room.
By Priya Ramaswamy-Quill, Editor, Actually Useful4 min read
Most incident response guidance assumes a human attacker and a compromised machine. This is for the other case, which is now common: an agent sent email it was not asked to send, a support bot pasted a customer's credentials into a chat, a coding assistant merged something, a model-backed workflow has been quietly exporting records to an address nobody recognizes. There may be no attacker in your network at all. There is a component with your permissions that took direction from someone else, and the first hour is going to be handled by whoever noticed.
The order matters. Under stress, people do the satisfying thing first, which here is usually turning the agent off and deleting the conversation. Turning it off is correct. Deleting the conversation destroys the only record of what it was told.
Minute 0 to 5: Stop the hands, keep the memory
- Revoke the agent's credentials first: its API keys, OAuth grants, service-account tokens, and any delegated user sessions it holds. Not "pause the workflow." Revoke, so a queued action cannot fire and a second injection cannot run. If the agent acts as a user, revoke that user's sessions too.
- Do not delete the conversation, the thread, the ticket, or the document it was reading. Do not reset the retrieval index or clear the queue. These are the evidence.
- Note the time, what you saw, and what you did, in a text file.
Minute 5 to 15: Get the right people
- Tell one person with authority, directly. Not a group chat.
- Open a dedicated channel for the incident, outside any system the agent had access to. If the agent could read your chat tool, it is not the place to discuss the incident, and neither is the inbox it could forward from.
- Assign one person to keep the timeline. Everyone else reports to them.
- Bring in whoever owns the agent's integration. They know what it was connected to; the security team usually does not.
Minute 15 to 30: Preserve the trail
AI incidents leave a different kind of evidence, and much of it has short retention.
- Prompt logs. The system prompt in effect, the user turns, and every piece of retrieved content the model saw: the email, the page, the document, the search result. Export them now; many platforms keep them for days, not weeks.
- Tool-call and trace logs. Every function the model called, with arguments and results, in order. This is the equivalent of a shell history. If your platform does not record it, reconstruct what you can from the receiving systems: sent-mail logs, API gateway logs, audit trails on the ticketing system.
- Model and configuration state. Which model version, which tools were enabled, what permissions the connectors held. Capture the configuration before anyone "fixes" it.
- Snapshot rather than rebuild. Copy anything you are about to change.
Minute 30 to 60: Scope, find the input, decide who to tell
- Scope by reach, not by action. The question is not "what did the agent do?" but "what could it reach?" List every data source and every tool it was connected to, with the permission level. That is your working blast radius, because you do not yet know what it read, and reading is not always logged. Assume anything it could reach, it may have sent out.
- Find the injected input. Work backwards from the first unexpected action in the trace to the content the model had just read. The instruction is usually there: a calendar description, a hidden paragraph in a shared document, a web page, a code comment, a ticket. Preserve it, note who could have placed it, and search for the same text elsewhere, because an attacker who planted one copy planted several.
- Check for persistence. Injected instructions can live in memory features, saved notes, generated documents, and retrieval indexes. Anything the model stored during the incident may re-inject it when the agent is turned back on.
- Decide who to tell. If customer or personal data was within reach, loop in counsel now; notification clocks in many jurisdictions start at discovery. If the agent sent messages externally, the recipients and the people it impersonated need to hear from you before they hear from the attacker. If the platform belongs to a vendor, open a ticket and ask for their logs before their retention window closes.
Things not to do in the first hour
- Do not ask the agent what happened. Its account of events is model output and is subject to the same injection you are investigating.
- Do not turn it back on with "better instructions." Instructions are not a control.
- Do not announce the cause externally. You do not yet know it.
- Do not blame the person who connected it. They were handed a setup guide that said to.
Before the incident
This is easier when a few things exist already: a kill switch that revokes an agent's credentials in one place, tool-call logging the agent cannot edit, an inventory of what each agent can reach, retention long enough to cover the weeks between an injection and its discovery, and a human approval step in front of anything outbound or irreversible. If you have none of these, this article is your plan. Print it, and keep it somewhere the agent cannot read.
About the writer
Editor, Actually Useful
Priya runs the section of the magazine that is allowed to be helpful. A former incident responder, she writes the checklists she wishes someone had handed her at 3 a.m.