Your AI Incident Playbook Should Disable Tools Before It Rewrites Prompts

The bridge declared the chat disabled. Five minutes later, the pricing API still received writes from an agent workflow that was already in progress. The frontend was dark, but the tool credentials were alive. That is the difference between an application incident and an agent incident.

An automotive pricing assistant began looping during a dealer promotion outage. It read stale inventory context, proposed a price adjustment, retried a slow pricing API, and posted conflicting updates for a handful of vehicles. The incident bridge disabled the chat entry point, but a background workflow continued because the tool registry still allowed writes. The actual containment happened only after the team froze the pricing tool, pinned the retrieval index version, stopped queued executions, and moved affected workflows to manual review.

Teams reuse application runbooks for agents. They stop traffic at the UI, redeploy a prompt, and assume the system is safe. That misses the layers that make agents dangerous: queued tool calls, cached context, delegated credentials, memory refresh jobs, and autonomous retries.

An agent incident playbook should be ordered by blast radius. Disable or degrade tools first. Freeze context refresh when stale or poisoned evidence is suspected. Pin or roll back prompt and model versions only after side effects are contained. Preserve traces before cleanup because tool-call history is often the only way to reconstruct impact. Communication ownership also matters: support needs to know which actions were blocked, which were completed, and which need manual correction.

For agent incident response, the release review should inspect the workflow before it inspects the model. Every production use case needs a task boundary, identity model, allowed tool list, context source registry, policy version, trace format, and rollback path. The architecture should say which actions are read-only, which create drafts, which require approval, and which are blocked entirely.

Agent incident response needs evidence, not confidence. Show a sample tool-call trace with user identity, tenant, policy decision, input redaction, output payload, cost, latency, and final action. Show how an incident commander disables one tool class without disabling the whole assistant. Show how a prompt, model, retrieval index, and tool schema change move through regression gates.

For agent incident response, the hard question is not whether the agent can complete the demo. The hard question is whether the system can explain what happened after a wrong answer, stale context, duplicate tool call, or permission denial.

User chat disabled
      |
      v
Queued workflow still running --> Tool broker --> Pricing API writes

Better containment:
Incident commander
  +--> disable write tools
  +--> pause queued workflows
  +--> freeze retrieval index
  +--> pin prompt/model route
  +--> preserve traces for audit

The honest tradeoff is not speed versus safety in the abstract. It is which actions deserve autonomy, which actions deserve draft mode, and which actions should never be delegated. The team should add control where the action changes customer data, money, access, or regulated records, then keep low-risk retrieval and drafting lightweight enough to keep learning.

Agent incident response tests should include empty retrieval, wrong-tenant retrieval, prompt injection through retrieved documents, stale index versions, duplicate tool retries, partial tool completion, malformed tool output, permission denial, long-session memory, and cost spikes. The expected result is not always a better answer. Sometimes the expected result is refusal, escalation, draft-only mode, or tool disablement.

For agent incident response, a useful artifact shows allowed actions by task, not only prompts by intent. Include the user role, autonomy tier, eligible context sources, tool list, approval rule, trace fields, kill switch, and rollback owner. That record is more valuable than a prompt library because it says what the system may do when the answer becomes an action.

A full shutdown is simple and sometimes necessary, but it also removes harmless read-only workflows. A tiered playbook is better: keep read-only status lookup alive, disable high-blast-radius tools, route ambiguous tasks to humans, and freeze memory or retrieval only for the affected domain. For regulated workflows, the playbook should also include evidence retention before any cleanup job runs.

Queued execution deserves its own line in the playbook. An agent may have already planned a tool call before the incident commander disables the chat surface. The runbook should say whether queued work is cancelled, paused for approval, or allowed to finish in read-only mode. It should also record the queue cursor, workflow id, tool id, and last committed side effect. Without that detail, the team can stop new traffic while old work continues changing state.

A weak agent incident response review asks whether the agent answered correctly in a demo. A useful review asks whether the system can prove why it answered, what it was allowed to touch, and how the team can stop it safely. The reviewer should ask for a golden workflow set with expected tool traces, not only expected final text. A case should specify which tools may be called, which sources are eligible, what refusal looks like, what approval state is required, and what audit fields must be written.

Agent incident response review should include rollout mechanics. Prompt changes, model route changes, retrieval index rebuilds, and tool schema changes should move through separate versioned gates because they fail differently. A model upgrade can change reasoning. A retrieval rebuild can change evidence. A tool schema change can change side effects. Treating all of those as one release type is how regressions hide.

For agent incident response, cost and latency should be first-class signals. An agent that takes eight tool calls to resolve a low-value task may be correct and still not production-worthy. Track cost per completed workflow, timeout rate, approval queue age, refusal quality, and human override rate. Those numbers tell leadership whether the agent is becoming operational software or a permanent demo with nicer logs.

Leadership should watch agent incident response signals: tool calls by tier, approval queue age, refusal quality, policy-block rate, cost per completed workflow, and the time it takes to disable one tool without disabling the whole assistant. Those numbers reveal whether the agent is becoming software or staying a guided demo.

The artifact worth keeping for agent incident response is a workflow control record. It should show the user role, allowed tools, autonomy tier, context sources, retrieval filters, approval state, trace retention rule, kill switch, and rollback owner. A prompt alone is not an artifact because it cannot prove authorization or side effects.

For agent incident response, include one sample trace from a real-shaped task. The trace should show source versions, tool calls, policy decisions, latency, cost, and final disposition. If the team cannot produce that trace, it is not ready to scale autonomy.

A practical agent incident response review should include one real-shaped workflow trace. The trace should show identity, tenant, prompt version, retrieval index version, selected sources, tool inputs, policy decision, approval state, cost, latency, and final disposition. If the trace cannot explain a wrong answer or a blocked action, the eval suite is not yet a release gate.

Agent incidents should be handled as workflow and tool incidents, not content incidents. If the first move is prompt editing while tools remain active, the response is theatre.