Control Tower Operations

It’s a Tuesday afternoon and forty agents are acting across your production systems right now — opening tickets, moving records, calling internal APIs, posting to channels, touching the billing pipeline. Ask the obvious question — what is agent #23 doing this minute, and is it behaving? — and you can’t answer it. There’s no console. You’ll learn that one went sideways the way you always do: from the refund it shouldn’t have issued, the table it shouldn’t have written, the customer who replies “why did your bot tell me that.” You find out from the damage, not from a dashboard. That blind spot is the entire problem this paper is about, and closing it is what a control tower does.

This is written for the person who gets paged when an agent misfires — the AI platform SRE, the fleet operator, the ops lead who owns “are the agents okay.” It is not a posture memo for executives and it is not about how agents delegate to each other. It is the operating discipline: what to watch on a live fleet, which intervention controls to build, and how to run an incident when one agent goes bad at 2 a.m.

The runtime-visibility gap

Most teams instrument the build of their agents and almost nothing about the run. You have eval suites, prompt versions, and a CI gate that passed before the agent shipped. Then it ships, and observability collapses to scattered logs, a few traces, and a Slack channel where humans relay “something seems off.” That’s fine for one agent a person can babysit. It falls apart at fleet scale for a specific reason: the unit you need to see is no longer a request or a trace — it’s an action. An agent reading a document is one thing; an agent writing to the orders table, issuing a credit, deleting a record, or messaging a customer is an action with consequences, and the fleet emits thousands of them an hour.

A control tower is the surface where those actions become visible, attributable, and stoppable in flight. It is not your APM dashboard with agent icons on it — the difference is the verb. APM shows latency and error rates; a tower can intervene: pause an agent, require approval before the next action commits, roll a botched action back, or kill the agent outright. Visibility without that intervention layer is just a faster way to watch the swarm misbehave.

This can’t live inside any single agent, for the same reason a fleet is hard: no agent knows what the others are doing. Agent #23 doesn’t know agent #11 already refunded the same account two minutes ago, or that the three of them hammering one downstream API are about to trip its rate limit and cascade. Each view is correct and partial. The tower is the only place every agent’s actions converge into one picture — which is why a tower bolted into one agent’s runtime is a contradiction. It sits above the fleet and reads from all of it.

What a control tower watches

If a tower is going to answer “what is every agent doing right now, and is any of it wrong,” it has to ingest four classes of signal — not generic metrics, but the things that tell you an agent is acting and whether it’s acting correctly.

The agent fleetN agents acting across prodeach one blind to the others — partial, correct, uncoordinated

Control toweraction stream · queues · anomaly checks · attributionone live picture of the whole fleet, with the controls to act on it

Operator consolehold · approve · rollback · killwhere a human stays in command of the fleet

The tower's job is horizontal: it stitches every agent's action stream into a single operational picture, then hands the operator the controls to intervene on any of it in flight.

The four signal classes:

Live actions — every consequential thing an agent does, emitted as a structured event the instant it’s attempted: which agent, which action-type, which target system, with what arguments, under which prompt and tool version. This is the heartbeat. If an action isn’t an event in the tower, it didn’t happen as far as governance is concerned.
Queues and backpressure — how many actions are pending approval, how deep the human-review queue is, which agents are blocked waiting on a gate, and where work is piling up. A backlog isn’t a metric to glance at; it’s a leading indicator that a gate is mis-tuned or an agent is stuck in a loop.
Anomalies and drift — an agent whose action mix suddenly shifts (it was 90% reads, now it’s writing constantly), whose error rate spikes, whose outputs diverge from its eval baseline, or that’s burning tokens far above its norm. Drift is the thing you most want to catch before the damage, and it only shows up against a per-agent behavioral baseline the tower maintains.
Attribution — every action traceable to one agent identity, one prompt/tool/model version, and one originating task. When you ask “who did this and on whose behalf,” the tower answers in one hop, not a forensic dig across logs.

That last one makes the other three governable. Attribution is the difference between “an agent issued a bad refund” and “agent billing-resolver, v3.4.1, on task T-88150, issued refund R-44192 at 15:18 — here’s the exact tool call.” The first is a shrug; the second is a containable incident.

Intervention controls: hold, approve, rollback, kill

Watching is half a tower. The half that earns the name is the set of controls an operator can pull on a live agent. Build exactly four, and be precise about what each one guards.

01Holdpause an agent or action-class mid-flight; pending work freezes, nothing commits
02Approvea queued high-blast-radius action waits for a human yes/no before it executes
03Rollbackreverse a committed action via its recorded inverse; only as good as the compensating step you wrote
04Killterminate the agent now; revoke its credentials and tokens so it can't act again

Four controls, escalating in force. Hold and approve prevent damage; rollback and kill contain it. The dashed gate marks where approval intercepts an action between intent and commit.

The hard one is rollback — a design obligation, not a button you get for free. An action is only rollback-safe if the agent emitted a recorded inverse when it acted: the credit memo that reverses the refund, the soft-delete that restores the record, the retraction that follows the message. Actions with no clean inverse — an email already sent, a payment already cleared, a webhook already fired — can’t be rolled back, only compensated, and the tower should know which is which and tell the operator. The discipline is upstream: when you give an agent a tool that mutates the world, build its inverse at the same time, or classify the action as approval-required so a human catches it before it commits. Kill is the blunt instrument — it stops the agent and revokes its credentials, so a wedged or compromised agent can’t keep acting after you’ve cut it off. An agent you’ve “stopped” whose API token still works hasn’t been killed; it’s been asked nicely.

Policy gates at runtime: auto-act vs. require-approval

You cannot put a human in front of every agent action — at fleet volume that’s the bottleneck the whole project exists to avoid. And you cannot let every action auto-execute — that’s the swarm. The resolution is a runtime policy gate that decides, per action-type, whether an action auto-commits or stops for approval. The decision is keyed to blast radius: how bad is it if this action is wrong, and how reversible is it?

Action classAuto-actRequire approvalBlock Read / query / summarizeauto Reversible write (draft, tag, internal note)auto Customer-facing messagegate Money movement / refund / creditgate Destructive / bulk / schema changeblock

Gate by action-type and blast radius, not by agent. A read auto-commits; a refund waits for a human; a bulk delete is blocked outright until someone lifts the rule. The same agent can sit at all three rows depending on what it's trying to do.

The point that’s easy to miss: autonomy is granted to an action-class, not to an agent. The same billing-resolver reads freely, drafts freely, and is held at the gate the moment it reaches for a refund — because a wrong read costs nothing and a wrong refund costs money and a chargeback. You tune each gate to the cost of being wrong in that direction: holding a legitimate refund for ninety seconds of human review is cheap; auto-issuing a fraudulent one is not. That asymmetry, not the model’s confidence score, is what sets the gate.

The gate itself lives as versioned, executable policy in the tower — not as a convention in a senior operator’s head:

RULE FleetActionGate
WHEN action.class == "read" OR action.is_reversible
THEN decision "AutoAct"

WHEN action.class in ["customer_message","money_movement"]
THEN decision "RequireApproval" WITH decision_packet

WHEN action.class in ["bulk_delete","schema_change","credential_grant"]
THEN decision "Block" UNTIL operator_override

WHEN agent.anomaly_score > threshold OR agent.action_rate > 3x baseline
THEN decision "Hold" AND alert "DriftSuspected"

That last rule catches the drifting agent automatically — a behavioral guardrail that holds an agent the moment its runtime profile diverges from its own baseline, no human required to notice first. Because gate evaluation runs hot — every action, every agent, continuously, over data that’s often sensitive — it runs local, next to the systems the agents touch. You don’t pay a per-call API tax to evaluate a gate on every action, and you don’t stream the fleet’s entire action history outside your boundary. External models are reserved for the genuinely ambiguous calls that reach a human.

Running an incident on the fleet: detect, contain, replay, learn

When an agent goes bad, the fleet doesn’t give you a stack trace and a stopped process — it gives you a stream of consequences already in motion. The incident loop has to be fast and it has to be muscle memory.

01Detectdrift alert, anomaly, or a human report names the suspect agent
02Containhold or kill the agent; revoke its tokens; stop the bleeding before you diagnose
03Assesspull the agent's action log; find what committed; roll back what's reversible
04Replay & learnreconstruct the decision path; turn the root cause into a new gate or eval

Contain before you diagnose — the inverted reflex of code incidents. A misbehaving agent keeps acting while you investigate, so you cut its ability to act first, then read the record.

The software-incident instinct — understand it, then fix it — is backwards for a fleet, and retraining it is the most important habit here. A crashed service does no more harm while you read the logs; a misbehaving agent is still issuing refunds while you investigate. So the order is contain first: hold or kill the suspect, revoke its credentials, and only then open the action log to assess what committed and what can be reversed. The faster you cut an agent’s ability to act, the smaller the blast radius — which is why kill and credential revocation have to be one motion, tested before you need it, not improvised at 2 a.m.

The replay record and the post-incident review

The reason every consequential action is emitted as a structured event is that the event stream is the audit trail. After containment, you don’t reconstruct what happened from fragments — you replay it. Each action is one record carrying everything the review needs: the agent and version, the action and its target, the gate decision, and the correlation that ties it to the task that spawned it.

{
  "eventType": "AgentActionAttempted",
  "actionId": "act-2026-0224-44192",
  "agentId": "billing-resolver",
  "agentVersion": "3.4.1",
  "actionClass": "money_movement",
  "target": "billing.refunds",
  "args": { "account": "acct-77310", "amount_usd": 240.00 },
  "gateDecision": "RequireApproval",
  "approvedBy": null,
  "anomalyScore": 0.82,
  "reversible": true,
  "inverseAction": "issue_credit_memo",
  "taskId": "T-88150",
  "correlationId": "2f17f6c2-1a3c-4d9f-a7b1-1142f81c4d01",
  "attemptedAt": "2026-02-24T15:18:12Z"
}

Five properties make that record a replay rather than a guess, and each is load-bearing: correlation IDs threading every action back to its originating task, versioned prompt/tool/model stamps so you can reproduce the exact agent that misbehaved, idempotent replay so re-reading the log never re-fires an action, the recorded inverse on anything reversible, and an operator-facing reason on every gate decision. Note anomalyScore: 0.82 and approvedBy: null on a money-movement action — that single record tells the reviewer the agent was already drifting and the action was caught at the gate before it committed. That’s a post-mortem that takes minutes instead of a day, and it’s only possible because the record was built to be replayed, not just logged.

Ownership keeps the loop from rotting back into a passive dashboard. One lightweight RACI on the fleet, not a binder:

Step	Tower (auto)	Fleet Operator	Ops / SRE Lead	Tower Owner
Detect drift / anomaly	R	I	I	A
Contain (hold / kill / revoke)	C	R	A	I
Assess & rollback	R (auto-inverse)	R	C	I
Replay & post-incident review	R (record)	C	A	C
Tune gates / drift thresholds	—	C	C	R/A

The last row is the one teams skip and then pay for. Someone owns the gate thresholds and the drift baselines as a standing job — because the failure that quietly kills a tower isn’t a missed incident, it’s a gate so noisy that operators start rubber-stamping approvals, at which point the tower is back to watching the swarm with extra steps.

Standing up a minimal control tower: the runbook

You do not build the whole tower before it earns its keep. You instrument one action-class on one fleet and close one loop.

Time to detect a bad agenthours / next shiftminutesContainment / risk
Actions auto-acted vs. gatedall gated / all open60–80% autoOperator capacity
Reversible high-risk actionsfew / unknown100% w/ recorded inverseRollback / safety
Actions with full attributionpartial logs~100% replayableAudit / learning

Sample targets for the first quarter — set the numbers against your own fleet size and action mix. Attribution and reversibility are the two that make the other two possible: you can't auto-act safely on what you can't trace or undo.

First 90 days — instrument and gate one fleetEmit the AgentActionAttempted event from every consequential tool call. Stand up the live action stream, per-agent baselines, and the hold + kill controls with credential revocation. Gate the two highest-blast-radius action-classes to require-approval, and rehearse a kill once so the muscle memory exists before the real incident.
Months 4–12 — earn auto-act and wire rollbackBuild recorded inverses for reversible high-risk actions and turn on rollback. Move proven, low-blast-radius classes from gated to auto-act — each promotion earned by its incident-free record, never granted up front. Switch on automatic drift-holds against the baselines. Run the detect→contain→replay loop on every real incident.
Ongoing — govern the fleetDaily: drift alerts and the approval-queue depth. Weekly: gate-threshold tuning and false-hold review. Every incident: a replay-driven post-mortem that ends in a new gate, a new inverse, or a new eval — so the tower gets harder to fool each time.

The unit of work is one action-class, fully governed — instrumented, gated, reversible, replayable — not “build the tower.” The first class carries the cost of the event contract, the action stream, and the four controls; every class after it reuses all of that and pays only for its own gate rule and its own inverse. That’s the compounding: the second loop is a fraction of the first, and every incident you replay sharpens a gate that makes the next agent safer to run unattended. You aren’t buying a wall of agent status tiles — you’re building the console that lets a handful of humans stay in command of a fleet already too big to watch by hand.

What we covered

The takeaways

The unit is the action, not the agent. A tower ingests every consequential action as a structured, attributed event — that stream is your only real-time picture of a fleet too big to babysit.
Build four controls: hold, approve, rollback, kill. Watching is half a tower; intervention is the half that keeps you in command — and rollback only works if you recorded the inverse when the agent acted.
Gate by action-class and blast radius. Reads auto-commit, refunds wait for a human, bulk deletes are blocked — the same agent sits at every gate depending on what it reaches for.
Catch drift automatically. Hold any agent whose runtime behavior diverges from its own baseline, before the damage — not after a human reports it.
Contain before you diagnose. A misbehaving agent keeps acting while you investigate; cut its ability to act first, then replay the record to assess, roll back, and turn the root cause into a new gate.

When a fleet of agents is acting across your systems, command is not a quarterly review or a tidy org chart — it is whether, at 2 a.m., one operator can see which agent went sideways, stop it inside a minute, undo what it did, and prove the next morning exactly what happened and why. A control tower is what makes that answer yes.

References: NIST AI RMF · NIST CSF 2.0 · OpenTelemetry semantic conventions · ITIL event & incident management · OASIS / EDA event-driven patterns.