ORBIT: keeping an AI system good after it goes live

The operations delta — five disciplines for running AI workflows in production

Design gets you to day one. Operations is every day after.

In the previous article we introduced BEACON — the six questions that make an agentic AI system production-ready. This article is about what happens next, because a well-designed AI system is not a finished one. It is a system that now has to survive contact with time: model updates, drifting data, changing policies, growing volume, and the slow erosion of human attention.

Here is the uncomfortable fact that makes AI operations its own discipline: an AI workflow can be technically healthy while being operationally wrong. The API responds. Latency is within target. Infrastructure is green. And meanwhile the AI is producing weaker answers, reasoning from stale knowledge, drifting toward the edge of its permissions, costing materially more per case than last quarter — or receiving "human review" that has quietly become a reflex click. Nothing in a standard operations toolkit is designed to catch any of that, because standard operations watches the machine, and these are failures of the work.

Five properties of AI systems create this gap — the same three that drove the design delta, plus two that only show up in production:

1. Probabilistic output — uptime and error-rate monitoring cannot tell you whether the content is still right.
2. Consumption-based economics — cost moves with input size, retries, routing and volume; a workflow can become uneconomic without a single feature change.
3. Evolutionary dependencies — providers change model versions, pricing, and behaviour on their own schedule; a model migration is an operational event, not a vendor detail.
4. Knowledge freshness — many workflows reason over policies, documents and data that change independently of the code; stale grounding produces confident, wrong output.
5. Autonomy and oversight erosion — agents act, not just answer; and human review, left undesigned, decays into rubber-stamping precisely because the system is usually right.

ORBIT is the operations delta: five disciplines added on top of — not instead of — the incident, change and service management you already run. Its unit of operations is the workflow, not the model: one workflow may contain extraction, retrieval, an agent, deterministic checks and a human approval, and the only question that matters is whether the whole thing is still producing the intended business result inside agreed quality, risk and cost boundaries.

One management layer sits above the five: a simple health state per workflow — green (operate normally), amber (sustained degradation or a control concern: investigate, constrain, set a dated corrective action), red (material risk: suspend or fall back, run Triage, resume only after validated remediation). One rule about it: do not automate the colour. A dashboard may propose the state; a named workflow owner decides it. The health state is a management judgement informed by evidence, not a metric.

To keep this concrete we'll revisit Northline, the retailer from the BEACON article, six months after its refund-request agent went live.

O — Observe: what is actually happening?

Why. You cannot manage what you only feel. Most teams running AI in production can tell you their uptime and their token bill, and nothing else. The drift that kills AI systems — quality, cost, behaviour — is invisible unless you decide, in advance, what to watch.

What. An ORBIT Scorecard: eight to twelve workflow-level measures, on one page, covering five things — the business outcome (is the workflow delivering the result it exists for), quality (evaluation scores, sampled correctness), operations (volume, timeliness, ask-rate), economics (cost per decision and its trend), and control (review response times, boundary events, refusal rate). Underneath it, enough instrumentation that any single case can be traced end to end: what came in, what was retrieved, what was proposed, what was decided, by whom, at what cost.

How. Pick the eight to twelve numbers with the workflow owner — resisting the urge to track forty. Set bands, not false precision: "ask-rate between 25% and 40%" is an operable threshold; "ask-rate = 31.7%" is a screenshot. Wire alerts to band exits, not to noise. Review the scorecard on a rhythm — weekly is right for most workflows.

Northline. The refund workflow's scorecard: refunds resolved per day, golden-set score, sampled-correctness rate, ask-rate, average review minutes, refusal rate, cost per decision, complaint rate on refund decisions, and time-to-review. Nine numbers, one page, read every Monday in ten minutes. When the ask-rate crept from 30% to 38% over three weeks — no alert from any technical system — the scorecard made it a conversation before it became a queue.

R — Review: is it still acceptable?

Why. Observation tells you what is happening; review tells you whether it is still good. The two are different because AI quality cannot be read off a graph — it has to be checked against known-correct answers and sampled from real production work, by people qualified to judge.

What. An Evaluation Pack: the workflow's golden set (graded cases with known-correct outcomes), a stored baseline score, written acceptance criteria, and a method for sampling live production outputs each week. Plus honest review design: human review is a control, not a ceremonial signature — reviewers need the evidence in front of them, the time to judge, and their outcomes recorded. If a model is used to help judge outputs at scale, it assists the review; it never replaces the human baseline, and its judgements are themselves spot-checked.

How. Run the evaluation on a schedule and — always — before and after any change: prompt, model, policy, knowledge source. Sample a handful of real production cases each week and have a qualified person grade them against the same standard. Hold a 30-minute weekly review: scorecard, evaluation results, sampled cases, review-time stats, and one decision — is the health state still green? Write down what was decided.

Northline. Thirty golden cases re-run weekly; five real refund decisions sampled and graded by a senior agent each Friday. The week the sampled cases showed two "correct but curt" customer messages, the evaluation caught a tone regression the golden set's pass/fail grading had missed — a new grading note was added, and the fix shipped through change control. Total ceremony: half an hour.

B — Bound: is it still inside its authority?

Why. Boundaries are designed once but hold — or erode — every day. Permissions accumulate, caps get "temporarily" raised, a new tool gets connected, review clocks quietly stretch. In an agentic system, boundary erosion is risk accumulation, and no one notices because nothing breaks.

What. The Operating Envelope, maintained as a living document: what data the workflow may access, what actions it may take — with every action carrying its act / ask / stop authority — who reviews what, on what clock, what the audit trail must capture, and, crucially, who holds suspension authority: the named person entitled to take the workflow out of service without a meeting. The envelope is enforced in configuration, so changing any part of it is a recorded, attributable act.

How. Verify the envelope on a rhythm, the way finance verifies access rights: are the caps in configuration still the caps on the signed page? Have any permissions widened? Are review clocks being met? Count and read the boundary events — every time the system hit a cap, refused a case, or was overridden. A rising override rate is either a boundary drawn wrong or a boundary being worked around; both need a decision, not a shrug.

Northline. Quarterly envelope check: the ₹5,000 act-cap and ₹25,000 stop-line verified against config; one finding — a well-meaning engineer had added a delivery-tracking tool the envelope never listed. Useful tool, wrong process: it went through change control retroactively, and the gap in the process that allowed it was closed. Suspension authority: the head of customer operations, by name, documented, and tested once — see Triage.

I — Improve: better, cheaper, more autonomous — from evidence

Why. An AI workflow that is merely maintained decays; one that is operated well compounds. The review discipline produces a stream of evidence — recurring exceptions, reviewer decisions, cost patterns, near-misses — and Improve is the discipline of turning that stream into a managed backlog instead of folklore.

What. An improvement loop with one bright line down the middle: the routine-vs-material rule. Routine refinements — a clarified prompt, a new golden case, a threshold tuned inside its band, a knowledge document updated — flow through lightweight change control inside operations. Material changes — a new model, a widened autonomy band, a new action, a new customer-facing behaviour — go back through design and validation: the design record gets updated and the evaluation gate is passed before production. Improve also owns the economics (cost-per-decision trend and the routing of steps to cheaper models as they prove out) and the knowledge operations (who updates the documents the AI reasons over, on what schedule).

How. Keep one backlog, evidence-tagged: every item cites the scorecard line, sampled case, or reviewer pattern that motivated it. Ship routine items weekly. Batch material items and treat each as a mini design pass. And treat provider model changes as scheduled operational events: pin versions, test migrations against the Evaluation Pack, migrate on your calendar, not the provider's.

Northline. The reviewers' decisions had accumulated: the same "delivery scan missing but courier portal shows delivered" pattern, decided the same way, forty times. Improve turned it into a deterministic rule — through the material-change path, because it widened autonomy. Result: ask-rate down six points, review queue down a fifth, and the two senior agents now spend their review time on cases that genuinely need judgement. That is what compounding looks like: the system earned more autonomy with evidence, not enthusiasm.

T — Triage: when it goes wrong anyway

Why. It will. And an AI quality incident rarely looks like an outage — nothing is down, nothing crashed. It looks like a reviewer's unease, a customer complaint, an evaluation failure, an odd cluster of refusals. Teams without a playbook debate whether it "counts" as an incident while it spreads.

What. A seven-step TRIAGE sequence: Detect (capture the evidence immediately, whatever the source), Assess (one case, a class of cases, one version, one knowledge source, or all traffic?), Contain (suspend automation, fall back to humans, disable a tool, pin or roll back a model — the envelope's suspension authority acts here), Correct (smallest safe fix, through change control), Validate (targeted incident cases plus the full Evaluation Pack — never resume on a spot check), Resume (the named authority approves return to service, staged if needed), Learn (post-incident review; the incident's cases join the golden set so this exact failure can never again happen silently). Severity is a simple overlay on your existing incident model — critical for external or irreversible impact, major for recurring recoverable failure, minor for contained deviation — and the whole thing plugs into the ITSM process you already run.

How. Write the playbook before the first incident, one page. Rehearse containment once — actually suspend the workflow and watch the fallback carry the load — the way you'd test a generator. Log even the incidents that turn out to be nothing: the pattern of near-misses is Observe's best early-warning data.

Northline. The provider shipped a model update; the weekly golden set caught a scoring drop in delivery-date reading before customers did. Detect: Monday's evaluation run. Assess: one case class — date parsing on courier scans. Contain: those cases dropped from act to ask automatically; the suspension authority chose staged containment over full stop. Correct: parsing step adjusted, one golden case added. Validate: full pack passed. Resume: approved Wednesday. Learn: migration testing moved earlier in the calendar. Customers saw nothing. The whole incident lives in the record as evidence that the controls work — which, at Northline's next audit, turned out to be worth more than the incident cost.

The minimum viable ORBIT

You do not need a platform to start. You need five artifacts and a rhythm: the Scorecard (8–12 measures), the Evaluation Pack (golden set, baseline, acceptance criteria, sampling method), the Operating Envelope (data, actions, review, audit, suspension authority), the Operating Rhythm (weekly review, monthly economics pass, quarterly envelope check, event-triggered evaluation — each with a named owner), and the Incident & Change Playbook (TRIAGE plus the routine-vs-material rule). Most teams can stand this up in two to three weeks for their first workflow, mostly from things the design phase already produced.

And one principle over all of it, worth pinning above the desk: if a control cannot be run at its agreed cadence, by a named person, using available evidence — it is not a control. It is documentation.

If you already have AI in production, try this today: three questions per workflow. Who is the named owner of its health state? When was it last scored against cases with known-correct answers? Who has the authority to suspend it — and have they ever practised? Three names-and-dates questions, no technology required. Teams that can answer them run AI. Teams that cannot are being run by it — they just haven't found out yet.

ORBIT is the operations discipline we use at AI for Growth; it picks up exactly where BEACON's design work leaves off, and the two share their instruments — the golden set built in design becomes the Evaluation Pack run in operations, and the boundary page becomes the living envelope. Design well, operate deliberately, and an AI system does something rare in enterprise software: it gets better with age.

By Sai Krupa Sagar. To see the complete ORBIT discipline — or to put it to work on an AI system you already run — reach us at aiforgrowth.io/contact.