Applied AI & back-office operations
Agentic Operations Desk
An agent that works a back-office queue end to end — triage, lookup, and writes into systems of record — on a durable workflow engine, with scoped tools, idempotent actions and a human gate on anything that cannot be undone.
At a glance
- Type
- Reference architecture
- Domain
- Applied AI & back-office operations
- Architecture
- 6 layers
- First production cut
- 10–13 weeks, phases in sequence
A reference architecture. Figures on this page are design targets, not measured client results.
The challenge
Most back-office queues are worked by people doing the same eleven steps in six different tabs: read the ticket, find the customer, check the order, check the payment, update the CRM, reply. The work is too structured to stay manual and too consequential to hand to a chatbot — a wrong write lands in the general ledger, not in a transcript. The engineering problem is not making a model competent at the task; it is making its actions survivable when the model is wrong, the process dies mid-step, or the downstream API times out after it has already committed.
Failure mode
The obvious approach, and why it breaks.
Almost every team reaches the same first design, and it is not a bad instinct — it is the shortest route to something that demos. Here is that design, and the point at which real volume takes it apart.
Discarded
The obvious approach
A while-loop agent in a long-lived process. The model is handed broad tools — an HTTP client, a database connection, a generic “update record” function — and the loop runs until the model declares itself finished. Failures are handled with a try/catch and a retry.
Why it breaks
What production does to it
The process is the only record of where the run had got to, so a deploy, an OOM kill or a node drain halfway through loses the run — and nobody can say which of the eleven steps had already committed. Retry makes it worse: a write that timed out downstream almost always succeeded, so the retry issues the refund twice, sends the third identical email, reopens the ticket that was just closed. Broad tools remove the last guardrail — one confused step updates four hundred rows, and the only evidence is a log line saying the tool returned 200. At ten cases a day the operator quietly cleans up after it. At ten thousand, the cleanup is the new job.
Architecture
How the system is put together.
6 layers, from the edge where work arrives to the operations that keep it honest. Each one names the components it owns and why it exists as a separate concern.
Layer 01
Intake & triage
Triage is deliberately boring and mostly deterministic. The case taxonomy is closed: the agent may only work case types it has been evaluated on, and anything it cannot confidently place is escalated rather than improvised. Starting the workflow with the case id as its id makes double-start impossible at the engine level.
Components
- Channel adapters — Microsoft Graph mailboxes, Zendesk and Freshdesk webhooks, Slack, internal queue topics
- Deduplication on provider message id plus a content hash, so a redelivered webhook opens no second case
- Classifier into a fixed case taxonomy, with an explicit “unknown” class that routes to a human
- Entity resolution against the CRM before any reasoning — customer, order, account, contract
- Case record created in Postgres and a workflow started with the case id as the workflow id
Layer 02
Durable execution core
This layer exists so that “the box died” is a non-event. Workflow state is reconstructed from history on any worker in the fleet, so a case that is three steps in stays three steps in across a deploy. The cost is real: workflow code must be deterministic, which is an unfamiliar constraint for most application engineers.
Components
- Temporal workflows — one workflow per case, event-sourced, replayable after any worker loss
- Activities as the only place non-deterministic work happens: model calls, tool calls, clock reads
- Signals for human input — an approval, a rejection, a correction — without polling
- Timers and heartbeats for long waits: a vendor SLA, an overnight batch, a reviewer who has gone home
- Saga compensation registered as each side effect commits, not planned in advance
- Workflow versioning so runs already in flight finish on the code they started on
Layer 03
Reasoning & planning
The model plans; it does not remember. Durable state lives in the workflow, so a resumed case rebuilds its context deterministically rather than depending on whatever was in a conversation buffer. Escalation is treated as a correct outcome, because an agent that never escalates is an agent that guesses.
Components
- Claude with tool use as the planning step — propose the next action, not the whole plan up front
- Context assembled by code from the case record and retrieval, never accumulated in an open-ended scratchpad
- Structured output validated against Zod schemas; a validation failure is a retry with the error, then an escalation
- Hard caps per case — maximum steps, maximum tokens, maximum spend — enforced outside the model
- Grounding pass that requires every claimed fact to cite the tool result it came from
- A reflection step that is allowed to say “I cannot do this” and hand back, scored as a success
Layer 04
Tool & action surface
Tools are the security boundary, so they are written as verbs a reviewer can read — issueRefund, updateShippingAddress, closeTicketWithReason — not as an escape hatch like runQuery. Each write is designed to be called twice with the same result, because under any real network it eventually will be.
Components
- MCP tool servers, one per system of record, each with a narrow verb-level contract
- JSON Schema on every argument — enums for statuses, bounded numerics for amounts, no free-form object blobs
- Idempotency key on every write: a hash of case id, step name and canonicalised arguments
- Idempotency ledger in Postgres with a unique constraint, written before the call and settled after
- Read-back reconciliation for systems of record that do not honour idempotency headers
- Per-tool rate limits and circuit breakers, so a degraded vendor pauses one tool rather than the desk
Layer 05
Authorisation, policy & approval
Two questions are kept separate on purpose: may this action happen at all, which policy answers, and should this particular action happen now, which a human answers for anything irreversible. Irreversibility is a property of the tool, decided by engineers — it is never inferred from the model’s own confidence.
Components
- Per-tool credentials issued from Vault, scoped to one system and one set of operations, rotated on a schedule
- OAuth 2.0 token exchange (RFC 8693) so writes carry the requesting case’s authority, not a shared admin account
- Cedar policy decision on every tool call — who, which tool, which arguments, which tenant, what value
- Irreversibility classification per tool, set by engineers at design time and stored with the tool contract
- Approval gate in Slack and the web console: proposed action, arguments, the evidence behind it, approve or reject
- Four-eyes rule and per-tenant daily budget ceilings for high-value actions, failing closed when exceeded
Layer 06
Observability, audit & evaluation
The trace is written before the action executes, not after it succeeds, so a failed or half-finished action is as auditable as a clean one. When an auditor asks why a ledger entry moved eight months ago, the answer is a reconstructable record, not a recollection.
Components
- Append-only decision trace — prompt version, model, inputs, proposed action, policy verdict, approver, result
- OpenTelemetry spans stitched across workflow, model call and vendor API, correlated by case id
- Langfuse for model-level traces, prompt versioning and cost attribution per case type
- Deterministic replay of any historical case against a candidate prompt or model before it ships
- A regression suite built from real escalations and real mistakes, run in CI on every prompt change
- Alarms on the things that actually signal drift: escalation rate, approval rejection rate, cost per resolved case
Engineering decisions
Six decisions, and what each one costs.
A decision without a stated cost is marketing. These are the choices that shape the system, the reasoning behind them, and what is given up in exchange.
- 01
Run every case on a durable workflow engine (Temporal) rather than an in-process agent loop.
Why
Case state survives deploys, crashes and node drains, and a resumed case knows exactly which steps already committed. Human approval becomes a signal on a sleeping workflow instead of a process held open for six hours, and compensation has somewhere reliable to run.
What it costs
Workflow code must be deterministic, which is an unfamiliar discipline and a recurring source of subtle bugs; versioning long-running workflows is a permanent tax on every change. There is a cluster to operate or a Temporal Cloud bill to pay before the first ticket is resolved — real overhead that a queue of thirty cases a day does not justify.
- 02
Expose narrow, typed, verb-level tools instead of general-purpose HTTP or SQL access.
Why
A schema with enums and bounded numerics eliminates whole classes of failure before the model is even consulted, makes each action reviewable by someone who is not an engineer, and gives policy something precise to reason about.
What it costs
Every action needs a hand-written adapter, tests and an owner, so coverage grows slowly and unglamorously. The agent cannot improvise its way through a case that needs an action nobody anticipated — it escalates instead, which raises the escalation rate and is the correct behaviour even when it is frustrating.
- 03
Give every write an idempotency key derived from case id, step name and canonicalised arguments.
Why
Retries are unavoidable — timeouts routinely mask successful writes — and this is what makes them safe. The ledger row is written before the call, so a crash between call and response is resolved by reconciliation rather than by a duplicate refund.
What it costs
Argument canonicalisation must be exactly stable, and a field-ordering or rounding change silently becomes a new key. A genuinely intended repeat action needs an explicit nonce to get through. Systems of record without idempotency support require a read-back on every write, adding an API round trip and a window where the truth is briefly ambiguous.
- 04
Recover with saga compensation rather than attempting distributed transactions.
Why
Salesforce, NetSuite, a payment gateway and an internal ledger will never share a transaction. Registering a compensating action as each side effect commits gives a realistic, ordered unwind for the failures that can be unwound.
What it costs
Compensation is best-effort and sometimes semantically impossible — an email cannot be unsent, a payment can only be refunded, a credit note is not the inverse of an invoice. That forces a conservative irreversibility classification, so more steps sit behind approval and some failed cases park for a human instead of rolling back cleanly.
- 05
Scope authority per tool with short-lived, exchanged credentials rather than one service account.
Why
Blast radius is bounded by construction: the refund tool holds refund authority and nothing else, and the audit trail in each vendor system attributes the write to the case that caused it instead of to a shared robot user.
What it costs
Significantly more identity plumbing — a Vault path, a rotation schedule and a policy per tool — and several SaaS vendors still do not support token exchange, so those fall back to scoped API keys with the weaker guarantee that comes with them. Debugging an authorisation failure now spans the policy engine, the secret store and the vendor’s own permission model.
- 06
Run in shadow mode before going live, and keep hard step, token and spend caps that fail closed.
Why
Shadow mode produces the only trustworthy number in this whole system: measured agreement between the agent’s proposed action and what an experienced operator actually did, per case type. Caps bound the cost of a loop that has quietly lost the plot.
What it costs
Weeks of infrastructure cost and reviewer effort with zero automation benefit, and stakeholders who were promised an agent watching one that only takes notes. Caps also abort genuinely difficult cases mid-flight — the tail of hard work gets handed back to humans precisely because it is hard, which is not the outcome anyone hoped for.
Design targets
The operating point we build toward.
Targets, with the basis for each one written underneath it. They describe what this architecture is designed to hold — not a result measured on somebody else’s system.
- Duplicate side effects after retry or worker loss
- ZeroDuplicate side effects after retry or worker lossDesign target, verified by a fault-injection suite that kills workers mid-activity, duplicates webhook delivery and forces vendor timeouts — not an observed production figure.
- Actions reconstructable from the decision trace
- 100%Actions reconstructable from the decision traceDesign target — the trace row is written before the action executes, so failed and partial actions are as auditable as successful ones.
- Routine cases closed without human action
- 40–70%Routine cases closed without human actionTypical range for narrowly scoped back-office queues once shadow-mode agreement is measured. It depends almost entirely on how tight the case taxonomy is, and a wide queue lands well below it. Not a commitment.
- End-to-end handling of a three-tool case
- p95 under 90sEnd-to-end handling of a three-tool caseDesign target on current frontier-model latency and typical SaaS API response times; cases waiting on a human approval are excluded, since that clock measures reviewer availability rather than the system.
Stack
What this is built with
Named, current technology. Substitutions are normal — the shape of the system matters more than the vendor behind any one box.
- Temporal — durable workflow execution, signals, timers and replay
- TypeScript on Node.js 22
- Anthropic Claude with tool use, via the Messages API
- Model Context Protocol (MCP) tool servers, one per system of record
- Zod and JSON Schema — tool contracts and structured output validation
- PostgreSQL 17 — case store, idempotency ledger, append-only decision trace
- Redis — rate limiting, per-tenant budget counters, short-lived locks
- Cedar — authorisation decisions evaluated on every tool call
- HashiCorp Vault — per-tool credential issuance and rotation
- OAuth 2.0 Token Exchange (RFC 8693) for on-behalf-of scoping
- OpenTelemetry — spans across workflow, model and vendor boundaries
- Langfuse — model traces, prompt versioning and cost attribution
- Next.js — approval console and case timeline
- Slack Block Kit — inline approvals where reviewers already work
Timeline
To a first production cut.
Four phases, run in sequence. The first one is not engineering — it is deciding precisely what the system owes, because that is what every later phase is measured against.
2 weeks
Queue archaeology and action inventory
A few hundred real cases sampled and labelled, the six to ten case types that carry the volume, and a complete inventory of every write the agent could ever need. Each write is classified reversible, compensatable or irreversible, and that classification is signed off by the operations lead — it decides where the approval gates go. An evaluation set is cut from real escalations.
3–4 weeks
Durable spine and first tools
Temporal workflows, the idempotency ledger, the decision trace, and two read tools plus one write tool behind the policy engine. The write tool runs against a sandbox tenant while fault injection proves the retry path is genuinely idempotent. The desk is on live traffic but proposing only.
3–4 weeks
Shadow mode at volume
The full tool surface, the approval console, budget and step caps, and per-case-type agreement measured against what operators actually did. Disagreements are triaged into prompt, tool contract and taxonomy fixes, and the regression suite grows from every one of them.
2–3 weeks
Live on the narrowest case type
One case type executing for real, with approval gates on anything irreversible, a per-tool kill switch, an on-call runbook and a weekly review of every escalation. Expansion to the next case type is gated on stable agreement, not on the calendar.
Keep reading
Related blueprints.
Systems that share a spine with this one — the same evaluation, budget and rollback discipline, applied to a different problem.
Next step
Walk this architecture against your constraints.
A 45-minute session on the Agentic Operations Desk: which layers you already have, which ones you do not, and the decisions on this page that would go the other way for you.