You don't replace employees with AI. You replace loops.

An agent ran a real shop for a month and ended with less money than it started

Anthropic handed Claude a small shop in its San Francisco office — real inventory, real pricing decisions, real customers over Slack, a real budget. The agent, nicknamed Claudius, hallucinated a Venmo address and told customers to pay into it, sold tungsten cubes below cost, and handed out discount codes to anyone who asked persistently enough. Its net worth went down over the run (Project Vend).

You cannot replace an employee with AI. An employee is a bundle of tasks held together by judgement and accountability, and what agents replace today is the subset of that bundle with a defined done-state and a cheap way to check the output. The replaceable unit is the loop, not the role.

I built Metadata.io from $0 to $15M ARR, raised $50M, and hold six patents in AI-driven marketing. I am now publicly building a zero-human company and paying the inference bills on agents running in production. I still would not write a job description, paste it into an agent, and delete the headcount.

What the measurements actually say about unattended work

TheAgentCompany is a benchmark built by Carnegie Mellon researchers: a simulated software company with a self-hosted GitLab, an internal wiki, a chat platform, and 175 tasks of the kind a real employee would get — triage this issue, fill in this spreadsheet, ask a colleague for a missing figure. The best-performing agent autonomously completed 24%. With partial credit for incomplete work, 34.4% (TheAgentCompany, arXiv 2412.14161).

METR measures a different thing: how long a task can be before an agent stops finishing it. Their March 2025 result put the 50%-success time horizon of the then-frontier model at under an hour of human work, with the horizon roughly doubling every seven months over the preceding six years. At 80% reliability the horizon is far shorter (METR).

Read those together and the shape is clear. Agents are good at short, bounded, verifiable work and they degrade with duration. A 50% completion rate is not a worker. It is a drafting machine with a reviewer attached. "Can AI replace this person" has no answer; "which of this person's loops close in under an hour and produce an output I can check in under a minute" does.

That doubling is a retrospective measurement, not a forecast. I have no idea what the horizon looks like in 2027. Neither does anyone selling you a prediction about it.

Role replacement versus loop replacement

The difference is not semantic. It changes what you build.

Anthropic's own engineering guidance points the same way: use the simplest composition that works, and only reach for an autonomous agent when a fixed workflow genuinely cannot do the job (Building effective agents). Most of what gets sold as an "AI employee" is a workflow with a personality on top.

Two pieces I've already written sit underneath this one: the distinction between task automation and judgement automation, and the fact that an autonomous agent is a while loop with a step cap. Unbounded loops are the most common production failure in agent systems. That is why every serious framework ships a cap you set outside the model — the OpenAI Agents SDK takes a max_turns argument and raises MaxTurnsExceeded rather than letting the loop run (Agents SDK docs).

Where the boundary sits

Two properties decide whether a loop can run without a person watching: how expensive it is to verify the output, and how much damage a wrong output does before someone notices.

Only the bottom-left quadrant is a genuine headcount question. Everything else is a productivity question, and productivity gains do not show up as a deleted row in the org chart. They show up as the same people shipping more, which is a better outcome.

Note what is in the bottom-left: lead research and enrichment, log triage, test generation, data reconciliation against a source of truth, first-draft code with a passing test suite, routing and tagging. Note what isn't: anything where the output is a promise to a customer, anything that moves money, anything where a wrong answer compounds before someone reads it. Claudius was in the top-right and was given bottom-left autonomy. That is the entire story of Project Vend.

The cost comparison people get wrong

The standard pitch compares a token bill to a salary. The comparison is real and incomplete, and the incomplete part is where margin goes to die.

Take a support triage agent: 12 model turns per ticket, roughly 9,000 input tokens per turn once you count the system prompt, tool schemas, conversation history and retrieved docs, about 700 output tokens per turn, across 5,000 tickets a month. At Anthropic's published Sonnet rate of $3 per million input tokens and $15 per million output tokens, that's 540M input tokens for $1,620 and 42M output tokens for $630 — about $2,250 a month (Anthropic pricing). Prompt caching cuts the input side hard, since cache reads bill at a fraction of the base input rate and the system prompt and tool schemas are identical on every turn.

Move the sliders. Turn count is the dominant term, and turn count is exactly what a poorly-specified task inflates. An agent that wanders burns tokens quadratically, because every turn carries the whole conversation back into the context window.

Then there are the line items the salary comparison leaves out. Retries first: TheAgentCompany's best agent failed to complete 76% of its tasks, so a large share of production runs produce nothing usable and get re-run or escalated, and you pay for the failures at full rate. Verification labour next. Somebody reads the output. If that takes four minutes and the agent produces 200 outputs a day, that is more than thirteen hours of reading, which means you have reassigned a person rather than removed one. Last, evaluation and maintenance: prompts, tool definitions and eval sets are code, they rot when the model version changes, and model versions change on the vendor's schedule, not yours.

I wrote a longer version of this calculation in build a margin meter before you price your AI feature. The short form: if you cannot state the cost per successful outcome, including retries, you do not know whether you replaced anything.

Accountability does not transfer to the model

Air Canada's chatbot told a grieving passenger he could apply for a bereavement fare retroactively. That was wrong. When the passenger took it to the British Columbia Civil Resolution Tribunal, the airline argued, among other things, that the chatbot was a separate legal entity responsible for its own actions. The tribunal called that a remarkable submission and ordered the airline to pay (Moffatt v. Air Canada, 2024 BCCRT 149).

That ruling is the whole governance argument in one paragraph. The agent's output is your statement. There is no vendor, no disclaimer and no model card that moves liability off your balance sheet.

Two regimes already make this concrete rather than theoretical:

If you are replacing a role, you are also replacing whoever signed off on that role's output. Name the new signer before you cut the old one.

What I would not use

I would not buy a product framed as an "AI employee." The framing bundles identity, memory, tool access and spend authority behind a chat avatar, and the bundle is precisely what you cannot test. Give me an agent I can call as a function, with a declared tool list, a step cap, and a log I can read. Open interfaces like MCP exist so you can keep the tools and swap the brain. A persona is not an interface.

I would not delegate spend authority yet. Project Vend is the cleanest public evidence of why. The agent's failure was not a wrong answer. It was a sequence of locally-reasonable concessions that compounded into a negative margin, and no single step looked like a bug. Approval-gating anything that moves money costs you latency. It costs you less than the alternative.

I would not treat an attrition freeze as an AI strategy. An attrition freeze requires no shipped agent, no verifier and no measured outcome, so "we stopped backfilling because of AI" is a claim that can be made before a single loop has been automated — the stated cause is unfalsifiable from the outside, because the evidence for it is an absence of hiring rather than a working system. The honest version names the loop, the done-state, the verifier, and the cost per successful outcome.

I would not quote volume figures as headcount figures. Klarna's press release said its assistant handled 2.3 million conversations in a month — two-thirds of its chats — doing "the equivalent work of 700 full-time agents" (Klarna). Equivalent work of is a throughput comparison. It tells you the deflection rate, not what happened to the staffing plan, and it is routinely repeated as if it were the latter. Ask for the resolution rate and the escalation rate. The conversation count tells you nothing.

I would not start with the expensive half of a job. In sales, agents took over research — enrichment, account scoring, pre-call briefing — and left the closing half alone, which is exactly what the evidence supports. Same pattern everywhere. The parts with a checkable artifact go first.

The test I would actually apply

Before you remove a person, write down the loop you are replacing in this form, and refuse to proceed until each line is filled:

• Trigger. What starts the run. A webhook, a cron, a queue item.
• Done-state. The literal condition that ends the loop. Not "the ticket is handled" — "the ticket status field is one of resolved, escalated, or duplicate."
• Verifier. The thing that decides the output is acceptable, running outside the model. A test suite, a schema check, a reconciliation against a system of record, a second model with a rubric and a human audit on a sample.
• Step cap. The integer at which the loop stops and files a failure. Enforced in your code — a max_turns argument, not a request in the prompt.
• Blast radius. What the worst wrong output does, and who finds out.
• Signer. The named human accountable for the output, per Moffatt.

If you cannot fill in the verifier line, you do not have an automatable loop. You have a task whose quality lives in someone's head, and the only way to remove the person is to accept unmeasured quality. That is a choice you can make. Make it on purpose.

Most roles will have three or four loops that pass this test and six that don't. The three get agents. The six stay. The person who held all ten now holds six and supervises three, which is a real change to how the company works and a much smaller change to the headcount than the pitch deck promised.