Running a company with AI agents: what actually holds

The unit of delegation is a task, not a job title

On GAIA, a benchmark of 466 real-world assistant questions, human respondents scored 92% and GPT-4 with plugins scored 15% (Mialon et al., 2023). That gap is the design problem in one line, and it has not closed evenly across task types.

Running a company with AI agents means cutting the company into tasks that have defined inputs, a checkable output and a bounded blast radius, giving those tasks to agents, and keeping approval, spend authority and legal liability attached to a named human.

That is the whole answer. Everything below is about where the cut goes.

Measure task length, not task difficulty. METR's time-horizon work measures the length of task a model completes with 50% success and found that length has been roughly doubling every seven months across the models they tested (METR, 19 March 2025). Two things follow. Short tasks are where the reliability is. And a workflow that needs eleven correct steps in a row is not one task — it is eleven, and you should be checking each seam.

I built Metadata.io from $0 to $15M ARR with humans doing that work, and before that took Silver Spotfire from $50K to $1.5M ARR. The org chart was the delegation unit then. It is the wrong unit now. "Account executive" is not a task. "Draft the follow-up email using last call's transcript and the three open objections" is.

What agents run, and where the human still signs

The split I would draw today, function by function. The right column is not a maturity stage to graduate out of — it is where the signature stays.

Two of these get delegated badly. Content gets shipped unsigned, and finance ops gets standing spend authority because the approval step was annoying. I would not delegate payment execution to an agent today, at any dollar limit, because the failure is not a bad payment — it is a loop that makes the same bad payment forty times before anyone reads the alert.

The sales side of this deserves its own treatment, and I wrote it task by task in Will sales be replaced by AI? and AI won't replace salespeople.

Where the controls live

The single most load-bearing architectural decision is this: the model proposes, and something outside the model decides whether the call executes. Not a system prompt that says "always ask before spending money." A step counter, an allowlist and a dry-run mode in the harness, where the model cannot talk its way past them.

Both major agent SDKs are built around this separation. The OpenAI Agents SDK ships guardrails, handoffs and tracing as framework-level features rather than prompt instructions (OpenAI Agents SDK docs). Anthropic's own engineering guidance is blunter: start with the simplest composable pattern, and add agentic autonomy only when a workflow cannot do the job (Building effective agents).

The dashed path is what most first drafts ship. Model output goes straight to a production tool, the retry loop has no ceiling, and nothing is written to a log you can replay. The concrete build for the solid path — cap, allowlist, dry run, in that order — is in Build an AI agent loop with a step cap, allowlist and dry run.

Tool wiring is solved. The Model Context Protocol is an open standard for exposing tools and data to models, released by Anthropic in November 2024 (announcement, spec). Use it for connection. Do not mistake it for a permission system — it moves the wire, not the authority.

The three failures you will actually hit

None of these are exotic.

Unbounded loops. An agent retries a failing call, re-reads the same page, or ping-pongs between two tools until something stops it. Nothing in the model stops it. The fix is a hard step cap and a wall-clock timeout enforced by the harness, plus a cost ceiling per run that kills the process rather than warning about it. This is why the retry arrow in the diagram sits before the gate.

The lethal trifecta. An agent that has access to private data, exposure to untrusted content, and the ability to communicate externally can be instructed by the untrusted content to exfiltrate the private data. Simon Willison named the pattern in June 2025; prompt injection and excessive agency are both on the OWASP Top 10 for LLM Applications. There is no prompt that fixes it. You break one leg of the three: either the agent reading untrusted web content has no secrets in context, or it cannot send anything outbound without a human in the approval queue.

Silent degradation. A tool returns a 200 with an empty body, an upstream schema adds a column, a scraper now hits a consent wall. The agent keeps producing confident output from nothing. This one is expensive because it does not page anyone. The detection is boring: assert on output shape, alert on distribution shifts in the agent's own outputs (length, refusal rate, tool-call mix), and sample runs for human review on a fixed cadence rather than when you feel like it.

Memory is where two of these compound, because a bad fact written once gets read for months. The two distinct senses of agent memory, and which one you actually need, are in AI agent memory.

What it costs, in the only terms that matter

Per-seat thinking does not transfer. The bill is tokens times turns times runs, and the multiplier that surprises people is turns — every tool call re-sends the accumulated context.

Take a support triage agent: 12 turns per ticket, roughly 15,000 input tokens per turn as the conversation and retrieved docs accumulate, 1,200 output tokens, 3,000 tickets a month. Move the model and the runs and watch what happens to the monthly figure. Current per-token prices are on the OpenAI pricing page.

Two levers cut that bill without touching the workflow, and neither one changes a line of the agent's logic. The first is prompt caching: Anthropic bills cache writes at 1.25x the base input price and cache reads at 0.1x (prompt caching docs). Your agent re-sends a stable system prompt, tool schema and document set on every single turn. It does. That gap between 1.25x once and 0.1x thereafter is the difference between viable and not. The second is batching — anything that does not need an answer now goes through the OpenAI Batch API at a 50% discount with results inside 24 hours. Nightly enrichment, backfills, eval runs.

Order the work by which of those two applies. Then meter it, because an agent you cannot bill for is an agent you cannot size — the argument for shipping one metered endpoint first is in Start a business with AI.

What I would not use

A single agent with thirty tools and an autonomy prompt. This is the most common architecture in demos and the least common one in production, for a reason. Anthropic's guidance is to reach for the simplest composable pattern and add autonomy only where a fixed workflow genuinely cannot do the job (Building effective agents). A router plus four narrow agents with six tools each is easier to eval, easier to cap, and the failure is localised.

Standing spend authority. Not "up to $500." Not "up to $50." The reason is not trust in the model's judgement on any single payment; it is that loops multiply and a per-transaction limit does not bound a per-hour total.

GUI automation where an API exists. A computer-use agent clicking through a browser is the fallback, not the default: it is slower, it breaks when a button moves, and it is far harder to log in a reviewable way. Reasons in full in A computer use agent drives a GUI with pixels, not APIs.

Public benchmark scores as a readiness signal. SWE-bench Verified is 500 human-validated samples drawn from GitHub issues (OpenAI, August 2024). It is a good benchmark. It contains none of your tickets, none of your schema, and none of your customers' phrasing. A model that tops it can still be wrong in a way that costs you money on your third most common support request.

The replacement is unglamorous and it is the actual work: pull real past cases with known-correct outcomes from your own systems, freeze them as an offline set, and re-run the set on every prompt change, tool change and model upgrade. If a model bump improves your eval, ship it. If nobody can tell, you did not have an eval.

What "zero-human" actually means

I am building a zero-human company in public, and the name misleads people, so here is the literal version: no humans doing the work inside the loop. Humans still own the loop. Someone defines the task, writes the eval set, reads the failures and signs the payments. That person is accountable for what the agents did whether or not they watched it happen.

NIST's AI Risk Management Framework organises this under Govern, Map, Measure and Manage (NIST AI RMF). The framework is voluntary and it is not exciting. The part worth stealing is the insistence that accountability lands on a named person, not on a system.

I hold six patents in AI-driven marketing and raised $50M for the last company. Neither fact tells you whether an agent will hold up in your stack on a Tuesday. What tells you is whether you can answer this: for any action an agent took in the last 90 days, can you produce the inputs, the tool calls, the model version, the cost and the human who approved it?

If you can, you have a company that runs on agents. If you cannot, you have a company that runs on hope and a large API bill, and the first time a customer asks why an agent told them something wrong, you will be reading Slack instead of a log.