Every AI demo has the same shape: a clean prompt, a confident answer, applause. Every AI product has a different shape: messy inputs, long-running work, partial failures, and someone who is paged when it goes wrong.
Prototypes have become a commodity. The gap between a prototype and a product, the harness, the verification and the operations, is turning into the core domain of AI engineering. Three recent pieces show the same shift from different angles.
Stripe tested agents on real integration work
Most coding benchmarks use isolated problems. Stripe's team, in Can AI agents build real Stripe integrations? (March 2026), deliberately tested something closer to a real job: building and changing integrations in full codebases.
Their benchmark has 11 environments in three groups: backend-only tasks such as data migrations and SDK upgrades, full-stack tasks that also need browser interaction, and smaller "gym" exercises on specific features like Checkout and subscriptions. Each environment has a realistic repository, a database, test API keys and deterministic graders. Grading is end to end, because, as the authors put it, payments require 100% accuracy.
The results are encouraging and instructive. The best model averaged 92% across four full-stack tasks, which shows agents can do serious multi-step engineering. But the failures are the useful part:
- Agents struggled with ambiguous situations, for example accepting an API error as if it were success.
- They got stuck in the browser, misreading what a page showed them.
- They lost focus partway through filling in forms.
Notice that none of these are "the model cannot write code". They are failures of judgment under ambiguity and of operating tools reliably, the long-horizon part of engineering that includes planning, persistent state and recovering from failure. (The scores are not a leaderboard: different models were run on different task sets.)
Agents are now both producers and consumers of software
Vercel's Agentic Infrastructure post reports that coding agents now initiate more than half of deployments on its platform, as of September 2026. Three months before its original publication the figure was over 30%. It also reports that projects deployed by agents are 20 times more likely to call AI inference providers than those deployed by humans.
This is company-specific telemetry, so don't generalize the exact numbers. The design implication still holds: infrastructure has to assume that the thing shipping the software and the thing calling it can both be agents. Vercel argues that platforms need to evolve in three directions: deployment surfaces built for agents, platforms for running agents themselves, and systems that can respond to operational problems without a human watching.
The harness is becoming a product
OpenAI's Agents API, which entered public beta in September 2026, is the clearest sign. As described at launch, it is a managed runtime for long-running agents: it handles context (including automatically compacting earlier context as a session nears its limit), tools, subagents and the execution environment, while the platform runs the agent loop.
The message is that model capability alone does not make a long-running agent work. Something has to manage context, tools, state and recovery. For years teams built that layer themselves. Now vendors are selling it, which tells you how much of the work lives there.
What "the harness" has to cover
Put the three pieces together and the checklist for crossing from demo to product is concrete:
- Context management: what the agent knows at each step, and what happens when the window fills up.
- Tool reliability: timeouts, retries, and clear handling of errors that look like successes.
- State and recovery: can a run resume after failure, and are repeated actions safe (idempotent)?
- Verification: deterministic checks on the outcome, not just on the output. See Evals are product analytics.
- Observability: traces that let you replay what the agent did.
- Guardrails and security: least-privilege tool access and defence against prompt injection. See Securing LLMs in production.
- Cost and latency budgets: limits per task, because loops can run away.
- A human owner: someone accountable for the result.
If you are connecting agents to real systems, native MCP integration covers the tool side in more detail.
A quick self-test
Before calling an agent feature "production ready", ask:
- What is the worst thing it can do with the access I gave it?
- How would I know, within an hour, that it silently started failing?
- If it stops halfway, what state is the world in?
- What does a passing run look like, in a check that is not another LLM's opinion?
- Who gets the alert?
If you can't answer three of these, you have a demo.
The upshot
Models will keep getting better and cheaper. That raises the value of everything around them: the specification that tells an agent what to do, the evaluation that tells you if it did, and the operations that keep it running. The people who build that layer well are doing the hard part of AI engineering.