Same salesperson, same afternoon, a different kind of request. Not a feature — a conversation. She wants an agent that can handle the refund objection this prospect keeps raising, the way their own support team would want it handled. Three candidate agents come back within the hour. She talks to each of them for a minute, picks the one that feels the most natural, and it ships the next morning.
Two weeks later, a customer with a slightly different version of the same complaint gets a refund nobody approved, because the agent improvised its way to an exception that sounded reasonable in the moment and wasn't. Nothing in the demo predicted this. The demo was fine. The demo was always going to be fine — that's not what it was measuring.
Everything in "Fast, and Still Yours" is still true. The compression is real, the gates are the right ones to keep, the memory and the reversibility scoring all still matter. What changes is that every one of those mechanisms was quietly leaning on an assumption that deterministic software gets for free and an agent doesn't: that what you saw once is what you'll keep getting.
The trick that stopped working
A deterministic prototype gives you a real inference. If the demo handled the input correctly, the shipped version will handle that same input correctly forever, because it's the same code running the same path. A five-minute taste test is legitimate evidence precisely because determinism makes one good run generalize.
An agent breaks that inference completely. A single conversation is one sample from a distribution of things the agent might do when it meets the next hundred customers, and a good sample tells you almost nothing about the tail. The demo isn't weaker evidence for an agentic candidate — it's close to no evidence at all, dressed in the same format that made it trustworthy for code. The taste test still belongs in the loop. It just can't be looking at a single conversation anymore. It needs to be looking at an eval — a score across a representative spread of scenarios, with the failure categories named, not smoothed over into one number. "Handled 94% of test conversations acceptably, failed by improvising exceptions in the refund-edge-case category" is a comparison a human can act on. "It felt right when I talked to it" is not, no matter how confident it feels in the room.
The spec is not the prompt, and someone has to own it
For deterministic software, approving a prototype is a build decision, done once. For an agent, what got approved is a system prompt, an eval suite, and a boundary on what the agent is allowed to do — and that's a standing specification, not a one-time artifact. It needs an accountable author the same way a technical standard does: version control, a change log, a named owner who's responsible for it, not just whoever happened to be in the room when it shipped. The uncomfortable part is that this spec can drift without anyone touching it, because the model underneath it can change. An approval that was correct in June can be silently wrong in September, with no diff to point to and no one who did anything wrong.
Reversibility has two layers now
The reversibility-and-impact scoring in "Fast, and Still Yours" asked whether a decision could be cheaply undone. For an agent, that question has to be asked twice. Can the deployment be rolled back — yes, same as any feature. But can the actions the agent already took on someone's behalf be undone — the email already sent, the refund already issued, the escalation already triggered? Those don't roll back with the code. So the tool and capability surface an agent is given — what it can read, what it can write, what it can act on without asking — becomes its own blast-radius dimension, independent of how reversible the feature launch looks on paper. A narrow, read-only agent and a broad one with write access to production systems deserve very different gate friction even when the surrounding feature is identical.
The gate doesn't close, it stays open
A deterministic feature clears its gate and the decision is done. An agentic one clears its gate and then keeps generating new instances of the same decision, live, every time someone talks to it — which means the gate can't be a single moment, it has to be a standing check. Re-run the eval when the model changes. Re-run it on a schedule even when nothing visibly changed, because nothing visibly changing is exactly the failure mode. Treat "we approved this in June" as a claim with an expiration date, not a permanent fact.
What to do now
Replace the demo with an eval before anything reaches the taste-call gate. A single conversation is not evidence for an agentic candidate. A distribution with named failure modes is the minimum bar.
Score the tool and capability surface on its own blast radius. What the agent can do to the world is a separate risk from how reversible the launch decision is, and it needs its own number, not a footnote.
Name an accountable owner for the standing spec, not just an approver of the ship decision. The system prompt, the eval suite, and the capability boundary are a product artifact with a change log — treat it like one.
Re-check what's already shipped, on a trigger and on a schedule. An approved agent isn't a closed decision. It's an open one that happens to be quiet right now.
Fast is still the right goal. The company that gets this right isn't slower — it just knows the difference between a decision that holds still once it's made and one that keeps deciding, live, on its own, every time someone talks to it. Treat the second kind like the first, and the thing that made "Fast, and Still Yours" true stops being true exactly where it matters most.
Source. This piece extends "Fast, and Still Yours" and the argument made in "When the Product Is the Agent" — specifically that acceptance criteria give way to evals, that a system prompt is a product specification requiring an accountable owner, and that tool and capability scope is a product decision with real blast radius. It is original synthesis, developed in conversation with Claude (Anthropic), rather than a response to any single external talk or publication. No third party's work is characterized here.