[ PRODUCT LEADER · AI STRATEGY · MELBOURNE ]
[ ← WORDS ]

[ AI · AGENTS · PRODUCT ]

Human in the loop is not a button

Most "human in the loop" designs are a confirmation dialog bolted on after the write. Here is what it takes to build approval into an agent properly: gates, evidence, reversibility and the slow, human parts.

Most "human in the loop" designs I see are a confirmation dialog bolted onto the end of a write. The agent does the thing, the UI says "Review", the user clicks something, and everyone agrees the box has been ticked.

That is not a loop. It is a label.

I've spent the last few months close to a set of agent write actions in an accounting product, and the thing that has surprised me most is how little of the hard work is the model. The model is the easy part. The hard parts are deciding where the write actually happens, what a human is really agreeing to, how you prove it afterwards, and what you do when it goes wrong. None of that is solved by a better prompt.

This post is the checklist I wish I'd had at the start. It is deliberately product-agnostic. The examples are invoices and contacts because those are the ones I know, but the shape applies to anything where an agent can change someone's data.

Start with a read-only world, and notice what changes

Most of us began with read-only agents. Ask a question, get an answer, worst case you get a wrong number. My Stride MCP side project is a good example: three boring tools, all reads. If the model gets it wrong, the user rolls their eyes and asks again.

The moment you add a write, the failure mode changes category. A wrong answer is an annoyance. A wrong write is a record in someone's books, an email to their customer, or a payment to the wrong place. The risk model, the legal posture and the engineering all shift at once, and you want to notice that shift on purpose rather than discover it in review.

The first test: where does the write actually happen?

Here is the question I now ask first, and it filters out a surprising number of designs.

At the moment the human is asked to approve, has anything been written yet?

If the answer is yes, you do not have human in the loop. You have human after the loop. The record is already saved, and the confirmation card is a notification in a nicer outfit. Changing the button from "Create" to "Review" changes the wording, not the behaviour.

I ran into exactly this while scoping a proof of concept. The flow looked like approval. The UI showed an invoice and invited the user to check it. But the invoice already existed as a saved record by the time they saw it. To make it a genuine gate you either hold the write until the human says yes, or you move the real approval to the next irreversible step (sending). Both are legitimate. Pretending the first one is the second one is not.

A simple way to hold yourself honest: draw the sequence and mark the exact line where state changes in the system of record. Then mark where the human decision sits. If the human line is below the state line, you are building a review step, not an approval gate. Review steps are fine and useful. Just call them that.

Gates, plural

The next thing that falls out is that "the approval" is rarely one thing.

Take the lifecycle of an invoice: create it, then send it. Those are different hurdles with different consequences and, importantly, different data you need from the human at each one. A draft sitting in a list is roughly as harmful as a draft a user made by hand. A sent invoice is a document in a third party's inbox with tax and consumer implications attached.

So I treat create and send as separate gates by default, and make anyone argue for collapsing them. Collapsing is sometimes right, but it should be a decision with a reason, not an accident of how the API was shaped.

This also gives you a useful ladder. You can start at "agent drafts, human sends" and earn your way up as the evidence comes in. I would resist locking draft-only in as a permanent position. Risk appetite moves as confidence grows, and your design should leave room for that without a rewrite.

What the human sees matters more than whether they click

A click is not understanding. Three things I now care about on the approval surface.

Show the artefact, not the model's description of it. If the thing that will go out is an invoice, the human should see the invoice as it will actually exist, rendered from the same payload that will be written. A model-written summary ("I've created an invoice for Acme for $4,000") is a claim about the write, not the write. Summaries drift from payloads in quiet, expensive ways.

Bind the approval to the exact payload. This is the engineering one people skip. If the user approves version A and the system executes version B because the model tweaked a field in between, your approval record is fiction. The approval should reference a specific, immutable proposed action (an id, a hash, a version), and the executor should refuse anything that doesn't match. It also means the pending action has to live somewhere durable enough to survive a refresh or a session timeout, which turns out to be a real design question (database versus session memory, with real latency and complexity trade-offs on each side).

Keep the model out of content it has no business inventing. In the proof of concept we scoped, a deliberate boundary was that the invoice should reflect what the user supplied, not what the model decided would be nice to write. The more generative freedom the model has over the content of a write, the more the human has to verify, and the less likely they are to actually do it.

Consent you can prove

There is a quiet difference between a user having asked for something and a user having acknowledged something. A chat message like "create an invoice for Acme" is the first. An LLM's interpretation of that string is not a captured acknowledgement of the specific record that results.

That is where the audit conversation lives, and it is usually where the interesting disagreements are. One camp says an explicit click is worth having because it gives you a clean event to record. Another says creating a saved, unsent draft is low risk, equivalent to a manual draft, and a forced extra click is a tax on every user. Both are making a fair point.

What helped me was separating two questions that people tend to blur together:

  • Is the approval a legal or audit requirement?
  • Or is it a risk preference?

They get decided by different people with different evidence, and you want to know which one you are arguing about. I would also be careful to treat any legal position as theirs to state, not mine to summarise on a blog. The useful product move is to put the question to legal and security in a form they can actually answer, with the flows drawn out side by side, rather than asking "is this okay?" in the abstract.

Whatever you decide, a few things are worth recording against the action:

  • Who approved, and in what capacity (themselves, or on behalf of someone else)
  • What exactly they were shown
  • When, and through which surface
  • What the system then did, linked back to the approval

And think hard about identity. One of the most practical risks in a multi-business product is simply writing to the wrong one. A confirmation gate that names the business being written to is a cheap and real control. A recorded "viewed" event is better than nothing, but be honest that it is weaker evidence than an explicit action.

Approval fatigue is a safety problem

Every gate you add trains the user. Add too many and they learn to click through, and now you have the worst of both worlds: friction plus a false sense of safety.

So gates should be proportionate to blast radius, not applied uniformly. Low-consequence, easily-corrected actions can be light. Anything that leaves the building, moves money or touches a third party deserves a real stop. If everything asks for confirmation, nothing does.

This is also why the "just add a confirmation" instinct is dangerous. It feels like a free safety margin and it isn't. It spends user attention, which is the one resource the whole pattern depends on.

Design for the bad write, not just the good one

Here is the question that tends to go quiet in a room: what happens when it's wrong anyway?

Approval reduces bad writes. It does not eliminate them. People approve things they skimmed. Models misread. Upstream data is stale. So you need a recovery path, and you need to know what your platform actually supports.

If the underlying system has no void, no reversal, no delete for the thing you are writing, then there is no in-conversation way to undo a mistake, and your recovery story has to be designed around that fact rather than assumed. That should push you toward actions that are naturally reversible first, and toward being very careful with the ones that aren't.

Related engineering hygiene that matters more with writes than reads:

  • Idempotency. Retries and double-submits should not create duplicates. Agents retry. Networks flake. Users double-click.
  • Scoped, least-privilege tools. Narrow tools with narrow inputs are easier to gate and easier to audit than one general "do anything" write.
  • Untrusted inputs. If the content driving a write came from a document, an email or a web page, treat it as data rather than instruction, and make the provenance visible to the person approving.

Start where you learn fastest, not where it looks most impressive

This one changed my mind in a real planning discussion, so I'll be specific.

The obvious demo for a write capability is something like "create invoice". It is high value and it looks great. But when we looked closely, it was a weak place to prove the human-in-the-loop pattern, for the reason above: the record was already saved by the time the user saw it. We'd have been validating the SDK, not the confirmation pattern.

The better candidate was something smaller and less glamorous: creating a contact. It has a genuine acceptance step before anything is written. It has a tiny required field set, so the confirmation card is simple. And you need it anyway, because you cannot invoice someone who doesn't exist yet. Low blast radius, real gate, useful regardless.

The argument underneath is about the speed of learning. The slow, uncertain parts of a write feature all cluster around the confirmation step: risk acceptance, architecture decisions, audit, on-behalf-of identity, rollout gating. If you prove the gate end to end on the smallest real action, you find the blockers early, and the people who need to weigh in get something concrete to react to sooner. Meanwhile the "bigger" action can proceed on a parallel track, since it is mostly an integration job once the pattern exists.

Thin slice first. Add requirements in stages. Do the part that teaches you the most before the part that impresses the most.

The slow part is not the code

If you take one operational lesson from this, take this one. The critical path on a safe-writes feature is almost never engineering. It is the sign-off from legal, security, AI safety and whoever owns identity and consent, and those groups move at the pace of the business, not the pace of your sprint.

Which means:

  • Start the clock early. Reviews that take weeks take weeks whether you begin now or later.
  • Run the technical and the legal and experience streams in parallel, not in sequence.
  • Make decisions legible. Write down the options, the positions people actually took, and what is still open. A decision record that says "proposed" and "agreed" in the right places is worth more than a polished deck.
  • Be upfront about what is a build constraint versus what is a risk preference. They have different owners.

Measure it, or you are just hoping

You cannot expand a write capability on vibes. Before you widen scope, decide what evidence you need. The numbers I would want:

  • How often agent-created records are wrong or unintended
  • How often approvals are rejected or edited (a useful signal that the proposal quality, or the preview, is off)
  • Reversal and correction rates
  • Support contacts attributable to agent actions
  • Completion rate of actions measured against attempts

And in your evals, report planning and tool execution separately. "The agent picked the wrong action" and "the agent picked the right action and the tool did the wrong thing" are different bugs with different owners, and a single accuracy number hides which one you have.

The checklist

If you only keep one section, keep this one. Before shipping an agent write, I'd want a clear answer to each of these.

  1. Where does state change relative to where the human decides?
  2. Is every irreversible step behind its own gate, and are create and send (or their equivalents) treated separately?
  3. Does the human see the real artefact, rendered from the payload that will be executed?
  4. Is the approval bound to that exact payload, and does it survive a session timeout?
  5. What do we record: who, in what capacity, shown what, when, and what happened next?
  6. How do we know it's the right account or business being written to?
  7. What is the recovery path when a bad write gets through, given what the platform can actually undo?
  8. Are the gates proportionate, or are we training users to click through?
  9. Who has to sign off, and have we started their clock?
  10. What evidence will we need before we widen the scope, and are we collecting it?

Where I've landed

Human in the loop is not a feature you add. It is a property of how the whole write path is designed: where state changes, what the person sees, what you can prove, and what you can take back. Done well, it is what lets you ship agent actions to real customers with a straight face. Done as a dialog box, it is a liability with a friendly button.

I'm still learning this, and I expect some of my current opinions to age badly as the tooling and the evidence improve. If you've shipped agent writes and found a gate that earned its keep, or one that turned out to be theatre, I'd like to hear about it.

[ GET IN TOUCH ]

Like what you’re reading?

I’m always up for a conversation about AI products, evals, or whatever you’re building.

[ SAY HELLO ↗ ]

[ MORE TO READ ]

  1. 01What I'd tell the next PM on an AI teamLessons from leading an agentic chat product inside an accounting platform: learn faster than you build, put legal and safety inside the harness, back your engineers' experiments, and be strict about the evidence even when you're loose about the code.[ PRODUCT · AI · AGENTS ]1 November 2026 · 7 min read
  2. 02MCP vs in-product AI: where should a job live?When a customer wants an AI to work with your product's data, should the job be served by an MCP server in the assistant they already use, or by AI inside your product? A working test, and why the line probably runs along consequence, not surface.[ MCP · AI · PRODUCT ]8 November 2026 · 13 min read