Somewhere in your team there is a dashboard with a single accuracy number on it. Let's say 84%.
It goes up when someone ships a good change. It goes down when someone ships a bad one. It gets quoted in planning meetings, and everyone nods, because 84% sounds like something you can improve.
Here is the problem. When that number drops from 84% to 80%, nobody can tell you what broke. Was the agent choosing the wrong action? Or did it choose the right action and the tool did the wrong thing? Those are two completely different bugs, owned by different people, fixed in completely different ways. A single score cannot tell them apart, and a team that cannot tell them apart ends up debugging blind.
This post is about splitting that number in two, and why it changes how you build and run an agent.
Two layers hiding behind one score
Most agents, whatever the framework, have the same basic anatomy. There is a decision layer, which reads the user's request and decides what to do. And there is an execution layer, which does it: calls a tool, queries data, runs a calculation, retrieves a document.
I'll call the first planning and the second tool execution. The labels matter less than the split.
- Planning accuracy: did the agent choose the right action, with the right parameters?
- Tool accuracy: given the right action, did the tool return the right result and report it correctly?
This applies whether you have a planner and sub-agents, a single agent with a skill library, or a plain tool-calling loop. In my last post on sub-agents I argued for composing skills into one context instead of routing between agents. That argument is about architecture. This one is about measurement, and it holds either way, because even in a single context window there is still a moment where the model decides what to do and a moment where something does it.
Most of your evals only see the final answer
Before splitting the number, it's worth listing the evals a typical agent team already runs and asking a simple question of each: which layer can this one actually see?
| Eval | What it looks at | Layer it can see |
|---|---|---|
| Offline golden-set accuracy | Final answer against a reference answer | End to end only |
| LLM-as-judge correctness, run on production traffic | Final answer | End to end only |
| Tone, formatting and style judges | Wording of the final answer | Generation only |
| Safety and advice-boundary checks | Final answer, sometimes the refusal behaviour | Generation, and some planning |
| Intent classification and routing accuracy | The planner's choice of action | Planning |
| Tool correctness (right tool, right arguments) | The tool call itself | Planning |
| Tool call count against an expected number | The trace | Planning behaviour |
| Direct tool tests with fixed inputs | The tool's output | Tool execution |
| Retrieval hit rate | The retrieved context | Tool execution |
| Faithfulness to the retrieved context | Answer against context | Generation |
This is a typical stack, not a description of any one product, and your own list will differ.
You can build most of this yourself, but there is a growing ecosystem of tools for it. Open-source evaluation frameworks like DeepEval cover many of the offline metrics above, including tool correctness. Observability and evaluation platforms such as LangSmith (and its Engine capability) and Arize focus on tracing and on running evaluators over production traffic. I'm not recommending one over another here. The useful question to ask of any of them is the same one I'm asking of your own evals: which layer can this see, and can it join a planner's decision to a tool's output for the same request?
Read down the right-hand column. The evals most teams lean on hardest, the golden-set score and the production judge, sit at the top and can only see the product of every layer at once. They're the ones that produce the single number. The evals that can actually localise a failure are further down, and they tend to be the ones teams build last, if at all.
That's not an argument for dropping the end-to-end evals. They're the closest thing you have to the user's experience, and you need them. It's an argument for treating them as the smoke alarm, and the layer-specific evals as the thing that tells you which room is on fire.
What the single number hides
Take a golden set of 200 queries. The end-to-end score comes back at 84%, so 168 are right and 32 are wrong. Now do the work of attributing those 32 failures.
| Count | Share of 200 | |
|---|---|---|
| Correct end to end | 168 | 84% |
| Wrong action chosen (planning fault) | 12 | 6% |
| Right action, bad result (tool fault) | 20 | 10% |
Two numbers fall out of that:
- Planning accuracy: 188 of 200 queries got the right action, so 94%.
- Tool accuracy: of those 188, 168 produced the right result, so about 89%.
The headline score is those two multiplied together (0.94 × 0.894 ≈ 0.84). The single number is a product of two very different things, and it tells you neither.
Now look at what you can do with the split. The team arguing about prompts and routing is looking at a 6% problem. The bigger share of the loss, 10 points out of 16, sits in the tools. If you only had the 84%, the instinct for most teams is to reach for the prompt, because the prompt is the lever everyone can pull. With the split in front of you, that instinct might be wrong.
I want to be careful here, because I don't think the answer is always "it's the tools". In plenty of systems it's the planner. The point is that you should be able to see which, and a single number guarantees you can't.
Different bugs, different owners
This is the part I care most about as a product person, because it's where measurement turns into organisation.
A planning fault is usually fixed by changing how the decision gets made: the prompt, the intent classification, the tool descriptions the model reads, the routing logic, the examples you give it. The person who owns this is whoever owns the agent's behaviour.
A tool fault is usually fixed by changing what the tool does: the query, the data, the calculation, the API it wraps, the way results are formatted before the model sees them. The person who owns this is often somebody else entirely, perhaps a data engineer or the team that owns the underlying service.
When you only have one number, every failure lands in the same bucket and goes to the same person. That person spends a week tuning a prompt to fix what was actually a stale data source, ships a change that moves nothing, and concludes the model is the problem. I've watched this loop run for much longer than it should.
Splitting the metric is also a way of splitting the blame fairly. A tool team that gets marked down for the planner choosing the wrong tool will stop trusting the eval, and an eval nobody trusts is worse than no eval.
How to actually measure each one
There are three practical moves here, and none of them are exotic.
1. Score tools conditionally. Tool accuracy should be measured only on the cases where the planner got it right. Otherwise you penalise the tool for a decision it never made. That's the 89% above, rather than a raw 168 out of 200.
2. Test each layer in isolation as well as together. Run the tools directly with fixed, known-good inputs and check the outputs. Run the planner with mocked tool responses and check it picks the right action and parameters. End-to-end evals tell you whether the system works. Isolated evals tell you why it doesn't.
3. Capture the trace, with one request id across everything. This is the unglamorous one and it's the real blocker. To attribute a failure you need to see the planner's decision and the tool's output for the same request. If your logs live in four platforms with no shared identifier, you cannot join them, and attribution turns into archaeology. A single request id that follows the query through every component is worth more than any clever metric you could add on top.
A cheap signal that catches a lot: tool call count
If you add one thing to a planning eval, add the expected number of tool calls for each query.
For each golden question, annotate how many tool calls a good answer should need. Then compare that with what the agent actually did. Track the deviation in both directions:
- Over-calling wastes latency and cost, and often means the agent is flailing.
- Under-calling often means it guessed instead of checking, which is how confident wrong answers get produced.
Both are deviations. It's tempting to only flag the first, because cost is easy to see, but the second is where the quiet correctness failures live. One practical detail: have a human (or a carefully checked judge) set the expected count when the dataset is built, not at eval time. And consider a small range instead of a fixed integer, since there can be more than one valid path.
There's usually a third layer: generation
Even with planning and tools split cleanly, there is often another failure sitting between "the tool returned the right data" and "the user got the right answer": the model's own writing.
This one is easy to miss with retrieval-heavy systems. You can have a retrieval step that looks excellent, finding the right passage almost every time, and still ship answers that are wrong, because getting the right context in front of the model does not guarantee it will use it correctly, cite it properly or avoid embellishing it. Retrieval can be strong while generation is the weak link. An eval that only checks the final answer can't tell you which.
So if your headline number is stuck, ask whether you can localise the loss through the whole chain: plan, retrieve or execute, then generate. If you can't, that's your next piece of work, ahead of any model change.
What a "correct" plan actually includes
One more thing about the planning side, because it surprises people.
Sometimes the right plan is to do nothing. If a user asks for something the product can't do, a planner that recognises that and says so has succeeded. A planner that confidently routes the request to the nearest tool and returns something plausible has failed, even if the answer looked fine.
This means your golden set needs unsupported and out-of-scope questions in it, and the scoring for those needs to reward a clean, honest decline. Most hand-built datasets are thin here, because the hard cases (unsupported, conversational, multi-step) are the ones that take effort to write. They're also the ones that most affect how a product feels to use. If you only test the questions your agent can answer, you've built an eval that measures the best case.
Why this matters beyond debugging
There's a strategic reason to get this right, and it shows up the moment someone proposes changing the architecture.
Suppose a team wants to try a different orchestration approach. The sensible question is whether it's better. But you can only answer that if you can measure both approaches against the same ground truth, on the same dimensions. A single end-to-end number can move for reasons unrelated to the change, such as an unrelated tool improving, and it can stay flat while the planner gets better and a tool gets worse. If you want to judge an architecture decision, you need planning accuracy and tool accuracy as separate, trendable numbers.
In other words, fixing your evals is not a side project that runs in parallel with the architecture question. It's the precondition for answering it well.
If you only do one thing
Skip the tooling for now and read your own failures. Take the last 30 or 40 wrong answers and label each one by hand: planner, tool or generation. It's tedious, and it will teach you more than another week of prompt tuning.
Once you've seen the split, the rest tends to follow on its own. You'll want a request id that survives every hop so you don't have to do the labelling by hand next time. You'll report planning and tool accuracy next to the headline number, not instead of it. You'll add the hard cases to your golden set, including the questions your agent should decline, and you'll start counting a clean "I can't do that" as a pass.
The part I'd push hardest on is ownership. Put a name next to each number. A metric nobody owns turns into dashboard decoration, and a metric with someone's name against it tends to get fixed.
Where I've landed
A single accuracy number is a fine thing to report upward. It's a poor thing to debug with. The moment your agent has more than one layer, which is nearly always, the honest question isn't "how accurate is it?" It's "where does it go wrong, and whose problem is that?"
I expect some of the details here to look naive in a year, as tooling for tracing and attribution gets better. But I'd be surprised if the underlying idea ages badly. If you've split your own eval this way and found the answer surprised you, I'd like to hear what you found.