Editor’s note
Over the past few weeks, we’ve been talking about context engineering, loops, GraphRAG, and all the machinery that sits around modern AI agents.
This week, we’re deliberately taking a step back.
A year ago, most discussions focused on evaluating models. Today, more teams are discovering that evaluating agents is a very different problem. Once an AI system starts planning, calling tools, retrieving memory, and coordinating multiple steps, checking whether the final answer looks right simply isn’t enough. The execution path matters just as much as the destination.
So before we continue exploring more advanced engineering patterns, it felt worth revisiting one of the foundations.
In this issue, Balamurugan Balakreshnan, who works with enterprise AI teams at NVIDIA, walks through how he thinks about evaluation in agentic systems, why trust has to be engineered from the beginning, and which parts of an agent deserve measuring before production.
One of the biggest challenges I see in getting agentic AI systems into production isn’t the technology itself. It’s trust.
We’ve made enormous progress in building models that can reason, write code, call tools, and orchestrate complex workflows. But once those systems start making decisions on their own, a different question seems to start taking up space: How do we know we can trust what they’re doing?
Of course, the simplest answer is to have humans validate every output. But that will never sustain once you move beyond prototypes. If an agent is handling thousands of requests a day, human review becomes the bottleneck. So we need another way to build confidence.
Trust has to be designed
Models are trained on data that reflects the world at a particular point in time. We can improve their responses by providing fresh context during inference, whether that’s enterprise knowledge, live business data, or current events. That helps, but better context doesn’t automatically make an output trustworthy. Trust still has to be earned through evaluation.
In my experience, the best place to start is during development, not after deployment. Before a model ever reaches production, it should be evaluated for things like correctness, relevance, completeness, grounding, and safety. None of this is easy. Good evaluation depends on good benchmark datasets, domain expertise, and clear success criteria. A coding model won’t be evaluated the same way as a reasoning model, and every organisation will have its own priorities beyond the standard metrics.
And the work doesn’t stop once the model ships.
Production systems need continuous evaluation because real users rarely behave as we expect. The data collected in production often tells you more about your system than anything you learned during testing. As models continue through fine-tuning, post-training optimisation, or other updates, both the data and the evaluation process evolve alongside them.
Why evaluating agents is different
Evaluating a model is one thing. Evaluating an agent is something else entirely.
An agent doesn’t just generate text. It interprets the user’s intent, builds a plan, decides which tools to call, retrieves information, and coordinates multiple actions before producing a result. In many systems, there’s also a planning or supervisor agent deciding which specialist agents, MCP servers, or data sources should be involved. In multi-agent systems, every additional agent introduces another layer of decision-making.
That’s why evaluating the final answer is no longer enough.
We also need to understand how the agent arrived there.
When an agent runs in production, I think about questions like these:
Was the execution plan appropriate for the user’s request?
Did it choose the right specialist agents?
Did it select the right tools or MCP servers?
Did it retrieve the right information from memory or external knowledge?
If a judge model was involved, has that model itself been properly evaluated?
These questions matter because agents are making decisions continuously as they execute a workflow. Trust comes not just from the quality of the final answer, but from confidence that the system made good decisions all the way through.
So what should we actually evaluate?
This is the question I get asked most often. My honest and perhaps bland answer is: it depends.
Every application is different. The metrics you care about will vary depending on your domain, the framework you’re using, and what success looks like for your users. There isn’t a universal scorecard for agentic AI.
That said, I’ve found there are a handful of areas that every team should think about.
Task performance
The first question is the simplest one. Did the agent accomplish what it was asked to do in the first place?
That sounds obvious, but it’s surprisingly easy to focus on intermediate steps and forget the end goal. At a minimum, I’d want to understand things like:
Goal achievement
Subtask completion rate
End-to-end success rate
First-attempt success rate
Human evaluation
Reasoning quality
A correct answer isn’t always the result of good reasoning. Sometimes an agent simply gets lucky.
Because agents plan before they act, it’s important to evaluate the quality of that planning as well. Questions worth asking include: Was the plan coherent? Did it retain the right context? Could it recover when something went wrong? Did it follow instructions, or did it drift off course?
Some useful metrics here include:
Plan coherence
Step accuracy
Self-correction ability
Hallucination rate
Context retention
Decision quality
Instruction adherence
Tool selection and orchestration
Most production agents don’t work alone. They’re constantly interacting with APIs, databases, MCP servers, and external tools.
Choosing the wrong tool can be just as damaging as producing the wrong answer, so it’s important to evaluate how well the agent orchestrates the systems around it.
Some useful measures include:
Tool call accuracy
Tool selection precision
Parameter correctness
API call success rate
Error recovery rate
Orchestration efficiency
Unnecessary tool invocation rate
Safety and reliability
No production system is complete without guardrails.
Agents should stay within their intended scope, handle failures gracefully, and avoid actions that could compromise users or data. Depending on your application, you might look at:
Refusal accuracy
Guardrail compliance
Harmful action rate
Scope creep
Failure mode coverage
Graceful degradation
Uptime and availability
Data leakage incidents
Cost and efficiency
Finally, there’s the operational side of the equation.
An agent that produces excellent results but takes five minutes and thousands of tokens to complete a simple task may not be practical in production. That’s why it’s worth tracking metrics such as:
Latency per task
Token usage
Cost per successful task
Number of execution steps
Retry overhead
Throughput
Memory efficiency
These aren’t meant to be a checklist that every team must follow. Think of them as a menu rather than a mandate. Different frameworks expose different signals, and different applications will naturally prioritise different measures.
The important thing is to be deliberate about what you’re evaluating. If you only measure the final response, you’ll miss most of what determines whether an agent is actually reliable.
For me, that’s the biggest shift agentic AI introduces.
Trust doesn’t come from a good answer alone. It comes from understanding how that answer was produced. That means evaluating the planning process, the decisions made along the way, the tools that were selected, the knowledge that was retrieved, and the actions the agent ultimately took.
That’s it for this one. We’ll pick up the conversation next week.
Until then, keep building.
Tanya D’cruz
Editor-in-Chief
