Monday morning. A data steward sharpens the finance agent. The new version handles cost centres better and cites its sources sooner. Before it goes live, the reviewer asks the question every reviewer asks: is it actually better, or only different?
Until now the honest answer was a few test questions and a good feeling. That is no criticism of the reviewer. There was simply nothing better.
Why a good feeling is not enough
An agent is not a report. In a report the figure is right or it is not. An agent answers in language, to questions nobody knows in advance. Change one instruction and its behaviour shifts in places you were not thinking about. The question about cost centres goes better. The question about depreciation, which went fine last month, perhaps no longer does.
On top of that, a language model never says exactly the same thing twice. One good answer in a test can be luck. Rely on a handful of test questions and you will sometimes approve a lucky answer.
The result is predictable. Teams that see the risk would rather leave their agent alone. Teams that do not see it change the agent and find out later. Either way the agent does not keep up with the work.
A handful of test questions versus a fixed set, asked several times and tested as the user.
What a benchmark changes
In Datahub a custom agent already follows a set path: a change starts as a draft, is reviewed and only then goes live. Agent benchmarks add evidence to that path. Four things make the difference.
From an own set of questions to a person's decision.
Every agent has its own set of questions. The questions it must get right, with the expected answer where there is one. The team that owns the agent owns the set. Every question that once went wrong is added, so the same mistake does not return unnoticed.
The new version is set beside the live version. Both answer the same questions, several times each. So a lucky answer does not count as an improvement.
The judge does not know who wrote what. An AI judge compares each pair of answers twice, in both orders. If the two verdicts disagree, it is a tie. The model that judged is on every scorecard, which keeps results comparable over time.
The scorecard is readable. Better, tie, worse. The questions that got worse come first, with both answers side by side.
Not "it feels better". Fourteen questions better, two worse. And here are the two.
Tested as the people who use it
What an agent answers depends on what it may see. A controller sees more than an intern and rightly gets a different answer. An agent gets meaning, not bulk. A test that runs with the author's rights therefore says little about what the user will get.
A benchmark runs the way the agent will really run. That can be as a role: the agent answers as a controller or a sales lead would see it, with exactly that access. The author never sees more than their own role allows.
It can also be with the agent's own rights, for agents that work under their own, approved access. Then only the people who approve that access read the answers. Everyone else sees the scores.
The scorecard advises, people decide
When an author submits a new version, the benchmark runs on its own. The scorecard appears in the review.
A version is never blocked by a number. That is deliberate. The owners of an agent know things no score captures: that a question is out of date, or that one loss is worth what comes back for it. What changes is the conversation. From "looks fine to me" to "twelve questions better, let us look at the one it lost".
There is one more reader: the auditor. Each run records who ran it, under which role, which versions were compared and which judge decided. Why a version went live can be retold. How roles and the log are set up beyond that is on the security page.
You ask your own assistant
Nobody has to learn a new program for this. Datahub is headless: benchmarks too are reachable over MCP. You ask the assistant you already use to compare the new version with the live one. You read the outcome there as well. Why you connect once and not per agent for that is in an earlier article.
Benchmarks run in your own Datahub environment, on the AI model your organisation already chose, under the same roles, guardrails and audit trail as every other agent action.
Where to start
- Pick the agent that is used most.
- Have the team that owns it write down ten questions it must get right.
- Add every question that goes wrong in practice.
- Compare the next change with the live version before you submit it.
Better agents are good. Better agents you can show are better are what earn trust.
Read on about agent benchmarks, custom agents and the company brain they run on.
