Agent benchmarks

    Not 'it feels better'. Fourteen questions better, two worse.

    Every custom agent gets its own set of questions. A new version answers them beside the live version. The reviewer decides with a scorecard in front of them, not on a good feeling.

    14 better4 tie2 worsewhat got worse comes firstlivenew
    How it works

    You ask the assistant you already use.

    Datahub is headless: benchmarks too are reachable over MCP. You start a comparison and read the outcome in Copilot, Claude, ChatGPT or another assistant.

    Compare my new version of the finance agent with the live version.

    your assistantBoth versions answer the same twenty questions, several times each. The scorecard follows.as role: controller

    You ask for the comparison.

    No test script and no separate program. Your assistant starts the benchmark within your rights.

    Where did it get worse?

    your assistantOn two questions about cost centres. I will put both answers side by side.14 better · 4 tie · 2 worse

    You read what changed.

    The questions that got worse come first. That is where the conversation is, not in the average.

    Submit this version for review.

    your assistantSubmitted. The scorecard is with the review. Approving is up to the steward.awaiting approval

    A person decides.

    Your assistant can submit, not approve. The reviewer sees the evidence and takes the decision.

    What is in it

    The same questions. Two versions. A fair judge.

    A change to an agent starts as a draft, is reviewed and only then goes live. A benchmark adds evidence to that path.

    went wrong once

    A question set for each agent.

    The questions your agent must get right, with the expected answer where there is one. The team that owns the agent owns the set. Every question that once went wrong is added.

    livenewsame question

    Set beside the live version.

    The new version and the live version answer the same questions, several times each. So a lucky answer does not count as an improvement.

    ABBAverdict

    A judge who does not know who wrote what.

    An AI judge compares each pair of answers twice, in both orders. If the two verdicts disagree, it is a tie. The judge model is on every scorecard.

    Tested as used

    Tested as the people who will use it.

    An agent's answer depends on what it may see. So a benchmark runs the way the agent will really run.

    As a role.

    The agent answers as a controller or a sales lead would see it, with exactly that role's access. The author never sees more than their own role allows.

    group

    With the agent's own rights.

    Some agents work under their own, approved access. Those are tested with exactly those rights, during review. Only the people who approve that access read the answers. Everyone else sees the scores.

    In the review

    The scorecard advises. People decide.

    When an author submits a new version, the benchmark runs on its own. The scorecard appears in the review. No version is blocked by a number, because the owners know things no score captures.

    draftin reviewapprovedreaders: v2

    The live version keeps answering while the new one is weighed.

    On record

    An auditor can explain why a version went live.

    Each run records who ran it, under which role, which versions were compared and which judge decided. Teams dare to improve more often, because every change is set against the same questions.

    whoroleversionsjudge

    Who, which role, which versions, which judge.

    Frequently asked questions

    What reviewers want to know.

    Next step

    Better agents. And the proof that they are better.

    An intro call looks at which agent in your organisation deserves its own set of questions first.