Custom LLM metrics for AI Agent Evaluation
Our evaluation metrics could prove an agent called the right tool. They could not tell a team whether the output was any good. This is how we gave agent builders a no-code way to write their own criteria, generate a judge from them, and earn confidence in it before it can affect anyone else's results.
- Surface
- Agent Studio, AutoEval
- Primary user
- Agent builders
- Target
- Q4 2026, internal first
- Status
- Prototype in review
The teams closest to the product had already stopped using our metrics
Before any design, research went to the people who evaluate agents for a living: ten quality engineers and their managers across seven business units, in hour-long interviews. They build and ship hundreds of the agents and skills customers get out of the box. If our metrics do not work for them, they will not work for a customer either.
Participants: six quality engineers and four managers, 60-minute semi-structured interviews.
Business units including HRSD, ITSM, SPM, SecOps, CRM and Platform.
Binary scoring. A right or wrong answer will not get a feature shipped. Teams rebuilt the same metric on a 1 to 5 rubric.
Process, not outcome. Our metrics measured whether the agent called the right tool. Customers ask what it achieved.
No sense of context. One definition of completeness cannot serve document summarisation, security analysis and an HR workflow at once.
Graceful failures score as wins. A workflow that exits through its own error path still counted as a completed task.
Scores that flatter. Results landed above 90% while manual review said the experience was worse.
"None of the customers are interested in tool calling correctness or tool choice accuracy. All they want is output eval. They want to do evaluations with their own criteria and in the manner they want to do it."
Research participant
"We are unable to use out of the box metrics because they are, like, zero and one. If you tell me it is sixty percent good, I'll be more confident to push the feature out."
Research participant
"The customers are not going to worry about AI metrics. Customers are going to worry about the ROI."
Research participant
| Level | How the team treats metrics | What they need from us |
|---|---|---|
| 1 Foundational | Uses the built-in metrics as they are, often unaware customisation is possible. | To know a better fit exists, and to be told when it applies. |
| 2 Developing | Clones a built-in metric and rewrites the judge prompt for one use case. Rarely reuses it. | A supported clone-and-edit path, and a reason to publish it for others. |
| 3 Advanced | Runs on outcome-based custom metrics, keeping the built-ins for debugging intermediate steps. | A library, versioning, and confidence that a judge scores consistently. |
A domain expert who is not a prompt engineer
The person who knows what a good resolution note looks like is rarely the person who enjoys writing an evaluation script. Sam is technical enough to build an agent and specific enough to know exactly where it falls short, which is why the design has to translate judgement into criteria rather than ask for prompt craft.
Creates and tests both the agent and the metric
Works inside the customer company, enabling its employees or its own customers
Evaluate quality criteria specific to their business
Create a metric without writing a script
Test a metric before publishing it
Understand why a score came out the way it did
Reuse and share metrics instead of rebuilding them
Metric needs are more nuanced than one built-in definition
The only workaround today is too technical for most builders
Judging the judge: no way to know whether to trust a score
Discoverability, and a workflow fragmented across surfaces
No guidance on how to actually create a metric
A design review that sent the work back to first principles
The first design surfaced every capability the platform could offer. In review, the critique was direct and it was right: this was platform-centric, not user-centric. It showed people what the system can do instead of guiding them to a metric that works. The precedent was uncomfortable and close to home. Agent Studio had shipped that way, and users built things that did not work.
Get it wrong and we enable people to fail at something they will then blame their agent for. Get it right and metric quality goes up while support load goes down. So the design stopped adding surface area and went looking for constraints worth enforcing.
Observe. How do people create a custom metric today? Where do they start, what is the first action, and what separates the ones who succeed from the ones who get stuck?
Prescribe. Which practices predict a working metric, and can we build them in as requirements rather than optional helpers?
Interrogate. Are custom metrics even the right answer, or should the built-in metrics get better first?
The bar we set: two or three opinionated constraints, each backed by evidence for why it improves the outcome.
Adding a prompt box is easy. Making a judge a team will ship on is not.
Four properties decide whether anyone trusts an LLM judge. Each one is a design problem, and each one has a mechanism in the release rather than a warning in the documentation.
Why it breaks trust
The same input can score differently across runs. A judge returning 0.9, then 0.2, then 0.85 on comparable input is worth nothing.
The mechanism that answers it
Temperature 0.1 by default, a fixed output format, and a results panel that accumulates runs so variance is visible before publishing.
Why it breaks trust
With no fixed rubric the model invents a new scoring framework every run, with different dimensions and scales. Scores stop being comparable.
The mechanism that answers it
The rubric is captured at creation. Dimensions and scale belong to the metric, not to the model.
Why it breaks trust
If test inputs do not map to the prompt's context variables, the judge scores incomplete context and fails silently. No error, just wrong numbers.
The mechanism that answers it
Variables come from the execution context, and several can be added to a single metric. Each one becomes its own labelled test field, never a single free-text box.
Why it breaks trust
Builders describe quality in domain terms. Judges need evaluator-precise criteria. That translation is where reliability is won or lost.
The mechanism that answers it
Guided elicitation turns plain-language anchors into structured criteria, and a prompt quality signal flags structural gaps before testing starts.
Six constraints we chose to enforce
Not guidelines. Each one closes a specific failure we watched happen, and each one costs the user a degree of freedom on purpose.
The full prompt is generated from the creation inputs, rubric included, before the user sees the editor. Refining a strong draft is a job a domain expert can do. Authoring a judge from an empty box is not.
Agent type first, because it decides which context variables exist. Then what the agent does, what to measure, a good anchor, a poor anchor, scoring type and levels, aggregation, name. Structured answers map cleanly to prompt sections.
Role, output format, chain-of-thought and bias guardrails stay system-owned and visibly so. Scoring downstream needs valid JSON in a fixed shape. Users can see every locked line, and cannot break it.
Publish stays disabled until a test run is on record, and it cannot be bypassed. Edit the prompt afterwards and the gate resets, because a score only describes the prompt that produced it.
Every test appends a row instead of overwriting the last one. Each run is judged against its own expected response, and no cross-run average is computed, because averaging unrelated examples produces a number that represents nothing.
A justification in plain language ships with every score, and a failed judge call shows as an error row rather than a zero. Users need to know what was missing, not just that the number was low.
Three dependent systems on a shared foundation
This is not one prompt editor. The systems are strictly sequential: the second is only as good as the first, and the third only as good as the second. A weak input is not caught automatically, so the quality signal and the test gate exist to catch it before it becomes a published score.
Input elicitation
Collect just enough structured knowledge to generate a reliable judge. Agent type, what the agent does, what to measure, good and poor anchors, scoring type and levels, aggregation.
Every input maps to a named prompt section.
Prompt generation and refinement
Assemble two feeds into one judge-ready prompt: what the user provided, and what the execution parser extracts from the real run. Then give them safe tools to improve it.
Locked zones, token-styled variables, edit with AI.
Testing and trust calibration
Turn a prompt into the confidence to publish it. Structured inputs, real executions, accumulating results, an advisory consistency signal and a gate that cannot be talked around.
Trust is earned by watching the judge score things you recognise.
Library, discovery and the always-on post-processor
Published metrics are versioned, carry real metadata, and are pre-selected for the agent they were built for. Behind the scenes a post-processor the user never sees maps judge output into the same result contract as the built-in metrics, so a custom score renders next to task completeness with no special casing downstream.
Every branch had to survive one constraint: nothing publishes untested, and editing the prompt sends you back through the loop.
Screen by screen
Walking a resolution note completeness metric from an empty library to a published, calibrated judge. Select any screen to open it larger.
System 1 · Input elicitation
System 2 · Prompt generation and refinement
System 3 · Testing and trust calibration
Foundation · Library and versions
Three scoring types, chosen before anything is generated
The industry has no settled answer here. One tool recommends categorical labels as easier to apply consistently, another supports five formats. What is not negotiable is when the choice happens: it determines rubric structure, aggregation options and the output schema, so it comes before generation, never after.
Numeric
A continuous scale for when partial credit is meaningful. Fixed anchors top and bottom, optional levels between, and the team picks how scores aggregate: average, percentage above a threshold, or strictest.
Pass or fail
Binary, for correctness checks with one right answer. The user defines what passing and failing look like. Aggregation is fixed to percentage passing, so there is nothing to get wrong.
Custom labels
Named tiers such as successful, partially successful and unsuccessful. The team sets each category name, its score value and whether it counts as a pass. Aggregation reports percentage per category.
Research was unambiguous on why this mattered: teams abandoned the built-in metrics partly because a binary result would not get a feature approved. Letting them choose the shape of the answer is not a preference setting. It is the reason they will use the metric at all.
Build the smallest loop that works, then earn the right to add
The recommendation was to prove that a builder can define, validate and reuse one business-specific judge before adding dataset-backed calibration, tools, multi-criteria scoring or automated optimisation. Deferrals here are sequenced, not rejected.
Define a metric in plain language, with no scripting
MVPInput-based creation, an auto-generated judge prompt, single-test calibration, the library and versioning. The one loop that has to work.
Validate a metric at scale against real datasets
DataKit integration, bulk testing and dataset-based calibration, once teams have metrics and data worth calibrating against.
Score complex, multi-part criteria
Multi-criteria metrics, weighted scoring, and building a metric conversationally rather than through a form.
Optimise judges automatically
HorizonStandardised context through OpenTelemetry traces so judges stop depending on one runtime, and automated prompt optimisation. It also needs labelled data, ground-truth examples and review of every prompt change, which is exactly what an MVP does not have.
| Area | In the MVP | Deliberately not yet | Reason |
|---|---|---|---|
| Authoring | Structured inputs | Conversational creation | Reliability over ambiguity |
| Validation | Single-test calibration | Bulk judge-the-judge, annotations | Avoid a dataset dependency |
| Context | Execution parser variables | Tools framework, trace dependency | Avoid a platform blocker |
| Optimisation | Metric quality signals | Automated prompt optimisation | Requires labelled datasets |
| Complexity | Single criterion | Multi-criteria, multi-judge | Prove the baseline first |
Three proofs, and the honest state of the numbers
Usability
A builder defines a business-specific metric without scripting or prompt-engineering expertise. Measured by input completion and how many people who start a metric publish one.
Trust
A builder tests judge behaviour before publishing or reusing it. Measured by whether every published metric has a recorded test. This one is definitional: the gate either works or it is a defect.
Operational fit
The metric runs inside AutoEval and maps to the same result contract as everything else. Measured by run completion and telemetry coverage.
Tests run per session before a metric is published, as the clearest read on how much iteration trust actually takes
Whether a custom metric selected at run setup is new or reused, which tells us if the library is doing its job
Whether anyone expands a justification, which tells us if they engage past the number
Runs completed with at least one custom metric, the primary adoption signal
Judge inconsistency could undermine trust before anyone builds any. A low temperature and a visible variance panel help; a baseline of real variance still needs an engineering spike.
Reliability depends entirely on input quality, and input quality cannot be validated automatically.
Prompt generation runs on a small model today. Generating a complete, context-aware judge may need a larger one.
Execution context is built against today's parser. Standardising on traces later is a real migration.
The numeric targets behind these proofs are product proposals, not engineering commitments. They are a starting point for the kickoff discussion, and saying so out loud is part of the work.
A team should be able to say what good looks like for their agent in their own words, watch a judge apply it to something that really happened, disagree with it, and only then let it score anyone else's work.
That sentence is the constraint every screen here was tested against.
- Product design
- Agentic AI
- Research
- UX Research, Agentic AI
- Product and engineering
- AI Evaluations