Custom LLM-based metrics — product journey

AI evaluations

Define what good means.
Then trust the score.

Our evaluation metrics could prove an agent called the right tool. They could not tell a team whether the output was any good. This is how we gave agent builders a no-code way to write their own criteria, generate a judge from them, and earn confidence in it before it can affect anyone else's results.

Surface: Agent Studio, AutoEval
Primary user: Agent builders
Target: Q4 2026, internal first
Status: Prototype in review

Anatomy of a judge prompt Yours Ours

# ROLELocked

Written from your description of what the agent does

# OUTPUT FORMATLocked

A fixed JSON contract the results pipeline depends on

# GOALEditable

What you said this metric should measure

# CONTEXTEditable

Variables the execution parser pulls from the real run

# CRITERIAEditable

Your good and poor response anchors

# SCORING RUBRICEditable

Your scoring type and the levels you defined

Lock the parts the score contract depends on, hand over the parts that carry a team's judgement, and never show a blank editor.

01Where this started

The teams closest to the product had already stopped using our metrics

Before any design, research went to the people who evaluate agents for a living: ten quality engineers and their managers across seven business units, in hour-long interviews. They build and ship hundreds of the agents and skills customers get out of the box. If our metrics do not work for them, they will not work for a customer either.

10

Participants: six quality engineers and four managers, 60-minute semi-structured interviews.

7

Business units: HRSD, ITSM, SPM, SecOps, an industry vertical, CRM and Platform.

6

Maturity facets, each scored across three levels, from metrics through to the production decision.

Five reasons the built-in metrics stopped being used

Binary scoring. A right or wrong answer will not get a feature shipped. Teams rebuilt the same metric on a 1 to 5 rubric.

Process, not outcome. Our metrics measured whether the agent called the right tool. Customers ask what it achieved.

No sense of context. One definition of completeness cannot serve document summarisation, security analysis and an HR workflow at once.

Graceful failures score as wins. A workflow that exits through its own error path still counted as a completed task.

Scores that flatter. Results landed above 90% while manual review said the experience was worse.

"None of the customers are interested in tool calling correctness or tool choice accuracy. All they want is output eval. They want to do evaluations with their own criteria and in the manner they want to do it."
Research participant

"We are unable to use out of the box metrics because they are, like, zero and one. If you tell me it is sixty percent good, I'll be more confident to push the feature out."
Research participant

"The customers are not going to worry about AI metrics. Customers are going to worry about the ROI."
Research participant

The maturity model that reframed the ask

Level	How the team treats metrics	What they need from us
1 Foundational	Uses the built-in metrics as they are, often unaware customisation is possible.	To know a better fit exists, and to be told when it applies.
2 Developing	Clones a built-in metric and rewrites the judge prompt for one use case. Rarely reuses it.	A supported clone-and-edit path, and a reason to publish it for others.
3 Advanced	Runs on outcome-based custom metrics, keeping the built-ins for debugging intermediate steps.	A library, versioning, and confidence that a judge scores consistently.

02Who we are building for

A domain expert who is not a prompt engineer

The person who knows what a good resolution note looks like is rarely the person who enjoys writing an evaluation script. Sam is technical enough to build an agent and specific enough to know exactly where it falls short, which is why the design has to translate judgement into criteria rather than ask for prompt craft.

Sam

AI practitioner

Primary user persona for the MVP

Maintenance releases only, at this stage
Creates and tests both the agent and the metric
Technical and non-technical builders alike

Core needs

Evaluate quality criteria specific to their business

Create a metric without writing a script

Test a metric before publishing it

Understand why a score came out the way it did

Reuse and share metrics instead of rebuilding them

Pain points

Metric needs are more nuanced than one built-in definition

The only workaround today is too technical for most builders

Judging the judge: no way to know whether to trust a score

Discoverability, and a workflow fragmented across surfaces

03What teams already build

The workaround was the requirements document

Teams had not waited for a feature. They had built their own metrics in script, and every one of them told us something about what the product needed to support. Sorting them produced four groups, and the fourth was the most useful of all.

Cloned metrics with a new rubric

Same intent as the built-in metric, rewritten for a use case and regraded. Completeness and faithfulness on a 1 to 5 scale, correctness as a five-point operational match, fluency at three or five points where output is text-heavy, and hallucination treated as a guardrail under 10%.

Deterministic checks in script

Where the answer is knowable, teams did not want a judge at all. One team built its own framework for scripted tool-calling checks. Another folded tool and input verification into its completeness prompt. Where the source of truth sat outside the platform, a judge prompt filled in.

Domain outcome metrics

The metrics leadership actually reads. Interaction-to-case deflection, expressed as sixty of a hundred interactions deflected. End-to-end accuracy as a semantic match, so "excellent" and "awesome" both pass. Whether a decision matched what the HR team would have done. Whether a field mapped correctly across tenant tables.

Asked for, impossible today

The clearest platform asks in the study. Was the agent's reasoning about which sub-agent to call correct, separately from whether it followed the plan. Did it follow instructions, split from whether it finished. Did the run exit through an error path. One metric per output. And component-level scoring, because 80% retrieval times 80% skill times 80% instructions is closer to 50% for a customer.

The fourth group is why this feature is not a text box. Every one of those asks is a criterion a team can describe in a sentence and no built-in metric can express.

04The turn

A design review that sent the work back to first principles

The first design surfaced every capability the platform could offer. In review, the critique was direct and it was right: this was platform-centric, not user-centric. It showed people what the system can do instead of guiding them to a metric that works. The precedent was uncomfortable and close to home. Agent Studio had shipped that way, and users built things that did not work.

Get it wrong and we enable people to fail at something they will then blame their agent for. Get it right and metric quality goes up while support load goes down. So the design stopped adding surface area and went looking for constraints worth enforcing.

Research agenda that followed

Observe. How do people create a custom metric today? Where do they start, what is the first action, and what separates the ones who succeed from the ones who get stuck?

Prescribe. Which practices predict a working metric, and can we build them in as requirements rather than optional helpers?

Interrogate. Are custom metrics even the right answer, or should the built-in metrics get better first?

The bar we set: two or three opinionated constraints, each backed by evidence for why it improves the outcome.

05The hard part

Adding a prompt box is easy. Making a judge a team will ship on is not.

Four properties decide whether anyone trusts an LLM judge. Each one is a design problem, and each one has a mechanism in the release rather than a warning in the documentation.

Difficulty	Why it breaks trust	The mechanism that answers it
Non-determinism	The same input can score differently across runs. A judge returning 0.9, then 0.2, then 0.85 on comparable input is worth nothing.	Temperature 0.1 by default, a locked output contract, and a results panel that accumulates runs so variance is visible before publishing.
Rubric drift	With no fixed rubric the model invents a new scoring framework every run, with different dimensions and scales. Scores stop being comparable.	The rubric is captured at creation. Dimensions and scale belong to the person who made the metric, not to the model.
Input-mapping fragility	If test inputs do not map to the prompt's context variables, the judge scores incomplete context and fails silently. No error, just wrong numbers.	Test inputs are derived from the prompt's own variables, one labelled field each. Never a single free-text box.
The language gap	Builders describe quality in domain terms. Judges need evaluator-precise criteria. That translation is where reliability is won or lost.	Guided elicitation turns plain-language anchors into structured criteria, and a prompt quality signal flags structural gaps before testing starts.

06Principles

Six constraints we chose to enforce

Not guidelines. Each one closes a specific failure we watched happen, and each one costs the user a degree of freedom on purpose.

Principle 01

Start complete, never blank

The full prompt is generated from the creation inputs, rubric included, before the user sees the editor. Refining a strong draft is a job a domain expert can do. Authoring a judge from an empty box is not.

Principle 02

Ask in a fixed order

Agent type first, because it decides which context variables exist. Then what the agent does, what to measure, a good anchor, a poor anchor, scoring type and levels, aggregation, name. Structured answers map cleanly to prompt sections.

Principle 03

Lock what the contract depends on

Role, output format, chain-of-thought and bias guardrails stay system-owned and visibly so. The post-processor needs valid JSON in a fixed shape. Users can see every locked line, and cannot break it.

Principle 04

Nothing publishes untested

Publish stays disabled until a test run is on record, and it cannot be bypassed. Edit the prompt afterwards and the gate resets, because a score only describes the prompt that produced it.

Principle 05

Show variance, never average it away

Every test appends a row instead of overwriting the last one. Each run is judged against its own expected response, and no cross-run average is computed, because averaging unrelated examples produces a number that represents nothing.

Principle 06

Every score carries its reason

A justification in plain language ships with every score, and a failed judge call shows as an error row rather than a zero. Users need to know what was missing, not just that the number was low.

07The architecture

Three dependent systems on a shared foundation

This is not one prompt editor. The systems are strictly sequential: the second is only as good as the first, and the third only as good as the second. A weak input is not caught automatically, so the quality signal and the test gate exist to catch it before it becomes a published score.

System 1

Input elicitation

Collect just enough structured knowledge to generate a reliable judge. Agent type, what the agent does, what to measure, good and poor anchors, scoring type and levels, aggregation.

Every input maps to a named prompt section.

System 2

Prompt generation and refinement

Assemble two feeds into one judge-ready prompt: what the user provided, and what the execution parser extracts from the real run. Then give them safe tools to improve it.

Locked zones, token-styled variables, edit with AI behind a diff.

System 3

Testing and trust calibration

Turn a prompt into the confidence to publish it. Structured inputs, real executions, accumulating results, an advisory consistency signal and a gate that cannot be talked around.

Trust is earned by watching the judge score things you recognise.

Foundation

Library, discovery and the always-on post-processor

Published metrics are versioned, carry real metadata, and are pre-selected for the agent they were built for. Behind the scenes a post-processor the user never sees maps judge output into the same result contract as the built-in metrics, so a custom score renders next to task completeness with no special casing downstream.

indicatorResponse ← { score, justification, status }

JourneyWhere a metric sits in the builder's day

Agent Studio

→

Define agent behaviour

→

Metric details

→

Test metric

→

Publish metric

Highlighted steps are the ones this release owns. Publishing is a gate, not a button.

Inside the metricThe loop the three systems have to carry

Metric list

→

Define metric

→

Choose a template

→

Metric prompt

→

Processors

→

Test on a real execution

→

Score comparison

Every branch had to survive one constraint: nothing publishes untested, and editing the prompt sends you back through the loop.

08The product

Screen by screen

Walking a resolution note completeness metric from an empty library to a published, calibrated judge. Select any screen to open it larger.

System 1 · Input elicitation

Step 1 of 3 modal: define the metric, select chat or voice scope, name it, describe what it evaluates — **Step 1 · Define the metric**Scope first, because it decides which context variables exist. The description field most determines whether the judge is any good, so it is asked for plainly and early.

Step 2 of 3 modal: searchable template list with a preview of the judge prompt a template will add — **Step 2 · Start from a proven pattern**Templates are seeded from judges that already exist but were never surfaced here. The prompt is shown before it is added, so nothing arrives unseen.

Step 3 of 3 modal: define output scoring with true or false, numeric and threshold formats, plus score categories with status and weight — **Step 3 · Define output scoring**Scoring is captured before generation, not after. Choosing the type here is what lets the rubric, the output schema and the aggregation agree with each other.

System 2 · Prompt generation and refinement

Creating your metric loading state shown while the judge prompt is generated — **Creating your metric**The one moment the product asks for patience. What lands on the other side is a complete prompt, which is the whole point of asking structured questions first.

Metric editor showing the steps to publish panel with three of five steps completed — **Steps to publish**Progress is stated as a count, not implied by a disabled button. The checklist names what is still required and links straight to it, so the gate teaches instead of blocking.

Prompt editor with an assistance panel open for editing a section of the judge prompt — **Editing with assistance, reversibly**Select a passage in an editable zone and refine it. Appending is safe. Replacing shows a diff first, because losing a manually tuned criterion silently is worse than a slower interaction.

Processors section explaining pre-processors that run before scoring and post-processors that run after — **Processors, clearly optional**Custom logic on the inputs before scoring or the results after. Marked optional and placed after the required work, so the people who need it find it and the people who do not are never asked.

Model settings section with provider and model selection and an explanation that these choices determine how the metric scores — **Model settings that explain their consequence**Temperature defaults low, structured output on, chain-of-thought off. Every default is a consistency decision, stated rather than hidden.

System 3 · Testing and trust calibration

Run test panel with agentic solution, version and execution selectors — **Test against something that really happened**Pick a solution, a version and an actual execution. When there are no logs yet the panel says so and tells you how to get one, rather than leaving an empty dropdown to interpret.

Test complete view showing the metric score, agentic solution output and setup details beside the prompt — **Score, output and setup in one view**The prompt stays on screen next to the result. Diagnosing a strange score means comparing the criteria and the output without losing your place.

Score comparison tab showing the user's scores match the metric 67% of the time, with a per-test table — **Calibration you can actually run**You score first, the judge scores second, and the panel reports how often you agreed. A lightweight trust check, not a claim of statistical agreement, and it works without a labelled dataset.

Test details panel showing a score and the full plain-language justification — **The justification is the product**A number tells a team to look. The justification tells them where, in plain language, per record.

Foundation · Library and versions

Evaluation metrics library listing custom LLM-based metrics with scope, created date, provider and status — **One library, four ways in**Recently edited, from ServiceNow, custom LLM-based and custom script-based. Each row carries scope, provider and status, because someone choosing a metric at run setup has to tell whether it fits their agent.

Version details panel with created by, created on, status, name and number fields — **Versions, because criteria change**Editing a published metric creates a new version, and runs record the version they used. A score is only interpretable against the exact criteria that produced it.

09A decision with consequences everywhere

Three scoring types, chosen before anything is generated

The industry has no settled answer here. One tool recommends categorical labels as easier to apply consistently, another supports five formats. What is not negotiable is when the choice happens: it determines rubric structure, aggregation options and the output schema, so it comes before generation, never after.

Numeric

A continuous scale for when partial credit is meaningful. Fixed anchors top and bottom, optional levels between, and the team picks how scores aggregate: average, percentage above a threshold, or strictest.

{ "score": 0.72, "justification": … }

Pass or fail

Binary, for correctness checks with one right answer. The user defines what passing and failing look like. Aggregation is fixed to percentage passing, so there is nothing to get wrong.

{ "score": true, "justification": … }

Custom labels

Named tiers such as successful, partially successful and unsuccessful. The team sets each category name, its score value and whether it counts as a pass. Aggregation reports percentage per category.

{ "score": "Successful", … }

Research was unambiguous on why this mattered: teams abandoned the built-in metrics partly because a binary result would not get a feature approved. Letting them choose the shape of the answer is not a preference setting. It is the reason they will use the metric at all.

10Sequencing

Build the smallest loop that works, then earn the right to add

The recommendation was to prove that a builder can define, validate and reuse one business-specific judge before adding dataset-backed calibration, tools, multi-criteria scoring or automated optimisation. Deferrals here are sequenced, not rejected.

1

Define a metric in plain language, with no scriptingMVP

Input-based creation, an auto-generated judge prompt, single-test calibration, the library and versioning. The one loop that has to work.

2

Validate a metric at scale against real datasets

DataKit integration, bulk testing and dataset-based calibration, once teams have metrics and data worth calibrating against.

3

Score complex, multi-part criteria

Multi-criteria metrics, weighted scoring, and building a metric conversationally rather than through a form.

4

Optimise judges automaticallyHorizon

Standardised context through OpenTelemetry traces so judges stop depending on one runtime, and automated prompt optimisation. In a walkthrough, an optimiser lifted a draft groundedness judge from 0.61 to 0.79 F1. It also needs labelled data, ground-truth examples and review of every prompt change, which is exactly what an MVP does not have.

Area	In the MVP	Deliberately not yet	Reason
Authoring	Structured inputs	Conversational creation	Reliability over ambiguity
Validation	Single-test calibration	Bulk judge-the-judge, annotations	Avoid a dataset dependency
Context	Execution parser variables	Tools framework, trace dependency	Avoid a platform blocker
Optimisation	Metric quality signals	Automated prompt optimisation	Requires labelled datasets
Complexity	Single criterion	Multi-criteria, multi-judge	Prove the baseline first

11How we will know

Three proofs, and the honest state of the numbers

Usability

A builder defines a business-specific metric without scripting or prompt-engineering expertise. Measured by input completion and how many people who start a metric publish one.

Trust

A builder tests judge behaviour before publishing or reusing it. Measured by whether every published metric has a recorded test. This one is definitional: the gate either works or it is a defect.

Operational fit

The metric runs inside AutoEval and maps to the same result contract as everything else. Measured by run completion and telemetry coverage.

Signals instrumented at launch

Tests run per session before a metric is published, as the clearest read on how much iteration trust actually takes

Whether a custom metric selected at run setup is new or reused, which tells us if the library is doing its job

Whether anyone expands a justification, which tells us if they engage past the number

Runs completed with at least one custom metric, the primary adoption signal

Where the risk still sits

Judge inconsistency could undermine trust before anyone builds any. A low temperature and a visible variance panel help; a baseline of real variance still needs an engineering spike.

Reliability depends entirely on input quality, and input quality cannot be validated automatically.

Prompt generation runs on a small model today. Generating a complete, context-aware judge may need a larger one.

Execution context is built against today's parser. Standardising on traces later is a real migration.

The numeric targets behind these proofs are product proposals, not engineering commitments. They are a starting point for the kickoff discussion, and saying so out loud is part of the work.

In one line

A team should be able to say what good looks like for their agent in their own words, watch a judge apply it to something that really happened, disagree with it, and only then let it score anyone else's work.

That sentence is the constraint every screen here was tested against.

Product design: Agentic AI
Research: UX Research, Agentic AI
Product and engineering: AI Evaluations

Internal material. This page draws on confidential product requirements, unreleased roadmap sequencing and research quotes from named internal teams. Redact the roadmap, quality bars and participant quotes before sharing it outside ServiceNow.

Define what good means.Then trust the score.