ServiceNow Agent Studio AI evaluation

Custom LLM metrics for AI agent evaluation

Our metrics could prove an agent called the right tool, not that its answer was any good. I designed a no-code way for builders to turn their own criteria into an LLM judge, and test it before it scores anything that matters.

View as

My role: Product designer
Worked with: UX research, product, AI Evaluations engineering
Surface: Agent Studio, AutoEval
Status: Prototype in review, Q4 2026 target, internal first

See where it landed

Where it landed

Describe what good looks like, get a complete judge, test it on something that really happened, and only then publish it.

No scriptingPlain-language answers generate the complete judge prompt, rubric included.
Tested before trustedPublish stays locked until a test is on record. Edit the prompt and it locks again.
Fits the pipelineCustom scores use the same result shape as built-in metrics, so they sit side by side.

Process

How I got there

Five phases, and what each one produced. Pick any of them to jump there.

The problem The solution Delivery

01Discover

Opened up: who stopped using our metrics, and why
02Define

Narrowed to one person and one hard problem
03Ideate

Opened up again: structure, flow and layout
04Design

Committed: three systems, one prototype
05Deliver

Cut the MVP and decided how we'd know

What I owned

Framing. Turned the research into a persona, a journey map and six design principles.
Workshops. Mapped the journey and the information architecture at the whiteboard with research, product and engineering.
Exploration. Drew both structural directions side by side and argued the trade-offs.
Interaction design. Designed and prototyped the end-to-end flow on desktop and mobile: creation, generation, testing and publishing.
Scope. Helped shape the MVP cut and the signals we'd measure it by.

The problem

Our metrics graded the process, not the outcome

They could confirm an agent chose and called the right tool. Customers wanted to know whether the answer was right.

An agent run, as our metrics saw it: they checked the steps they could verify, and passed a run that failed gracefully.

13%of agents in production had auto evals run on them

14%of agents with auto evals run made it to production

3 in 4teams customise our metrics or write their own

0of those paths worked without writing code

01Discover

Ten people who evaluate agents for a living

UX research interviewed six quality engineers and four managers, an hour each. They build the agents customers get out of the box: if our metrics failed them, they would fail customers too.

Quality engineer Manager 60-minute semi-structured interviews

7 business units, including HRSDITSMSPMSecOpsCRMPlatform

Five reasons teams stopped using the built-in metrics

Binary scoring

A right-or-wrong result won't get a feature approved. Teams rebuilt the same metric on a 1–5 rubric.

“If you tell me it is sixty percent good, I'll be more confident to push the feature out.”

Process, not outcome

Metrics checked whether the agent called the right tool. Customers ask what it achieved.

“All they want is output eval.”

No sense of context

One definition of completeness can't serve summarisation, security analysis and an HR workflow at once.

Graceful failures score as wins

A workflow that exits through its own error path still counts as a completed task.

“The agent has taken a failure route but I see that the metric says success”

Scores that flatter

Results landed above 90% while manual review said the experience was worse.

“None of the customers are interested in tool calling correctness or tool choice accuracy. All they want is output eval. They want to do evaluations with their own criteria and in the manner they want to do it.”

Research participant

The maturity model that reframed the ask

Teams sat at three levels, and each needed something different from us. The design had to serve all three.

Level 1 Foundational

Uses built-in metrics as they are, often unaware customising is possible.

Needs from usTo know a better fit exists

A library that shows what already exists

Level 2 Developing

Clones a built-in and rewrites the judge prompt for one use case. Rarely reuses it.

Needs from usA supported clone-and-edit path

Start from a template, edit it safely

Level 3 Advanced

Runs on outcome-based custom metrics; keeps built-ins for debugging steps.

Needs from usA library, versions and consistent scoring

Versions, and judges tested before publishing

02Define

A domain expert, not a prompt engineer

The person who knows what a good resolution note looks like is rarely the person who enjoys writing an evaluation script.

Sara

AI practitioner

Technical enough to build an agent, and specific enough to know exactly where it falls short.

Builds and tests both the agent and the metric
Works inside the customer's company, enabling its employees or customers

Needs

Judge quality on business-specific criteria
Create a metric without writing a script
Test a metric before publishing it
See why a score came out the way it did
Reuse and share metrics instead of rebuilding

Pain points

One built-in definition can't capture the nuance
The only workaround is too technical
No way to know whether to trust a score
Hard to find; the workflow is split across surfaces
No guidance on how to create a metric

Sara's path to a custom metric, today

The research findings laid out as a journey. Each low point lines up with the part of the design that answers it.

Team working at a whiteboard covered in journey mapping: persona needs, pain points, goals and a branching map of evaluation results — The journey mapping session, with research, product and engineering

Doing

1Spots a gapScores look great. Manual review disagrees.
2Looks for a metricBuilt-ins are binary and generic, with no guidance on making one.
3Builds a workaroundClones a metric and rewrites the prompt, or writes a script.
4Runs itGets a number, with no way to tell whether to trust it.
5Starts overThe metric is rarely reused. The next use case begins from zero.

Feeling

Where the design steps in

Criteria in their own words
Templates and a findable library
A complete judge, generated from answers
Test on real runs; compare with your own score
Versioned, shareable metrics

What the session settled. An evaluation result has to answer three things: what went wrong, where, and how to fix it. That branch of the map became the testing system.

A prompt box is easy. A judge a team will ship on is not.

Four things decide whether anyone trusts an LLM judge. Each one got a mechanism in the product, not a warning in the docs.

Non-determinism

A judge that returns 0.9, then 0.2, then 0.85 on comparable input is worth nothing.

Temperature 0.1 by default, a fixed output format, and results that pile up so the spread is visible before publishing.

Rubric drift

Without a fixed rubric the model invents a new scale every run, so scores stop being comparable.

The rubric is captured at creation. Dimensions and scale belong to the metric, not the model.

Input-mapping fragility

If test inputs don't map to the prompt's variables, the judge scores half the context and fails silently.

Variables come from the execution context. Each becomes its own labelled test field, never one free-text box.

The language gap

Builders describe quality in domain terms. Judges need evaluator-precise criteria. Reliability is won or lost in between.

Guided questions turn plain-language anchors into structured criteria, and a quality signal flags gaps before testing.

How might we

turn a domain expert's judgement into a judge they can trust, without making them a prompt engineer?

Six rules the design enforces

Each closes a failure we watched happen, and each costs the user a little freedom on purpose.

Start complete, never blankThe full judge prompt, rubric included, is generated before the editor opens.
Ask in a fixed orderAgent scope first, because it decides which context variables exist.
Lock what scoring depends onRole, output format and guardrails stay system-owned: visible, never breakable.
Nothing publishes untestedPublish unlocks after a recorded test. Edit the prompt and it locks again.
Show variance, never average it awayEvery test adds a row. No cross-run average hides the spread.
Every score carries its reasonA plain-language justification ships with every score. A failed call is an error, not a zero.

03Ideate

Principles on the wall first, layouts drawn against them

Anything that couldn't be defended by a line on the left of the board didn't leave the room.

Information architecture

One library, a short creation modal, and a single scrolling metric page that the test panel and version history open beside.

Designer at a whiteboard writing numbered principles beside sketched layout options for the metric page — The IA session: principles first, then layouts

Evaluation metricsThe library Recently editedFrom ServiceNowCustom LLM-basedCustom script-based

New metricA modal over the library 1 Define2 Template, optional3 Scoring

MetricOne scrolling page VersionSteps to publishRun testPublish

Version detailsSide sheet: details and activity

Sections, with a rail to move between them

Metric promptRequired

ProcessorsOptional

Scoring and output

Model settings

Metric settings

Test panelOpens alongside the prompt

Configurable variablesExecution context, additional inputs

Prompt editorRole and output format locked

Prompt edit with AIAppend or replace, with undo

Run testSolution, version, real execution

Current resultScore review, output, setup

Score comparisonYour score against the judge's

The loop every metric goes through

Three systems in strict sequence, and one rule that bends the line back on itself: change the prompt and you test again.

Two structural choices, drawn both ways before either was argued

Drawing each as a pair kept the conversation on the trade-off instead of on taste.

Decision 1How much should creating a metric lead you?

Layout study comparing a guided stepper with five numbered steps against a partial stepper shown as a create modal over the page — **Guided stepper**Teaches the model of a metric, but turns a two-minute task into onboarding and strands you far from the library. **Partial stepper**Just enough structure to collect what makes a good judge, in a modal over the library you started in.

Decision 2How should a metric be laid out once it exists?

Layout study comparing a single pane with menu, details and test columns against a scroll-through page with a menu rail — **Single pane**Everything side by side. It reads as powerful and behaves as cramped: the prompt gets a third of the screen. **Scroll through**One scrolling page with a section rail. The prompt keeps its width, and the test opens alongside it.

04Design

Three systems, each only as good as the one before it

A weak answer early becomes a wrong score later, so two checkpoints sit between the systems.

System 1Input elicitation
System 2Generation and refinement
Prompt quality signal
System 3Testing and trust
Publish gate
FoundationLibrary, versions and one result contract

System 1Input elicitation

Ask for judgement, not prompt craft

Three short steps, in a fixed order, in a modal over the library.

Scope and intent first

Agent scope decides which context variables exist. The description shapes the judge most, so it is asked plainly and early.

Step 1 of 3 modal: define the metric, select chat or voice scope, name it, describe what it evaluates

Start from a proven pattern

Templates come from judges that already existed but were never surfaced here. The prompt is previewable before it is added.

Step 2 of 3 modal: searchable template list with a preview of the judge prompt a template will add

Choose the shape of the answer

Scoring type, categories and a good and a poor example are captured before anything is generated, so rubric, schema and aggregation agree.

Step 3 of 3 modal: define output scoring with true or false, numeric and threshold formats, plus score categories with status and weight

Every answer has a home in the prompt

This is the translation a domain expert shouldn't have to do by hand. Select an answer to see where it lands.

What Sara answers

What the judge receives

Role and output format are set for you.1 of 4

# ROLE System-owned

You are an expert quality evaluator for IT service management resolution notes.

# GOAL

Evaluate whether the resolution note contains enough detail for a human agent to understand what happened without reopening the ticket.

# CONTEXT

Original incident: {{incident_description}}
Resolution note: {{resolution_note}}

# EVALUATION CRITERIA

Score against three required elements: 1. Action taken 2. Root cause 3. Next steps

# SCORING RUBRIC

1.0 — All three elements present with specific detail0.7 — Two of three · 0.4 — Only one0.0 — None present, or note is generic/vague

# OUTPUT FORMAT System-owned

{"score": <float between 0 and 1>, "justification": "…"}

Three shapes of answer, chosen before anything is generated

The choice sets the rubric, the aggregation and the output schema, so it can't come after.

Use it when: Partial credit is meaningful
Aggregate by: Average, % above a threshold, or strictest

{ "score": 0.72, "justification": "…" }

Use it when: There is one right answer
Aggregate by: % passing, fixed so nothing can be misread

{ "score": true, "justification": "…" }

Use it when: Tiers read better than numbers
Aggregate by: % in each category

{ "score": "Successful", "justification": "…" }

Teams dropped the built-ins partly because a binary result wouldn't get a feature approved. The shape of the answer is why they'll use the metric at all.

The one wait in the flow. What arrives is a complete judge, which is the point of asking structured questions first.

System 2Generation and refinement

Start complete, then refine safely

The judge arrives finished. Editing it is reversible, and the parts scoring depends on are locked.

Progress you can count

Steps to publish states what's done and what's left, and links to it, so the gate teaches instead of blocking.

Metric editor showing the steps to publish panel with three of five steps completed

Edit with assistance, reversibly

Select a passage, refine it with AI, then append or replace, with undo always there. The pattern is borrowed from another team.

Prompt editor with an assistance panel open for editing a section of the judge prompt

Processors section explaining pre-processors that run before scoring and post-processors that run after — **Processors, clearly optional**Custom logic before scoring or after it, placed after the required work so only the people who need it go looking.

Model settings section with provider and model selection and an explanation that these choices determine how the metric scores — **Settings that explain their consequence**Low temperature, structured output on. Every default is a consistency decision, stated rather than hidden.

System 3Testing and trust

Earn trust by watching it judge something real

Pick a real execution, read the score and its reason, give your own score, and see how often you agree.

Test on something that happened

Pick a solution, a version and an actual execution. Each prompt variable gets its own labelled input.

Run test panel with agentic solution, version and execution selectors

Score, output and setup in one view

The prompt stays beside the result, so diagnosing a strange score never loses your place.

Test complete view showing the metric score, agentic solution output and setup details beside the prompt

The justification is the product

A number says look. The justification says where. Then you give your own score.

Test details panel showing a score and the full plain-language justification

Calibration you can actually run

You score, the judge scores, and the panel reports how often you agree. A lightweight trust check that needs no labelled dataset.

Score comparison tab showing the user's scores match the metric 67% of the time, with a per-test table

FoundationLibrary and versions

A library that remembers which criteria made which score

Published metrics are versioned and pre-selected for the agent they were built for.

Evaluation metrics library listing custom LLM-based metrics with scope, created date, provider and status — **One library, four ways in**Scope, provider and status on every row, so you can tell at run setup whether a metric fits your agent.

Version details panel with created by, created on, status, name and number fields — **Versions, because criteria change**Editing a published metric creates a new version, and runs record the version they used.

{ score, justification, status }the same result shape as every built-in metric, so a custom score renders next to task completeness with no special casing

On a phoneThe same flow, every screen

Five rules decided how every screen fits a phone

Fifteen phone screens, grouped by the rule that shaped them.

Tables become cards Rows turn into cards that keep what people scan for.
LibraryEach row keeps status, scope and provider
Modals become full-screen steps A progress bar replaces the step title. Back and Continue stay at the bottom.
DefineProgress on top, actions at the bottom
TemplateOptional, so Skip is the main action
ScoringStill chosen before anything is generated
CreatingThe one wait, full screen
The section rail becomes tabs Save, Run test and Publish move into a bar at the bottom.
Metric promptSections become tabs; actions dock below
ProcessorsOne section per tab
Model settingsSave, Run test and Publish stay in reach
Popovers and side panels become bottom sheets Each one opens over the page it belongs to.
Steps to publishThe sheet names the next step
Prompt editThe passage being refined stays in view
Version detailsPlaceholders say what to write
Run testSearch runs, or tap a recent one
Side by side becomes its own page The one trade-off: prompt and result no longer share a screen. The test is one tap back.
Test resultThe test gets a page of its own
Setup detailsTabs replace side-by-side panels
Score comparisonIts table becomes cards too

04Design, edge cases

Designed for the moments it goes sideways

Six moments where a quieter design would have left people guessing, and what happens instead.

No runs to test against yet

The panel says so and tells you how to get one, instead of leaving an empty dropdown to interpret.

The prompt changed after testing

Publish locks again. A score only describes the prompt that produced it.

The judge and the builder disagree

That's the point of the check. You see how often you agree and adjust the scoring before anyone else relies on it.

Same kind of input, different scores

Every test appends a row instead of overwriting the last, and no cross-run average hides the spread.

The judge call fails

It shows as an error row, never as a zero, because a zero would read as a verdict.

The definition of good changes

Editing a published metric creates a new version, and every run records the version it used.

05Deliver

Build the smallest loop that works, then earn the right to add

First prove one builder can define, test and reuse one business-specific judge. Everything after that is sequenced, not rejected.

MVPDefine a metric in plain language, no scripting

Input-based creation, a generated judge prompt, single-test calibration, the library and versions.

NextValidate at scale against real datasets

DataKit integration, bulk testing and dataset-based calibration.

ThenScore complex, multi-part criteria

Multi-criteria metrics, weighted scoring, building a metric conversationally.

HorizonOptimise judges automatically

Trace-based context and automated prompt optimisation, once labelled data exists.

In the MVP, and deliberately not yet

Area	In the MVP	Not yet	Why
Authoring	Structured inputs	Conversational creation	Reliability over ambiguity
Validation	Single-test calibration	Bulk judge-the-judge, annotations	Avoid a dataset dependency
Context	Execution parser variables	Tools framework, trace dependency	Avoid a platform blocker
Optimisation	Metric quality signals	Automated prompt optimisation	Needs labelled datasets
Complexity	Single criterion	Multi-criteria, multi-judge	Prove the baseline first

A version already shipped, and almost nobody uses it

Custom LLM metrics already exist inside the Skill Kit evaluation experience. Three months of telemetry, June to August 2026, is the number this work has to move.

Sub-production users, June to August 2026

Opened evaluations1,435

Reached metric setup41

Started a custom metric6

Published a metric6

Under 3% of evaluation users reached the page where a metric is defined.

See the numbers as a table

Signal	Production	Sub-production
Metric setup page	6 views, 3 users, 3 customers	297 views, 41 users, 34 customers
Metric playground page	None	659 views, 26 users, 23 customers
Custom metric creation started	None	7 times, 6 users, 6 customers
Metric published	None	29 times, 6 users, 6 customers

1,435 sub-production users opened the evaluations page in the same period.

custom metrics started or published in production. Three customers opened metric setup there all quarter.

Sub-production, by month

Setup views

JunJulAug

Playground views

JunJulAug

Creations started

JunJulAug

Publishes

JunJulAug

Growth came from a few heavy users: one customer drove about 60% of playground views and 22 of the 29 publishes.

See the monthly numbers as a table

Month	Setup views	Playground views	Creations started	Publishes
Jun	85	82	0	0
Jul	103	237	2	11
Aug	109	340	5	18

How we'll know it worked

Usability

A builder defines a business-specific metric without scripting or prompt engineering.

Measured by input completion, and how many who start a metric publish one

Trust

A builder tests judge behaviour before publishing or reusing it.

Measured by every published metric having a recorded test. The gate works, or it's a defect.

Operational fit

The metric runs inside AutoEval and maps to the same result contract as everything else.

Measured by run completion and telemetry coverage

Instrumented at launch

Tests run before a metric is published: how much iteration trust actually takes
New or reused custom metric at run setup: whether the library does its job
Justifications expanded: whether people engage past the number
Runs completed with at least one custom metric: the primary adoption signal

Where the risk still sits

Judge inconsistency could undermine trust early. A real variance baseline still needs an engineering spike.
Reliability depends on input quality, which can't be validated automatically.
Generation runs on a small model today; a complete, context-aware judge may need a larger one.
Context is built on today's parser. Moving to traces later is a real migration.

The numeric targets behind these signals are product proposals, not engineering commitments: a starting point for kickoff, and saying so is part of the work.

Reflection

What I'd carry into the next one

Trust is a design material.The gate, the variance rows and the justifications do more for confidence than any explanation could.
Structure is kinder than a blank box.A domain expert can refine a strong draft. Nobody should have to author a judge from nothing.
Say “not yet”, with a reason.Sequencing the bigger ideas, instead of rejecting them, kept the MVP small without losing the roadmap.

A team should be able to say what good looks like for their agent in their own words, watch a judge apply it to something that really happened, disagree with it, and only then let it score anyone else's work.

That sentence is the test every screen here had to pass.

Product design: Agentic AI
Research: UX Research, Agentic AI
Product and engineering: AI Evaluations