All work

Teaching evaluation to tell right from finished

Agent evaluation could prove an agent finished its task, not that its answer was right. I designed how AI practitioners create ground truth, the answer key every score is checked against: an agent drafts it, a person verifies it, and only then does anything depend on it.

Role
Product designer, end to end
Product
Ground truth in Now Assist Skill Kit
Timeline
Q4 2025 to Q4 2026
Status
Live internally; customer flow in design
View as
Start from the agentIts datasets, and how much of each a person has verified, are already on screen.

How the work unfolded

Five stages. Each one left behind something the next was built on.

  1. 01

    Understand

    Why the internal version couldn't ship

  2. 02

    Define

    Who curates, and at what moment

  3. 03

    Decide

    Manual or automated curation

  4. 04

    Design

    Structure first, then screens

What I owned

  • Framing. Turned "manual or automated?" into a decision about trust, and wrote the principles the flow follows.
  • Structure. Placed curation in the practitioner's journey and defined the IA and flow, including the one step nobody can skip.
  • Interaction and UI. The home, the entry point and the three-step curation flow: 13 states, end to end.
  • Failure cases. Records that never ran, bad runs, version drift, review that outlasts one sitting, values the machine guessed.
  • Alongside. Platform on the standalone data model; data science on the curation agent.

01Understand

Finished isn't the same as right

Our metrics checked that a task completed, which tools ran and the path the agent took. None of that says whether the outcome was correct, because there was nothing to compare it against.

Agent runResolve incident INC0001
What gets checked
What the agent did
Ground truth expects
Result
Task completion
Completed
CompletedNothing to compare
PassPass
Tools calledtool choice
Get incident details, then Apply resolution
Get incident details, then Apply resolutionNothing to compare
PassPass
Record fieldpriority, exact match
"Low"
"High"Nothing to compare
Not checkedFail
Outcome
Resolved, no work notes
Priority raised, work notes with the root causeNothing to compare
Not checkedFail
Verdict CompletedCompleted, but not correct Illustrative run

Teams were already doing it by hand

Checked by hand. Business units compared agent output against ground truth manually, for their top customers.

Built one-off. Implementation teams had written ground truth generators for single enterprise customers.

First customers named. Leadership review confirmed aviation and banking accounts would use it first.

It worked internally. Customers couldn't touch it.

Three ground truth metrics went live for internal users in Q4 2025. A script wrote the records; people edited them as raw JSON.

List view, ground truth records

NumberPayload
GT0001{"expected_outcome": "Priority raised to High, work no
GT0002{"tools": [{"name": "Get incident details", "inp
GT0003{"fields": {"priority": "High", "state": "In Prog

Form view

Payload

{
  "expected_outcome": "…",
  "tools": [ … ],
  "fields": { … }
}

Written by a script, from a run that already happened

Edited by hand, as raw JSON

To reach customers, three things had to change

  • AuthoringDesign led, meA flow to select, generate, review and edit. No list views, no raw JSON.
  • Data modelPlatformA standalone table: use case, source record, payload. Nothing that leans on internal tables.
  • CurationData scienceAn agent instead of a script, so new records aren't stranded and failed runs aren't copied in as answers.

02Define

A practitioner, not a labeller

The person who needs ground truth builds agents for a living. If authoring feels like a second job, they'll skip it, so it had to fit work they were already doing.

Sam

AI practitioner

Builds agents and skills, creates datasets, iterates. Ground truth is one task among many.

Not a dedicated data labeller

Needs to

  • Know the answer is right, not just finished
  • Check tools ran with the right inputs, in order
  • See where the agent diverges, without reading logs
  • Iterate without re-authoring ground truth

Designing against

  • One reference spans tool calls, decisions and outputs
  • The work crosses AI Agent Studio, Skill Kit and Data Kit
  • Runs come from execution logs and AutoChat
  • Some records have never been run

Curation has a right moment

After the agent roughly works, before anyone trusts a number.

  1. 1Build and testSingle-record tests until there's a stable baseline.
  2. 2Create and review ground truthAutogenerate, review, save. Review can wait for later.This project
  3. 3Evaluate and iterateGround truth metrics beside reference-free ones, scored per record.

A run looks right? Nudge to capture it as ground truth.

Back here only if the use case changes

Improve, re-run, reuse the same ground truth

Authoring is paid once per use case. That's what makes a required review affordable.

03Decide

It looked like a cost question. It was a trust question.

Manual curation is accurate but slow. Automated curation is fast but leans toward the model that wrote it. Picking one meant giving up either scale or trust.

Trust in the reference Speed and scale Slow, and still unsure Manual Automated Hybrid
  • ManualNuanced and accurate on edge cases. Slow, costly, open to human bias.
  • AutomatedFast and varied. Biased toward its own model, with no view of its quality.
  • HybridAn agent drafts at speed; a person verifies before anything depends on it.

What settled it

Low scores get blamed on the agent. Look closer and the ground truth is often what was wrong. If the reference is wrong, every score built on it is a confident lie.

So: automate for speed, and make review a step nobody can skip.

  1. Agent
  2. Records
  3. Autogenerate
  4. Review and save

Six rules the flow follows

Each traces back to the problem analysis or the leadership review. The screens cite them by number.

  1. 1Autogenerate, never auto-trustGeneration assists. Verification is human and required.
  2. 2One destination, not threeNavigation and the primary action hold position across every step.
  3. 3Empty isn't brokenA state you haven't reached names the next action. Errors are for errors.
  4. 4Name things the way a person wouldPre-filled, editable names, because datasets get found again months later.
  5. 5Show the seam between machine and humanGenerated and verified are separate progress, so it's clear what still needs judgement.
  6. 6Make ground truth traceableEvery record carries where it came from, so a surprising score can be traced back.

04Design

Structure before screens

Curation touches three tools, so the real risk was getting lost. Everything lives under one destination, and the flow has a single gate.

Information architecture

Orientation Select data Generate Review Evaluate

Ground truth homeHow to navigate, overall analytics
Agents and workflowsSearch, filter, dataset count
Agent detailAnalytics, datasets by status
Curate ground truthAgent or workflow, then version
1 · Data selectionWhat to build from
2 · AutogenerateWhat the answer covers
3 · ReviewVerify before it counts
Dataset configurationInput type, pre-filled name
Runs or table recordsSuggested set, selection
Record detailsUtterance, context, execution log
Coverage analysisNormal, negative, corner cases
AspectsOutcome, fields, tool outputs, sequence
Start curationLocks the selections
Review progressGenerated and verified, apart
Records with statusGenerating, pending, reviewed
Record detailsOutcome, tools, fields, provenance
Evaluation setupGround truth as a named step, with reference-free metrics

User flow

One fork for records with no history, one gate before anything is trusted, and a loop that reuses ground truth instead of rebuilding it.

1 · Data selection 2 · Autogenerate 3 · Review has runs never run yes Pick agentand its version runs? Execution logssuggested set pre-selected Table recordsutterance autogenerated Choose aspectswhat it covers Confirmselections lock Agent draftsevery record Revieweach record all? Verifieddataset saved EvaluateGT + reference-free Improve the agentsame ground truth not yet: exit, verify later re-run evaluation, reuse ground truth only if the use case changes: curate again the step nobody can skip

Three steps, one destination

The breadcrumb, the stepper and the primary action stay put on every step, so moving forward never feels like switching products.

    A wrong reference fails quietly

    It doesn't throw an error. It just makes every score built on it wrong. These are the moments I designed for, shown where they're handled on the real screens.

    Build from table records instead. Utterance and context are autogenerated, then copied to every record, so new records aren't stranded.

    Check the run before it's promoted. Its plan, conversation and status are one click away, and suggested sets pre-select suitable runs.

    Version is required up front. A dataset stays tied to the agent version it was authored for.

    Progress is split, and nothing is lost. Generated and verified are tracked apart, every record keeps its status, and you can exit and verify later.

    Guesses are flagged. AI-suggested values carry a tag, and each field sets its own match: exact, semantic or contains.

    The reference can be traced. Every record keeps its execution plan, conversation and source, so the score can be checked and argued.

    05Validate

    Live internally, earning its way to customers

    Three ground truth metrics already run for internal users. Reaching customers takes a standalone data model, a curation agent, and testing this flow with the people it's for.

    The curation agent reads

    Agent definition Base record context Similar past records Prior execution outputoptional

    Drafts

    Curation agent

    Ground truth aspects

    Expected outcome Record field updates Tool outputs Tool sequence

    Can't be skipped

    A person reviews and verifies

    Stored as

    A standalone ground truth tableUse case, source record, payload. No internal tables.

    Metrics it unlocks

    Overall outcome correctnessLive Record field correctnessLive Tool correctnessLive Tool sequence correctnessLater
    Ground truth metrics sit beside reference-free ones in the same selection step: one shows where an agent struggles, the other whether it got the outcome right.

    The honest baseline

    Page views over the last three full months, before customer release. They count arrivals, not authoring: no ground truth actions are recorded yet.

    Review pageProduction22 views, 1 user, 1 customer
    Review pageSub-production1010 views, 2 users, 2 customers
    Dataset homeProduction3838 views, 12 users
    Dataset homeSub-production263263 views, 62 users

    One production user reached the review page, in August. This is the number the customer experience has to move.

    Road to customers

    1. Q4 2025Shipped to internal usersThree metrics. A script wrote records; list and form views edited them.
    2. Q2 2026Scoped for customersThree experience gaps named; agents explored for curation.
    3. Q3 2026 In designThis flowBuilt on Glide, on a standalone data model, mapped as a named step in evaluation setup.
    4. Q4 2026AI-native platformData Kit inside Skill Kit; versioning and peer review under consideration.

    Next:Heuristic evaluationInternal usability testsCustomer research

    Still open

    Questions the next round of testing and review has to answer.

    • Does ground truth need versions?Rollback and audit, against storage and interface weight.
    • Where should curation live?Data Kit centralises datasets; evaluation saves a trip. Leadership leaned toward one studio.
    • Should these scores feed optimisation?A comparison may be too thin to suggest a fix.
    • How do generated runs stay predictable?An AutoChat run can take a route a good reference never expected.
    • Does conversational ground truth fit?It has its own idea of an expected outcome.
    • Can production become the authoring surface?Admins already judge output while running agents.

    What I'd carry forward

    1. The hard part wasn't the payloadI expected the challenge to be a complex JSON structure. It was sequencing, naming and orientation; the structure got easy once people stopped getting lost.
    2. Trust is a UI problem tooA score is only as good as its reference, so where each reference came from has to be on screen. Here, provenance is the point, not a power-user extra.
    3. Test before addingNew features on an unvalidated flow inherit its problems. The order is fixed: a heuristic pass, then real users, then the next features.

    Evaluation is only as right as its answer key. This flow lets an agent draft the key, and makes sure a person signs it before anything is graded against it.

    © 2026 ServiceNow, Inc. All rights reserved. All work