Product design case study · AI platform
Teaching evaluation to tell right from finished
Agentic evaluation could already tell you an agent completed its task. It could not tell you whether the answer was correct. Ground truth based evaluation closes that gap, and the design problem was turning a structurally complex artefact into something a practitioner would actually author.
The short version
Ground truth is a trusted reference dataset of expected outcomes, used to check whether an AI agent produced the right result. It answers the single most requested thing enterprise customers asked for: did the agent solve the task correctly.
Reference free metrics tell you where and why an agent may be underperforming. Ground truth metrics confirm whether the desired outcome was achieved. Neither replaces the other, so the product had to make that pairing legible instead of forcing a choice.
01 · Problem space
Execution success is not correctness
The existing framework measured task completion, tool usage and trajectory logs. None of that validates the semantic quality of the final output. Without an expected outcome to compare against, a customer cannot verify what actually matters.
Unverifiable answers
Nothing confirmed whether the agent's answer was correct or complete, only that it finished.
Invisible trajectory
No out of the box way to check whether tools ran in the right order, with the right inputs, or in the fewest steps.
Manual log archaeology
Spotting redundant or unnecessary steps meant opening logs record by record. It did not scale past a handful of cases.
Challenges as documented
- Existing metrics verify task execution and tool choice accuracy, not whether the final output is factually or semantically accurate.
- No out of the box way to assess whether the agentic workflow executed optimally, for example in the fewest steps.
- Trajectory analysis is only reachable through custom evaluation metrics.
- Users must inspect logs manually for every record to find wasted steps.
- With no defined ground truth, there is nothing to compare agent output against.
How we knew it was real
- Multiple business units were already comparing agent outputs to ground truth by hand for their top customers, specifically to validate outcome rather than execution path.
- Implementation teams had built one off ground truth generators and evaluators for individual enterprise customer use cases.
- Leadership review confirmed named aviation and banking accounts as the customers who would use this first.
When three teams independently build the same missing capability, the gap is not a nice to have. It is a product.
02 · Framing
Why reference free metrics still matter
An early temptation was to treat ground truth as the successor to reference free judges. It is not. Ground truth is expensive to author and gates on data readiness, so pushing everything through it would have made evaluation slower rather than better.
Early iteration
Teams can assess execution health, tool usage patterns, latency and failure modes without blocking on ground truth readiness.
Execution behaviour
Task completion, tool invocation, retries and errors describe reliability even when correctness cannot yet be judged.
Regression detection
Reference free metrics apply broadly across executions, so they can monitor trends and surface systemic issues at scale.
Ground truth metrics sit alongside reference free metrics in the same selection step, each with its own description, rather than behind a separate mode. The user composes a signal, they do not pick a camp.
03 · Users and requirements
One persona, three tools, one live handoff
The target user is an AI practitioner: someone who builds agents and skills, creates datasets, and iterates. Not a dedicated data labeller. That distinction shaped almost every decision, because a practitioner will abandon authoring work that feels like a second job.
Jobs to be done
- Verify that the agent's output is correct and complete, not just finished.
- Check that tools were executed with the right inputs, in the right order.
- See exactly where the agent diverges from expected behaviour.
- Do all of this without reading execution logs one record at a time.
- Iterate on the agent repeatedly without re-authoring ground truth each time.
Constraints I designed against
- Ground truth for agentic execution is structurally complex: an ideal sequence of tool calls, intermediate decisions and expected outputs across a multi step workflow.
- The journey spans AI Agent Studio, Skill Kit and Data Kit, with a live handoff between the last two.
- It must work with both existing execution logs and AutoChat generated runs.
- It must be cold start safe, producing valid ground truth when no execution history exists.
Three obstacles between internal use and a shippable product
Ground truth evaluation went live for internal use in Q4 2025. Getting it to customers required more than polish.
| Area | What had to change | Why it blocked GA |
|---|---|---|
| Authoring Design led |
A dedicated customer facing experience for creating and editing ground truth, supporting per record authoring, inline editing, and clear mapping between a use case, a source record and its payload. Integrated into evaluation setup as a named step. | No customer accessible interface existed. Internal teams used list and form views that were never going to ship. Without authoring, ground truth metrics could not be configured at all. |
| Data model Platform |
Redesign ground truth as a standalone table holding only what it needs: use case identifier, source record identifier and payload. Remove every dependency on the internal Artifact and Autochat tables. | Those tables are internal constructs that may not exist in a customer instance, which made every ground truth record dependent on infrastructure customers would not have. |
| Curation Data science |
Replace script only generation with an agent that synthesises ideal path ground truth from four inputs: the agent definition, base record context, similar past records, and optionally prior execution output. | The script needed a prior execution to work from, so net new records were stranded. It also had no way to tell a good run from a bad one, risking failures being encoded as expected behaviour. |
04 · The pivotal decision
Hybrid curation, and a review step nobody can skip
The choice between manual and automated ground truth creation looked like a cost question. Analysis showed it was a trust question.
Manual curation
Strengths
- Captures nuanced human judgement, essential for subjective qualities such as tone and empathy.
- High accuracy for high stakes edge cases.
Costs
- Slow, expensive, poorly scalable, and exposed to human bias.
Automated curation
Strengths
- Rapid generation, at volume and with variety.
Costs
- Bias toward the generating model, needing rigorous fidelity and utility checks.
- No visibility into how well the automated method is actually performing.
A recurring pattern in evaluation is that low scores get attributed to agent failure. Detailed analysis often shows the poor result was caused by flaws in the ground truth itself. If the reference is wrong, every score built on it is a confident lie.
Relying exclusively on either method is inefficient at enterprise scale. Automate for speed, then make review a first class step in the flow rather than an optional prerequisite. The user validates the quality of the generated ground truth and edits it before anything downstream depends on it.
Select the agent
Choose the agent or agentic workflow to evaluate, and its version.
Select the records
Pick execution logs, or a source table and records for cold start cases.
Autocurate
The ground truth agent generates expected outcomes for every selected record.
Review and save
The user reviews, edits and verifies. This step cannot be skipped.
05 · Principles
Six rules the design had to obey
Each one is traceable to either the problem space analysis or the leadership review. Together they are the reason the flow looks the way it does.
Autogenerate, never auto-trust
Generation is assistive. Verification is human and mandatory, because a bad score usually means bad ground truth rather than a bad agent.
One destination, not three
Navigation, breadcrumb and primary action hold position across every step. A step change should never feel like a product change.
Empty is not broken
A state the user has not reached yet gets an empty state that names the next action. Errors are reserved for actual errors.
Name things the way a person would
Pre-filled, editable, human readable names over system generated identifiers, because these datasets get found again months later.
Show the seam between machine and human
Autogenerated and verified are tracked as separate progress, so the user always knows what still needs their judgement.
Make ground truth traceable
Every record carries its provenance, so a surprising metric score can be traced back to how its reference was made.
06 · The journey
How a practitioner gets from agent to evidence
Ground truth creation only makes sense at a specific moment: after the agent roughly works, before anyone trusts a number. Placing it correctly in the arc mattered as much as designing the screens.
Agent creation and manual testing
- The practitioner builds the agent or agentic workflow.
- Single record manual testing confirms it behaves roughly as expected.
- Configuration is iterated until there is a stable baseline.
- When an execution looks correct, the user is nudged to capture ground truth from it.
Ground truth creation and review
- With a stable baseline in place, the curation flow is launched.
- The ground truth agent autogenerates expected outcomes for the selected records.
- The user reviews and edits before saving.
- Review can happen asynchronously, since datasets are large and attention is not.
Evaluation and iteration
- Evaluation runs with ground truth metrics plus applicable reference free metrics.
- Per record scores are reviewed in the evaluation report.
- The agent is improved and re-run.
- Ground truth is not recreated unless the use case materially changes.
That last line is the whole economic argument. Authoring cost is paid once per use case, and every subsequent iteration reuses it. Designing for reuse is what makes a mandatory review step affordable.
07 · The experience
Features, screen by screen
Thirteen states from the design walkthrough, grouped by where they sit in the flow. Select any screen to view it full size.
Home and orientation
A landing surface that answers three questions before the user commits to anything: what is ground truth, how do I curate it, and what already exists.
Entry point
One decision, made explicit.
Data selection
Configure the dataset, then choose what to build ground truth from. Two input types, because half the problem was records with no execution history at all.
Comprehensive evaluation means covering normal flows, negative scenarios and corner cases. Surfacing coverage at selection time makes a thin dataset visible before the user spends review effort on it.
Autogenerate ground truth
Rather than asking the user to write a payload, this step asks which aspects of ground truth to generate. Each choice determines what gets reviewed next and which metrics can use the result.
Review ground truth
The step carrying the most design weight, because it is the one thing that cannot be skipped and the one thing nobody wants to do.
08 · What it unlocks
The metrics ground truth makes possible
Three metrics went live for internal use, with a fourth in development. The existing evaluation framework was already extensible enough to display their scores, so score visualisation needed no change.
Overall outcome correctness
Whether the final outcome matches the expected outcome for that record.
Tool correctness
Whether the agent called the tools the reference expects, with the right inputs.
Record field correctness
Whether the values written back to the record match the expected values, field by field.
Tool sequence correctness
Whether those tools ran in the expected order, without redundant or missing steps.
09 · Road to GA
From internal customers to customer release
Ground truth metrics with internal customers
Three ground truth metrics land for internal users. An automation script generates records by reviewing prior execution plans and emitting JSON. Editing happens through list and form views delivered as an update set.
Gaps named, curation rethought
Three experience gaps documented: no way to select records for ground truth, no way to review what was generated, no way to edit and save changes. Engineering explores agents for curation. Support added for standalone agents and existing execution records.
A purpose built curation experience
The flow in this case study, built on Glide. Ground truth decoupled into a standalone data model. Mapping integrated into evaluation setup as a named step. Heuristic evaluation and internal usability testing ahead of customer research.
Migration to the AI native platform
Move to an AI native surface, with versioning and peer review under consideration, and Data Kit APIs surfaced inside Skill Kit so datasets appear where evaluation happens instead of behind a tab switch.
10 · Still open
Questions yet to be answered
These shall be answered in the future.
Does ground truth need version support?
Versioning enables rollback from bad edits and improves auditing and trust. It also adds storage complexity and another layer in the interface. For an artefact authored once and reused, the case is not obvious.
Where should curation live?
Hosting in Data Kit centralises dataset management and reuses existing data workflows. Hosting inside the evaluation module reduces navigation friction. The leadership review pushed hard toward the second, and toward a single studio.
Should ground truth scores feed the optimisation pipeline?
These metrics compare execution against a reference, so the justification may be too thin to infer a suggested fix. Feeding weak signal into an insight pipeline risks producing confident, useless recommendations.
How do we keep generated runs predictable?
Ground truth comparison assumes a predictable execution path. Without enough context per record, an AutoChat run can take an unintended route, producing a misleading score against a perfectly good reference.
Does conversational ground truth fit this architecture?
Conversational evaluation has its own notion of expected outcome. Whether the two can share a model, or need to stay separate, is unresolved.
Can production become the authoring surface?
Admins already judge agent output while running agents in production. If that judgement could be captured in place, ground truth would stop being a separate authoring task and become a by-product of real work.
11 · Reflection
What I would carry forward
The hard part was not the payload
I expected the design problem to be representing a complex JSON structure. It turned out to be sequencing, naming and orientation. The structure became manageable once the flow stopped disorienting people.
Trust is a UI problem too
Ground truth exists to make evaluation trustworthy, which means the interface has to show where each reference came from. That history is not a power user feature here. It is the point.
Test with users before adding more
The strongest correction was to stop widening scope and go and watch people use what already exists. New features layered onto an unvalidated flow multiply the problems rather than offset them, and every one of them inherits the same orientation debt. So the sequence is now fixed: a heuristic pass, then testing with real users, then the next set of features, and each addition has to earn its place against what testing actually showed.