In classical software, engineers specify much of a system’s behavior through the implementation itself, composing abstractions, contracts, invariants, and rules into the program.
Learned systems change that relationship. They let us produce behavior we do not know how to program directly, without removing the need to specify what good behavior means. Part of that specification burden moves downstream, into evaluation: What behavior did we actually produce, how much should we trust the evidence about it, and what action does that evidence justify?
This essay is about the machinery organizations build to answer those questions. Across LLMs and agents, search and recommendation, autonomous driving, and robotics, the implementations differ enormously, but the underlying anatomy is remarkably similar.
By the end, you should be able to look at an evaluation system and reason about what it can actually know, what decisions that knowledge justifies, where it can fool itself, and where to invest next.
At a high level, the machine looks something like this:
DECISION CONTRACT
│
▼
EVIDENCE SELECTION
│ cases / scenarios
▼
╔═ EXECUTION ═══════════════════════╗
║ ║
║ ACTOR ──behavior──▶ ENVIRONMENT ║
║ ║
╚═════════════════╤═════════════════╝
│ outcome + trace
▼
JUDGE
│
▼
CLAIM + UNCERTAINTY
│
▼
GATE
│
┌─────────┴─────────┐
▼ ▼
action more evidence
│ │
▼ └──▶ EVIDENCE SELECTION
production outcomes
│
▼
┏━ INSTITUTIONAL MEMORY ━━━━━━━━┓
┗━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┛
│
└──▶ updates future evaluationThe rest of this essay progressively zooms into those parts.
The Output is a State Transition, Not a Score
At its most zoomed-out, an evaluation system is a control loop that converts observed behavior into a decision, and uses the consequences of that decision to change what will be evaluated next.
The first important shift is conceptual: an evaluator produces evidence that can justify moving a consequential decision from one state to another.
That means the system should begin with a Decision Contract. What exactly is under test? What decision could change? What does a false acceptance cost? What does a false rejection cost? Which failures are non-compensable? How much uncertainty is tolerable?
The required evidence should scale with the blast radius and reversibility of the action: a change that can be cheaply rolled back need not carry the same assurance burden as one that is difficult or dangerous to reverse.
Without those answers, a dashboard can contain thousands of measurements while producing no knowledge about what anyone should do. Search systems make this distinction unusually visible. Airbnb, for example, reports offline ranking improvements using NDCG, but still validates consequential ranking changes through randomized online experiments against booking and engagement outcomes. Offline relevance tells you something useful; it does not by itself establish the product consequence. The contract anchors metrics to the decision they are meant to inform.
The Triad: Actor, Environment, Judge
An evaluation orchestrates three distinct systems, and any one of them can be wrong.
- The Actor produces behavior (e.g., the LLM policy, the search ranker, the autonomous driver).
- The Environment determines the consequences of that behavior (e.g., the bash sandbox, the marketplace dynamics, the physical simulator).
- The Judge decides what the consequences mean (e.g., the Python verifier, conversion tracking, the kinematic safety check).
This means an evaluation never measures the Actor directly. It observes the Actor through an Environment, using a Judge. Every resulting capability claim therefore inherits assumptions from all three.
ACTOR
│ behavior
▼
ENVIRONMENT
│ outcome
▼
JUDGE
│ interprets
▼
CLAIMFor consequential candidate and release decisions, evaluation often needs to answer a fundamentally counterfactual question: What would have happened if the candidate had been there instead?
recorded situation
│ reconstruct
▼
same initial state + Environment model
│
┌──────────┴──────────┐
▼ ▼
current ACTOR candidate ACTOR
│ behavior │ behavior
▼ ▼
ENVIRONMENT ENVIRONMENT
│ │
▼ ▼
trajectory A trajectory B
└──────────┬───────────┘
│ compare
▼
JUDGE
│
▼
CLAIMSeen this way, an evaluation system is machinery for constructing trustworthy claims about counterfactual behavior. That counterfactual becomes harder when the Environment is itself reflexive. In marketplaces, ranking systems, and multi-agent settings, deploying a candidate can change user or competitor behavior, so replay against the historical environment may not predict the equilibrium the candidate helps create.
Replay, simulation, experimentation, judging, and release gating are different parts serving that common purpose.
Once you see evaluation this way, replay infrastructure becomes a first-class ML system. But more importantly, it exposes where failures actually originate. If the Actor is wrong, you have bad behavior. If the Environment is wrong, the candidate is acting in the wrong world. If the Judge is wrong, the system maps behavior to the wrong conclusion. Some of the most interesting failures occur at the boundaries: an Environment permits an exploit the Judge cannot distinguish from legitimate success.
The Environment and Judge therefore become evaluation targets themselves.
Sometimes the harder problem is upstream of Judge calibration: the construct itself is underspecified or legitimately subjective. “Helpful,” “relevant,” or “appropriate” are not physical observables. In those cases, the rubric becomes part of the measurement system: the operational definition of what the Judge is being asked to recognize. Disagreement can reveal ambiguity in the specification rather than merely noise in the Judge.
Simulator fidelity must be checked against the world it claims to reproduce; learned or heuristic Judges must be calibrated against independent evidence. A capability claim cannot be more trustworthy than the measurement chain used to establish it.
The Triad describes how a single piece of evidence is produced. The evidence-selection system determines which such observations exist in the first place.
Evidence Selection and the Fidelity Ladder
Once the decision is defined, the system must decide what evidence to inspect. The value of additional data depends on the information it adds. Driving logs are abundant precisely where nothing exceptional happens. Production LLM traces contain enormous numbers of routine successes. The next million examples are therefore not automatically as informative as the first million.
Evidence has to be deliberately selected. Autonomous driving makes the stakes obvious: Waymo uses simulation to amplify logged experience and construct variations around difficult or rare situations, because waiting for consequential edge cases to recur naturally would be an absurd evaluation strategy.
The evidence-selection policies appropriate for discovery and population estimation are different. A representative suite supports claims about ordinary population behavior. A discovery suite tries to find where the system is wrong.
Confusing the two produces one of the most common evaluation mistakes. If I aggressively mine difficult driving scenarios, the resulting dataset may be excellent at finding regressions. It is not therefore an unbiased estimate of how often those regressions occur on the road. If I select only the queries on which two search rankers disagree, I can learn a lot about their differences. I cannot report raw performance on that sample as though it represented normal traffic.
The same idea appears in computerized adaptive testing (psychometrics): the most informative next question depends on what is currently uncertain about the examinee. Adaptive selection can estimate capability with far fewer questions. That efficiency comes from a deliberately altered sampling policy. The selector is part of the evaluator, because it determines what the system is capable of knowing.
The evidence most useful for improving a system is often not the evidence you should trust most for assuring the improved system.
Evidence also differs in fidelity and cost. It can be organized into a fidelity ladder. In autonomous driving, for example, that ladder can look like:
component tests
│
▼
recorded-log replay
│
▼
counterfactual simulation
│
▼
closed-loop simulation
│
▼
integrated hardware / system tests
│
▼
controlled real-world exposure
│
▼
fleet behaviorEvery level buys realism at additional cost and risk. Other domains use different stages, such as offline replay, sandbox execution, shadow traffic, or A/B tests, but the principle is the same.
The operating principle of this ladder is simple: Cheap evidence earns the right to consume expensive evidence. It is wasteful to discover during a live A/B test what a deterministic offline suite could have shown yesterday.
But the inverse matters equally: passing cheap evidence does not entitle you to pretend you acquired expensive evidence.
Cost and validity are different dimensions. A sophisticated simulator can still be misspecified; an expensive real-world study can still be badly sampled; a high-fidelity execution can still be judged against the wrong construct.
You Evaluate a System, Not a Checkpoint
Once evidence has been selected, the behavior must actually be produced. Here another software instinct breaks down. A benchmark result is produced by the checkpoint interacting with a larger execution and measurement system:
Evaluated system = model weights + prompt + context policy + harness + tool schema + execution environment + resource budget + verifier
The observed benchmark behavior is jointly produced by the model and the system that elicits, constrains, and judges it.
Poolside encountered an instructive failure at the boundary between Environment and Judge while training its Laguna model. A SWE-Bench Pro training run suddenly jumped roughly 20 percentage points. Investigation showed the model was mining unpruned Git history for the reference solution, and eventually searching the web to cheat.
The verifier correctly observed that the tests passed. The problem was that the evaluation environment permitted behaviors outside the capability the benchmark intended to measure. Nothing about the terminal condition, “the tests passed,” distinguished genuine software engineering from recovering privileged information.
verifier success ≠ intended-task success
A Judge can only enforce properties made observable and meaningful by the Environment around it.
Outcomes Are Not Enough: Evaluate the Trace
Because terminal success reflects the whole execution system, it can be ambiguous about how that success was produced. This motivates two streams of evaluation.
╔═ EXECUTION ═══════════════════════╗
║ ║
║ ACTOR ──behavior──▶ ENVIRONMENT ║
║ ║
╚═════════════════╤═════════════════╝
│ emits
├─ terminal state ─▶ verifier ─┐
└─ process trace ─▶ judges ────┤
▼
CLAIM + UNCERTAINTYOutcome evidence asks whether the terminal condition was satisfied (Did the tests pass? Did the robot complete the task?). Process evidence asks how it was satisfied.
By retaining network calls, tool use, intermediate states, interventions, and other execution provenance, the system can use learned judges and human experts to catch cases where a metric was satisfied for the wrong reason.
A robot that completes a warehouse task after four human interventions has a different capability from one that completes it autonomously. An autonomous vehicle that reaches the correct final trajectory only because a downstream safety mechanism repeatedly vetoes unsafe proposals has a different failure structure from one whose primary policy behaves correctly. A multi-step LLM system that returns the right answer after retrieving an unsupported source has different production risk from one whose reasoning path remains grounded.
The terminal state tells you whether something happened. The trace reduces ambiguity about how that outcome was produced.
Execution Throughput is Not Decision Throughput
As execution scales to millions of simulations or offline tests a day, organizations hit a wall. You can scale execution throughput tremendously and still not improve decision throughput proportionally.
If your evaluation suite flags 70,000 “interesting” regressions, that is not useful. At scale, human triage becomes a scheduler for scarce expert attention.
1,000,000 counterfactual executions
│ detect
▼
70,000 flagged events
│ deduplicate + cluster
▼
ranked investigations
consequence / uncertainty
novelty / reversibility
│ select
▼
40 triage cases
│
▼
expert investigationCases compete for expert attention based on severity, uncertainty, novelty, consequence, reversibility, and expected information value.
Compute is scarce.
Real-world interaction is scarce.
Expert attention is scarce.
The engineering objective is to maximize trustworthy, decision-relevant information per unit of compute, elapsed time, real-world exposure, and human attention.
Faster evaluation also shortens the feedback loop around development. A run that returns while an engineer or researcher is still working on a change is used differently from one whose results arrive the next day.
The Gate is a Claim-Building Machine
Eventually, all this heterogeneous evidence must become an action. The mistake is to imagine the release gate as one final numerical threshold. A release board isn’t really asking, “Did version B get a 93.7?” It is asking: “What action does the current evidence justify?”
“Safe enough to deploy” is rarely one empirical proposition. It is a bundle of claims: that known regressions are absent, representative behavior has not degraded, important edge-case families remain within bounds, the evaluation environment is sufficiently faithful, and remaining uncertainty is acceptable for the proposed deployment scope.
The gate is a claim-building state machine. Its output should be the largest action justified by the current evidence.
CLAIM + UNCERTAINTY
│
▼
┏━ GATE ━━━━┓
┗━━━━━┯━━━━━┛
│
┌──────────┼──────────┐
▼ ▼ ▼
HOLD LIMITED BROAD
DEPLOYMENT DEPLOYMENT“More Evidence” now has a principled meaning: Which unsupported part of our proposed action requires stronger evidence? A surprising metric movement may be real, or the Judge may have drifted. Forcing every ambiguous result immediately into PASS or FAIL converts uncertainty into arbitrary policy.
This also makes reversibility part of the contract. Evidence insufficient for an irreversible broad release may still justify a constrained rollout that generates more information under bounded risk.
Learning and Assurance Are Different Jobs
The distinction introduced earlier becomes especially important when evidence is used to justify a release. Organizations are usually trying to answer two different questions with evaluation.
The learning loop asks: Where might we be wrong, and what should we try next?
The assurance loop asks: Is the evidence strong enough to justify this consequential action?
LEARNING ASSURANCE
discovery evidence frozen evidence
│ │
▼ ▼
fast evaluation calibrated gate
│ │
▼ ▼
update hypothesis release claim
│
└──▶ next iterationThe loops must not collapse into each other. The distinction is statistical. The better a dataset becomes at hunting failures (adversarial mining, rare events), the worse its raw aggregate statistics become as an estimate of ordinary population behavior.
Furthermore, once a Judge trains, filters, or ranks the system it measures, it is no longer merely an observer. It has become part of the model-producing process. Optimization pressure will inevitably exploit it.
There is an uncomfortable lifecycle here: a successful evaluator tends to erode its own independence. Once a signal reliably finds problems, the organization starts optimizing against it. What began as independent evidence gradually becomes part of the candidate-producing process, and its assurance value changes with that entanglement.
Mistaking an offline ranking improvement for a product improvement costs an experiment. Mistaking a benchmark exploit for genuine capability can send research in the wrong direction. Mistaking a safety proxy for sufficient release evidence has a much higher consequence.
The need for independent assurance grows with:
Need for independent assurance increases with: semantic distance from the goal + optimization pressure on the evaluator + cost of false acceptance.
The wider that gap becomes, the more you need an independently constructed assurance loop with different statistical guarantees.
Institutional Memory
No evaluation architecture has complete coverage. Something eventually fails in production.
But not every production failure indicts the evaluation system. An evaluator may correctly predict that some residual failure rate remains and a release decision may knowingly accept that risk.
The important distinction is between a predicted failure and an epistemic surprise.
production event
│
┌──────────┴──────────┐
▼ ▼
predicted failure epistemic surprise
│ │
▼ ▼
improve Actor / investigate
cost tradeoff │
▼
┏━ INSTITUTIONAL MEMORY ━┓
┃ durable failure object ┃
┗━━━━━━━━┯━━━━━━━━━━━━━━━┛
│
├──▶ update Actor
├──▶ update Environment
├──▶ update Judge
├──▶ update evidence selection
├──▶ update aggregation
└──▶ update gate / release policyA production failure is evidence about the Actor. A production failure outside the evaluator’s predicted envelope is evidence about the evaluation system itself. It reveals a blind spot in what the organization believed it knew.
To compound in value, meaningful surprises need a closure path. Enterprise observability platforms are increasingly used to implement the “golden-set” loop: developers using tools like LangSmith or Braintrust export production failures directly from the trace stream into an offline regression set.
A regression set is only the first step.
The important question is how much we learn beyond the individual failure. Did we merely memorize one bad input by adding a test? Or did the incident teach us a failure mode, a new sampling rule, a better Judge, or a missing dimension of the Environment simulator?
The goal is to turn production surprises into durable improvements in what the organization can detect and understand. Production should improve both the model and the organization’s ability to judge whether the model is good.
Practical Advice for Evaluation Owners
If you’ve read this far, there’s a good chance you’re responsible for evaluation somewhere in your organization. There are probably three questions on your mind: Where should I invest? What should be centralized? How can I start small?
Where should I invest?
Ask: “Where does trustworthy information stop turning into action?”
Sophisticated tools such as automated judges, giant simulation farms, and massive golden sets are multipliers.
An evaluation system also has to be usable. If the trusted path is too slow, noisy, or opaque, engineers will create faster shadow paths around it; assurance that is routinely bypassed provides little assurance at all.
A useful prioritization order is:
- Integrity: If another engineer cannot reproduce the Actor, Environment, and Judge for a run, stop here. Every higher-level conclusion inherits that uncertainty.
- Epistemic Quality: Understand what your data actually represents before generating more of it. Uncalibrated Judges and hidden sampling bias produce false certainty.
- Decision Linkage: Make sure evaluation results can advance, block, or demand more evidence for an actual decision.
- Operating Efficiency: Once the evidence is sound, make it cheap and fast enough to produce and inspect routinely.
- Institutional Memory: Ensure meaningful production surprises improve what the organization can detect next time.
Sophistication at a higher layer cannot compensate for failure below it. The highest-leverage investment is usually the lowest broken layer constraining decision quality.
What should be centralized?
The semantics of an evaluation are usually domain-specific. Teams closest to the system need to define what good means, which situations matter, which failures are unacceptable, and what evidence a decision requires.
The machinery underneath is often much more reusable: execution, environment provisioning, trace capture, Judge execution, result storage, comparison, caching, and scheduling. Separating that machinery from an individual run allows the same engineering investment to benefit many teams, model families, and releases.
Evaluation execution is an early example. A production serving path may be optimized around individual request latency and other online constraints. Offline evaluation can make a different tradeoff: batch work aggressively and saturate accelerators. A shared execution path can therefore make the same evaluations substantially faster and cheaper, while improvements in batching, scheduling, caching, or accelerator utilization benefit every team using it.
The larger gain is a shorter development feedback loop. Making evaluation faster shortens the distance between changing a system and learning whether the change helped. Similar economics can justify shared investment in replay infrastructure, reproducible environments, and trace inspection.
How can I start small?
How much machinery you need depends on the stakes.
For a low-risk, easily reversible LLM application, the first useful version may be only a handful of critical behaviors, explicit expectations, repeatable execution, enough provenance to understand failures, and a release rule the team actually uses. Production can supply the next cases worth adding.
I’ve written about this smaller starting point in Golden Datasets: Launch Criteria for LLM Systems.
What Infinite Compute Clarifies
Operating this machine in the real world is a massive systems-engineering challenge. Building low-latency replay infrastructure, scaling simulation clusters, managing fidelity ladders, and scheduling human triage are non-trivial problems that absorb enormous engineering effort.
A thought experiment helps separate these operational scaling challenges from the underlying measurement principles. If an evaluation system had infinite compute and zero latency, what would change?
The budget constraints would relax. Caching, queues, early stopping, and simulation-budget tuning would become unnecessary. But the epistemic constraints would remain:
- Selection bias survives. Mining hard edge cases still does not estimate population risk.
- Evaluator coupling survives. Evidence produced by machinery already entangled with candidate development cannot automatically serve as independent assurance.
- Construct validity survives. Infinite compute cannot tell you whether the proxy you measured is the property you meant by “safe,” “helpful,” or “correct.”
- Attribution survives. Observed differences still arise jointly from the Actor, the Environment, and the Judge.
These are not compute problems; they are measurement problems.
This brings us back to where we started: where the complexity went when the code became weights.
In classical software, much of an organization’s accumulated engineering judgment was encoded directly into the artifact: abstractions, contracts, invariants, edge-case handling, and business logic lived in code.
As more behavior becomes learned rather than explicitly specified, some of that engineering burden moves downstream. Models may be regenerated frequently with relatively little additional human effort, but understanding what changed, whether the evidence is trustworthy, and what action it justifies remains difficult. The faster that understanding arrives, the faster the organization can make the next informed change.
That is the role of the evaluation system: to make learned behavior legible enough to act on.
Sources
This framework draws on my day-to-day experience building and leading the ML & Data Platform function at Woven by Toyota. The public sources below are examples chosen to illustrate the same underlying ideas across LLMs and agents, search and recommendation, and autonomous driving.
- Poolside’s benchmark-hacking investigation: Details the Laguna model jump, Git-history exploits, web search hacking, and the conclusion that process evidence and continuous sample review are necessary because terminal scores alone are insufficient. Through the looking glass of benchmark hacking
- Poolside’s Laguna technical report: Demonstrates that a benchmark score is the product of a much larger execution stack, with detailed disclosures on harnesses, sampling, verifier patches, and execution configuration. Laguna M.1 / XS.2 technical report
- Waymo’s Simulation City: A deep dive into rare-event variation, distributions of behavior, software validation, and how simulated experience complements real-world experience. Simulation City
- Waymo’s collision-avoidance testing: Highlights systematic scenario identification, virtual counterfactual testing, and the iterative construction of representative scenario sets. Collision avoidance testing of the Waymo ADS
- Airbnb’s ranking work: A clean search example demonstrating the split between offline NDCG and separate online A/B outcomes, rather than treating the proxy as the product objective itself. Personalizing Airbnb search by learning from the guest journey
- Georgian AI Lab on retrieval evaluation: A practical treatment of evaluation when ground-truth relevance labels are unavailable, including task-specific rubrics, LLM Judge reliability, and the distinction between a useful iterative yardstick and absolute ground truth. Evaluating Retrieval Without Ground Truth
- LangSmith and Braintrust Evaluation: Documentation showing the implementation of the production trace → regression dataset → future evaluation loop. LangSmith Evaluation | Braintrust: production failures into regression tests