# Evals

Evals run fixed cases against an agent and record whether its turns,
tools, events, and output meet a contract. The
[Evals guide](/docs/evals.md) covers the regression workflow.

## Eval discovery and case IDs

Eval files live under the project-root `evals/` directory and end in
`.eval.ts` or `.eval.js`. The path under that directory becomes the
eval ID, so `evals/readiness.eval.ts` becomes `readiness`,
`evals/builds/api.eval.ts` becomes `builds/api`, and
`evals/builds/index.eval.ts` becomes `builds`.

Export one `defineEval` that sets exactly one of `test` or `cases`. A
case ID adds one path segment to the file ID:

```ts
export default defineEval({
  tags: ["smoke"],
  cases: [
    {
      id: "checkout",
      async test(t) {
        await t.send("Review checkout PR 42.");
        t.succeeded();
      },
    },
    {
      id: "search",
      async test(t) {
        await t.send("Review search PR 7.");
        t.succeeded();
      },
    },
  ],
});
```

Those cases are `prs/checkout` and `prs/search` when the file is
`evals/prs.eval.ts`. Case IDs must be unique single path segments.

A file can also export an array of `defineEval` calls, named with
zero-padded indexes such as `sql/0000`.

`iterations` repeats one datapoint 1 to 100 times. With
`iterations: 3`, `weather/nyc` runs as `weather/nyc/1` through
`weather/nyc/3`, and selecting `weather/nyc` runs all three.

## Eval configuration

Every suite needs `evals/evals.config.ts` with `maxConcurrency`:

```ts
import { defineEvalConfig } from "@grok/agent-sdk/evals";

export default defineEvalConfig({
  maxConcurrency: 20,
  judge: { model: "gpt-5.4-mini" },
});
```

| Option | Contract |
| --- | --- |
| `maxConcurrency` | Required. Concurrent authored datapoints, from 1 to 200 |
| `timeoutMs` | Per-case timeout. A case or file value wins, then the CLI flag, then this config |
| `judge` | Default model for `t.judge` |
| `reporters` | Objects notified as cases and runs complete |
| `maxPlaygroundRuns` | Server-side batches kept in playground history. Defaults to 20 |

A case can override `description`, `tags`, `timeoutMs`, `iterations`,
`judge`, `reporters`, and `metadata`. Case metadata merges over file
metadata, and case reporters add to the file's reporters.

## Send turns in a case

`await t.send(message, options?)` runs one turn and waits until it
finishes or fails. Sends in one test share a session.

The returned turn has the assistant `message`, `sessionId`, `events`,
ordered `toolCalls`, `ok`, and a one-based `index`. Assertions on a turn
inspect only that turn, and assertions on `t` inspect the whole case.
The case also exposes `t.reply`, `t.events`, `t.turns`, `t.sessionId`,
`t.iteration`, `t.iterations`, and the timeout `t.signal`.

On the first send only, `workspaceFiles` seeds the session workspace
from a map of relative paths to content, and `workspaceDir` sets an
absolute working directory for a new local session.

`turn.expectOk()` throws when the turn failed, so later steps that
depend on it don't run.

## Trajectory assertions

Assertions record failures without stopping the test.

| Assertion | Checks |
| --- | --- |
| `t.succeeded()` | Run finished without failure |
| `t.calledTool(name, matcher?)`, `t.notCalledTool(name)` | A matching tool request occurred, or none did |
| `t.loadedSkill(name)` | Agent opened a named skill |
| `t.toolOrder(names)`, `t.eventOrder(matchers)` | Tool requests or event groups appeared in relative order |
| `t.usedNoTools()`, `t.maxToolCalls(max)` | No tool was requested, or the count stayed under the bound |
| `t.noFailedActions()` | No tool result failed |
| `t.calledSubagent(name, matcher?)` | Matching subagent call occurred |
| `t.taggedArtifact(kind?, predicate?)` | Matching durable artifact was tagged |
| `t.event(type, matcher?)`, `t.notEvent(type, matcher?)` | A matching session event occurred, or none did |
| `t.eventsSatisfy(label, predicate)` | Predicate accepted the event stream |
| `t.check(value, expectation)` | Value met an expectation builder |
| `t.score(name, value)` | Recorded a numeric score from 0 to 1 |

`t.requireToolCall`, `t.requireInputRequest`, and `t.require` return the
matched value and stop the rest of the test when nothing matches.

## Tool and event matchers

Tool matchers can narrow by `input`, `output`, `status`, and `count`.
Values can be literals, regular expressions, or predicates. Object
literals match partially:

```ts
t.calledTool("inspect_pr", {
  input: { verdict: "review" },
  count: 1,
});
```

Tool status is `completed`, `failed`, `pending`, or `rejected`.
Subagent matchers accept `output`, `status`, `count`, and `callId`.
Event matchers accept `data` and `count`.

## Expectation builders

`includes(string | RegExp)`, `equals(value)`, `matches(schema)`, and
`satisfies(predicate, label)` are hard gates by default, and
`similarity(expected)` is a soft score. `matches` accepts Standard
Schema implementations such as Zod.
`normalizedSimilarity(value, expected)` returns the same 0-to-1 value
that `similarity` uses.

## Severity and verdicts

Every assertion returns a severity handle:

```ts
t.calledTool("search").soft();
t.check(t.reply, similarity("Expected answer")).atLeast(0.8);
t.judge.closedQA("names the failing step").gate(1);
```

- `.gate(threshold?)` fails the case on a miss.
- `.soft(threshold?)` records a value and never fails the case.
- `.atLeast(threshold)` sets a soft bar, and a miss gives the case a
  `scored` verdict.

Case verdicts are `passed`, `failed`, `scored`, or `skipped`. A `scored`
case fails the run only under `--strict`.

## Judges

Judges grade what deterministic checks can't express:

```ts
t.judge.factuality(expected).atLeast(0.8);
t.judge.summarizes(expected).atLeast(0.8);
t.judge.closedQA("names the root cause").gate(1);
t.judge.sql(expected).atLeast(0.9);
```

Each judge grades `t.reply` unless you pass `{ on }` with another value.
Set judge options in the project config, the eval, the case, or the
call, and the nearest value wins.

`t.judge.model(prompt)` returns a raw model reply for custom grading.
Agent output is untrusted input, so wrap it with `fenceUntrusted` and
include `EVAL_JUDGE_INJECTION_GUARD` in the judge prompt.

## Judge host

A judge needs a judge model and a Cursor credential from
`agent-sdk login`, `CURSOR_API_KEY`, or `CURSOR_SERVICE_ACCOUNT_KEY`.
Without a usable credential the runner skips the case, and without a
judge model it fails the case and names where to set one.

## Datasets and fixtures

Load committed JSON, JSONL, or YAML relative to the project root:

```ts
import { defineEval, includes } from "@grok/agent-sdk/evals";
import { loadYaml } from "@grok/agent-sdk/evals/loaders";

const rows = await loadYaml<Array<{ prompt: string; expected: string }>>(
  "evals/data/cases.yaml",
);

export default rows.map(row =>
  defineEval({
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, includes(row.expected));
    },
  })
);
```

## Reporters and results

Built-in reporters include `JUnit({ filePath, suiteName? })` and
`Artifacts({ dir })`. A custom reporter can implement `onRunStart`,
`onEvalComplete`, and `onRunComplete`.

JSON output has run totals and one result per case. Each result has the
case ID, verdict, assertions, session ID, inputs, final text, tool
calls, metrics, logs, duration, metadata, tags, and any error or skip
reason.

Each local run writes `summary.json`, `results.jsonl`, and
`evals/<case-id>.json` under `evals/<stamp>/` in the project state
directory. `--state-root` doesn't move them, and `--artifacts <dir>` or
`--no-artifacts` does.

## Select cases

```text
agent-sdk eval --dir . --list
agent-sdk eval --dir .
agent-sdk eval --dir . builds/checkout
agent-sdk eval --dir . --tag smoke
```

An ID matches itself and its descendants. Repeated IDs or tags combine
with OR, and a case must match both groups when you pass both.
[Run evals](/docs/reference/cli.md#eval) covers runs on a server.

## Suite summary cases

A case tagged `EVAL_SUMMARY_TAG` aggregates its file, so selecting it by
ID or with `--tag summary` also runs every other case in that file.

```ts
import { defineEval, EVAL_SUMMARY_TAG } from "@grok/agent-sdk/evals";

export default defineEval({
  cases: [
    {
      id: "checkout",
      async test(t) {
        await t.send("Review checkout PR 42.");
        t.succeeded();
      },
    },
    {
      id: "overall",
      tags: [EVAL_SUMMARY_TAG],
      async test(t) {
        await t.send("Summarize the checkout review.");
        t.succeeded();
      },
    },
  ],
});
```
