Custom Evaluations

Conceptual

Building a suite from the work you actually do.

The highest-value evaluation set is a collection of real tasks from your own history, with known-good outcomes. Fifty of these beat any public benchmark.

Building the set#

1
Collect
Pull real tasks from history, including the ones that went badly.
2
Label
Record the correct outcome and what made it correct.
3
Stratify
Cover easy, typical and hard cases — not just the interesting ones.
4
Define graders
Programmatic where possible; rubric-based where judgment is required.
5
Baseline
Measure the current configuration before changing anything.
6
Gate
Run the suite in CI and block regressions.

evals/checkout.eval.ts

import { defineEval } from class="tok-str">"@pimsy/evals";

export default defineEval({
  name: class="tok-str">"checkout-regressions",
  cases: [
    {
      id: class="tok-str">"eu-tax-null",
      objective: class="tok-str">"Guest checkout returns class="tok-num">500 for EU addresses. Diagnose and fix.",
      fixture: class="tok-str">"repos/checkout@a91f3c",
      graders: [
        { type: class="tok-str">"tests_pass", command: class="tok-str">"pnpm test" },
        { type: class="tok-str">"diff_size", maxFiles: class="tok-num">3, maxLines: class="tok-num">120 },
        { type: class="tok-str">"regression_test_added", mustFailBefore: true },
        { type: class="tok-str">"rubric", criteria: [class="tok-str">"Root cause is stated, not just the symptom"] }
      ],
      budget: { steps: class="tok-num">40, usd: class="tok-num">3 }
    }
  ]
});

Grader types#

exact_matchgrader: Deterministic comparison against expected output.
tests_passgrader: Runs a command and requires exit code zero.
citation_checkgrader: Verifies each claim resolves to a cited source.
diff_sizegrader: Bounds the size of the produced change.
policy_compliancegrader: Asserts no denied calls and correct approval behaviour.
rubricgrader: Model-graded against explicit criteria, with human spot-checks.

Last updated 2026-09-11