Verification

Preview

Why Pimsy does not assume its first answer is correct.

Verification is the difference between an agent that produces output and an agent that produces results. Every step can declare a check; the task cannot complete while a required check is failing.

Verification strategies#

Strategy	Applies to	Method
Execution	Generated code	Run it; compile, execute, assert on output
Test-based	Software changes	Run the suite, plus a test that fails pre-change
Recomputation	Numeric results	Recompute by an independent method and compare within tolerance
Source resolution	Research claims	Re-fetch the source and confirm the cited span
Cross-source agreement	Factual claims	Require independent corroboration for material claims
Contradiction scan	Long documents	Detect internal inconsistency across the artifact
Adversarial review	High-stakes output	A verifier agent argues the result is wrong
Schema validation	Structured output	Validate against the declared schema before delivery

For high-stakes work the orchestrator spawns a verifier with read access to the artifact and the trace but no authorship stake in the result. Its instruction is to find the strongest reason the result is wrong.

producer agent ──▶ artifact ──▶ verifier agent
                                      │
                 ┌────────────────────┤
                 ▼                    ▼
           issues found          no issues
                 │                    │
                 ▼                    ▼
        revision step          mark verified
                 │
                 └──▶ re-verify (max N rounds, then escalate)

Reporting uncertainty#

When verification cannot be completed — a source is offline, a test environment is unavailable, evidence remains contested — Pimsy returns the result marked unverified with the reason attached. It does not upgrade an unverified result to a confident one.

json

{
  class="tok-str">"status": class="tok-str">"completed_with_caveats",
  class="tok-str">"verification": {
    class="tok-str">"passed": false,
    class="tok-str">"checks": [
      { class="tok-str">"name": class="tok-str">"tests_pass", class="tok-str">"result": class="tok-str">"pass" },
      { class="tok-str">"name": class="tok-str">"citations_resolvable", class="tok-str">"result": class="tok-str">"partial", class="tok-str">"detail": class="tok-str">"class="tok-num">2 of class="tok-num">14 sources unreachable" }
    ]
  },
  class="tok-str">"confidence": class="tok-str">"medium",
  class="tok-str">"caveats": [class="tok-str">"Two cited sources could not be re-fetched at delivery time."]
}

Last updated 2026-09-14

Verification strategies#

Verifier agents#

Reporting uncertainty#