How the referee works, and where it stops short.

01 — Where tasks come from

History mining

Your repository’s past is a free, perfectly realistic task dataset. Guignet walks git history — --no-merges, so the actual fix-plus-test commits aren’t hidden behind merge commits — and reconstructs each task-shaped commit into a tuple.

prompt

The task statement, rebuilt from the human record: commit subject and body, and issue or PR text where available.

base

The parent commit the agent starts from.

ground truth

The real source fix. Held out.

verifier

The tests the commit added or changed. Also held out.

Discovery runs three heuristics as peers

No single signal is enough, so Guignet unions their matches and records which heuristics surfaced each task:

• Commits whose diff touches both test and source files — the strongest signal, and the only one that guarantees a verifiable task.
• Strict conventional-commit prefixes (fix:, feat:, …).
• A configurable loose “scope: summary” prefix, plus issue-linked commits.

On a real repository the test-plus-source signal does most of the work — conventional prefixes alone were a minority of history.

The leak firewall

A benchmark is worthless if the agent can see the answer. Guignet enforces this structurally, not by convention: the prompt reconstructor accepts only a PromptContext that has no field capable of carrying a diff, ground truth lives behind a single module, and a CI import-boundary check fails the build if any stage that shouldn’t touch ground truth so much as names its path. The mining stage writes ground truth once and never reads it back.

02 — Why the tasks are trustworthy

The validity gate

A reconstructed task is admitted only after Guignet proves it reproduces reality, in a clean git worktree using the repository’s own setup and test commands:

Any task that flakes — passes at base, fails at the fix, times out, or whose environment won’t stand up — is discarded, never patched. Discarding is the feature: it is what makes the surviving suite sound.

Guignet publishes the soundness rate (admitted ÷ discovered candidates) in every report, and records a per-candidate discard reason. A low soundness rate is not hidden; it is a signal about mining quality on that repository.

03 — Running agents fairly

Execution

Each attempt runs in its own disposable git worktree checked out at the task’s base commit. Guignet installs the repository’s dependencies first, so the agent starts from the environment a developer would have — able to resolve imports and run tests. Setup time is not charged against the agent. The harness never pushes and never touches your real checkout.

Agents are stochastic, so Guignet runs N attempts per task (default 3). A single attempt is watermarked “anecdote, not measurement” in the report — one sample is not a benchmark. Adapters are thin per-harness drivers: v0 ships claude-code headless and a generic-cli escape hatch that runs any command, so Guignet is not tied to one agent.

Cost is captured by parsing each harness’s own transcript for tokens and dollars — never self-reported by the agent — and wall-clock is measured around the agent call alone. When an attempt has no priceable transcript (a crash, a generic command), its cost is recorded as unknown rather than zero, and any aggregate built from partial coverage is marked a lower bound.

04 — The verdict

Scoring

The primary verdict is binary and judge-free: apply the agent’s solution to a fresh worktree, overlay the held-out verifier, run it. Pass or fail. Every un-judgeable condition — the solution won’t apply, the environment won’t build, the verifier crashes or hangs — resolves to a fail. No credit for a solution that couldn’t be objectively verified.

The verifier is authoritative over its own paths

Agents routinely fix the source and write their own tests, often in the very files the held-out verifier patches. Before the overlay, Guignet sets aside every part of the solution that touches a held-out path. The agent is judged on its source change alone — the mirror of what the validity gate proved — so its own test edits are neither punished nor rewarded.

This cannot credit a wrong fix: the real verifier is restored over any agent edit to it, and a solution whose only change is test edits has nothing to judge and fails. Verdicts record a testEditsFiltered flag whenever the projection removed anything, and secondary metrics are computed on the same source-only projection.

The numbers

A task is solved by a config if at least one of its attempts passes (resolve@n). The headline number is $ per solved task. Solve rates carry a 95% Wilson confidence interval, always rendered — on the small suites a single repository yields, the interval is the honest part of the number. Secondary metrics: tokens, wall-clock, and a bloat ratio comparing the agent’s change size to the ground-truth fix.

05 — And their honest limits

Contamination controls

A model may have trained on your public history. Guignet dates every task, splits every score by the model’s training cutoff, and flags solutions whose token-level Jaccard similarity to the held-out fix is suspiciously high on pre-cutoff tasks — verbatim reproduction is memory, not skill. Flag rates are reported per model. The cutoff registry ships with defaults, is overridable in config, and every report prints the version it used.

What the split means depends on your history

Guignet frames it accordingly, via repoVisibility:

In all cases, post-cutoff tasks are also simply more recent commits, which can differ in difficulty for reasons unrelated to the model. The split is a flag to investigate: suggestive, not dispositive.

06 — Stated plainly

Known limitations

A benchmark that only advertises its strengths is marketing. These are the places Guignet stops short.

1. 01Task quality is the whole game

   Terse commit messages produce weak prompts. The soundness rate is the honest measure of how well mining worked on a given repository — publish it, don’t average it away.

2. 02The suite is small

   A repository yields tens to low-hundreds of sound tasks, so confidence intervals are wide. That is exactly why they are always shown.

3. 03Pattern resemblance is unmeasured

   Even on a private repository, a task may resemble ubiquitous public code the model has seen many times. That is a real form of “not novel” the cutoff split does not capture.

4. 04Environment fidelity caps yield

   A task is admitted only if its verifier reproduces in a hermetic worktree. Tasks needing services the worktree can’t provide are discarded — conservative, but it limits yield on integration-heavy suites.

5. 05Dependency isolation is partial

   Replays currently preserve node_modules between resets. Other ecosystems’ in-tree dependency directories are a known gap.

07 — Auditable by construction

Reproducibility

The filesystem is the database. Every stage writes to .guignet/ and is idempotent and resumable — a run interrupted at attempt 23 resumes at 23. The report is pure: guignet report regenerates from stored runs with no re-execution, and emits a --json twin of every number in the HTML, so results are auditable and programmatically consumable.

Nothing here requires trusting Guignet’s word over your own inspection of the store.