Harness Bandit Experiments — Methodology

The first published multi-armed-bandit study over agentic coding harnesses: same KYM agent cards, same memory stack, same skills — vary only the runtime harness, score with signed receipts and dual (agentic + human) evaluations, publish the reward-shaped winner per task class.

Goal

Publish the first multi-armed-bandit study over agentic coding harnesses. The prior art (Harness-Bench, arXiv 2605.27922) benchmarks harnesses on fixed suites; nothing published to date runs a live bandit that adapts arm selection per task class against a receipted, rubric-scored reward signal on identical agent cards. That is the gap this study fills.

Architecture rule — harness is operational code, agent definitions live in KYM

What the harness OWNS (operational)

• Worktree fan-out (worktree-mcp setup_workspace per subagent; base branch resolution; branch naming wt/<wave>/<slug>).
• Wave launching (run_batch orchestration, job seeding from wave-config, engine-adapter selection via --engine).
• Gate wiring — package-scoped validate during concurrency; root validate on the merged tree post-merge; --tool-profile implementation + --implementation-approved as machine-checked pair.
• Permission enforcement — OPENCODE_PERMISSION (edit/bash/task/external_directory deny) for read-only waves; hard-denied git push / gh pr create / gh pr merge / publish / deploy for all subagent sessions. "Never commit, never push, never open PRs" enforced by the harness process boundary, not by prompt trust.
• Telemetry + receipt plumbing — every turn emits an Ed25519-signed receipt to KYM (signAndSubmitReceipt) and a tool_benchmark_events row to console D1; batching, retry, and dead-letter are harness concerns.
• Merge-ladder automation hooks — steps 1–3 of the technique ladder (git merge-tree --write-tree, mergiraf, git apply --3way) are harness-invoked; steps 4–5 (manual fix, re-dispatch) return control to the orchestrator agent.
• Engine adapters — one per runtime under scripts/engines/ (opencode default, claude-code parity, ACP-generic). New engines join by writing an adapter, not by forking the runner.
• Wave/job schema — wave-config.json, tracker.csv, state.json, manifest.json, results/*.md. Schemas versioned in the harness; content is agent-authored.

What KYM OWNS (agent definitions)

• Orchestrator agent definitions — the "who plans, dispatches, reviews, merges" agent lives in KYM as an Agent Card + versioned skill set. A new orchestrator variant is a new Agent Card version.
• Subagent definitions — every subagent role (search, implementation, audit, evaluator) is a KYM Agent Card with its own skill versions. The harness dispatches by referencing kym://agents/<slug>@<version>, not by inlining prompt text.
• Dispatch-contract clauses as versioned skill content — the nine clauses from /research/parallel-agent-orchestration §dispatch-contract ride as versioned prompt/skill content on the orchestrator Agent Card. A clause revision is a skill version bump; A/B-testing a clause is an experiment on that skill.
• Rubric + evaluator agents — the rubric IS the reward function (§rubric); both the rubric and the agentic evaluator are KYM artifacts with signed manifests. A rubric bump is a KYM event.
• Experiments — MAB / A/B comparisons key on the tuple (agent_card_id, agent_card_version, skill_id, skill_version, wave_spec, engine_adapter). Comparisons run without harness changes; the harness resolves whichever Agent Card version the wave-config points at.
• Receipts — every completed turn writes a signed receipt back to KYM. Receipts carry the full experiment-key tuple so read-back can slice by any axis (agent version, skill version, engine, wave).

Experiment keying under the split

A KYM experiment is a MAB / A/B over the tuple (agent_card_id, agent_card_version, skill_id, skill_version, wave_spec, engine_adapter). The harness resolves whichever Agent Card version the wave-config points at, runs the wave, and emits one signed receipt per turn carrying the full tuple. Reward flows back to KYM keyed by the tuple; console-side D1 (experiments + experiment_variants.config, see §Data model) accommodates the same shape so a single dashboard can slice by any axis. Changing an agent-definition version and re-running the same wave requires zero harness commits and zero harness restarts — that is the exit criterion for the split (roadmap wave-harness-kym-split).

Migration ledger — honest current-shape assessment

The harness today ships prompt bodies and behavior contracts under templates/. Anything in the "MIGRATE" column below is agent-definition-shaped content that must move to KYM before the exit criteria are provably met. Anything in "STAY" is operational and must NOT be moved. This is a design ledger; the actual migration lands in a follow-up wave gated on operator approval.

• templates/*/claude-agent-prompt.md — agent-definition (subagent prompt body).
  MIGRATE — republish as KYM Agent Card + versioned skill; harness references by kym://agents/<slug>@<version>.
• templates/*/augment-orchestrator.md — agent-definition (orchestrator prompt body).
  MIGRATE — republish as KYM orchestrator Agent Card.
• templates/shared/spawned-AGENTS.md — agent-definition (spawned-run behavior contract).
  MIGRATE — the behavior contract IS the dispatch-contract skill; publish as a versioned skill on the orchestrator Agent Card.
• scripts/prompt_rendering.py (any embedded clause strings) — mixed — templating is operational, clause TEXT is agent-definition.
  PARTIAL — keep the template engine; move every embedded clause string to a KYM skill body.
• scripts/run_batch.py, scripts/provision_worktrees.py, scripts/waves_status.py, scripts/report_parsing.py — operational.
  STAY — engine.
• scripts/engines/* — operational (engine adapters).
  STAY — engine adapters are operational surface; new engines join by adding an adapter, not by editing agent definitions.
• OPENCODE_PERMISSION wiring, --tool-profile implementation + --implementation-approved gate — operational (permission enforcement).
  STAY — process-boundary enforcement; NEVER moved to prompt/skill content.
• wave-config.json / tracker.csv / state.json / manifest.json schemas — operational (schema).
  STAY — schema versioned in the harness; content is agent-authored and references KYM Agent Card IDs.

KYM surface facts (verified) and open questions

• VERIFIED — @nexartis/knowyourmodel-sdk exposes getAgentCard(id) → KymAgentCard (skills, registry memberships, receipt stats). Src: knowyourmodel-sdk/src/core/agents.ts.
• VERIFIED — signAndSubmitReceipt(payload) posts to POST /api/receipts/verify with Ed25519 signing supplied by the caller; the SDK does not read secrets from env. Src: knowyourmodel-sdk/src/core/receipts.ts.
• VERIFIED — SDK is zero-runtime-npm-dep; platform fetch + node:crypto / Web Crypto. Safe to consume from the harness process and from every engine adapter identically.
• VERIFIED — the x-nexartis-runtime Agent Card extension is used in practice today for the local-mac / cubicle host axis (tech-debt open in wave-1); the same extension shape is a natural hook for the "role" axis (orchestrator vs subagent) but NOT yet formalised.
• OPEN — Agent Card field carrying "role" (orchestrator vs subagent vs evaluator). Candidate: x-nexartis-runtime extension or a first-class role field. Owner: knowyourmodel-ai.
• OPEN — dispatch-contract clauses as ONE versioned skill vs N versioned skills. One skill makes A/B on the whole contract cheap; N skills lets a MAB search clause-by-clause. Direction likely: N skills.
• OPEN — receipt payload accommodation for (agent_card_id, agent_card_version, skill_id, skill_version, wave_spec_hash, engine_adapter). ReceiptPayload shape needs a check-in — likely fits under a metadata object without a schema break.
• OPEN — MAB read-back projection: how experiments.paper_slug + experiment_variants.config align with the KYM experiment surface so a single dashboard can show both console D1 rollups and KYM receipt-anchored winners without double-writing.

Cross-link: parallel-agent-orchestration §lifecycle-v2 marks each step A (automatable by the harness) vs J (orchestrator judgment). The harness bakes the A steps; KYM defines the J agents. That is the same split, stated lifecycle-side.

Baseline

The baseline is the production KYM orchestrator + subagent set that runs today. pegasus-horizon-breakthrough is the reference client for that baseline and is out of implementation scope for this study — we clone the agent set, not the client. The study varies runtime harness against the cloned set; anything else that varies is a bug in the study, not a signal.

Design

Clones of the production agent set — same KYM agent cards, same memory stack, same skills — run in parallel against the tier-1 harness slate below. The only varying axis is the runtime harness. Every arm reports through the same nexartis-parallel-waves-agent-harness base contract (Terminal-Bench BaseAgent shape), and mini-swe-agent runs alongside as the deliberately-minimal control arm: if the fancy harnesses do not beat mini on our task set, that is a headline finding, not a footnote.

Two-tier adapter model per memory-tooling D10: native adapters per engine, plus a generic ACP adapter for anything speaking Agent Client Protocol. New engines join the study by writing an adapter, not by forking the runner.

Library — nexartis-parallel-waves-agent-harness

Single library + engine adapters: memory-tooling D10 landed on option (a) after weighing per-repo duplication (rejected) and plugin packages (overengineered for this stage). base.py follows the Terminal-Bench BaseAgent shape so any Terminal-Bench-shaped harness is a mechanical adapter port; the control arm is mini-swe-agent for the same reason. All wave launches go through nexartis-parallel-waves-agent-harness; v1 is frozen at nexartis-parallel-waves-archived and no longer runs waves. Implementation waves require the machine-checked pair --tool-profile implementation --implementation-approved; without both flags the runner refuses to launch.

Tier-1 harnesses under test

Ten tier-1 arms + one control. Each row records what the study needs to know to instrument the arm: how it is invoked headless, and what artefacts can be captured as receipts. Measured task-completion and reward numbers land in the ledger below once the first bakeoff cycle completes.

Auggie v0.33.0 — measured empirics

Adapter live at nexartis-auggie-harness/. The findings below were measured on this machine, not read off vendor pages. Auggie's tool surface, MCP-server surface, JSON envelope, daemon semantics, and rules-interop story all differ enough from the other tier-1 arms to warrant a dedicated record before it enters the bandit.

• Working baseline: 2 identical one-shot `auggie --print` runs on a zero-dependency Node fixture moved it from 2/10 to 10/10 tests in ~30 s each, with minimal two-file diffs, respected constraints (did not touch tests, added no dependencies, ran the test suite itself, reported the true root cause). Harness lives at nexartis-auggie-harness/.
• 34 tools enabled by default, including `linear` and `github-api`, plus 7 sub-agent tools (`sub-agent-explore`, `-plan`, `-research`, `-code`, `-validate`, `-general-purpose`, `-auggie-guide`) and two distinct retrieval tools (`codebase-retrieval` and an undocumented `codebase-retrieval-raw`).
• `auggie --mcp` exposes exactly ONE tool — `codebase-retrieval` taking `{information_request: string}`; `resources/list` and `prompts/list` both return -32601. So auggie-as-MCP-server is a retrieval-only surface — edits, subagents, web, and GitHub are unreachable over MCP.
• `--output-format json` envelope is exactly `{type, result, is_error, subtype, session_id, num_turns, request_id}` plus `retry_stats` only on retry. `subtype ∈ {success, empty_completion, error_during_execution, error_max_turns}`. `num_turns` is reset per thinking-turn and reads 0 even after tool use — unusable as an effort metric. A bare non-JSON line (`Applying --max-turns override: …`) is printed to stdout ahead of the JSON when `--max-turns` is passed, so consumers must parse line-by-line rather than `JSON.parse` the whole stream.
• `auggie daemon` is a hidden command and is Augment's on-prem worker for their Cosmos cloud, not a local agent pool: it dials out over WebSocket, binds nothing locally, and fail-closes without login, a live API, and the `cliEnableCloudAgents` feature flag. `--vm-id` / `--pool-id` / `--space-id` are cloud constructs. Not usable for our worktree fan-out, but the `--worktree-dir` + `use_worktree` + `cleanup_worktrees_after_stop` semantics are worth borrowing as design references.
• Rules interop, at no cost: auggie natively reads `AGENTS.md` and walks the ancestor chain (also `CLAUDE.md`, `.augment-guidelines`, `.claude/commands/`, `.agents/skills/`). An auggie arm inherits our workspace and repo conventions with zero adapter work — a real advantage over harnesses needing rule shims, and consistent with ADR-002 (no third-party editor rule shims).

Tier-2 dark-horse arms (5)

Five tier-2 arms round out the slate. Slots #4 and #5 are the Grok Build harness paired with two distinct xAI models (grok-4.5 and grok-build-0.1) — same closed-source, xAI-model-locked harness (headless streaming-JSON, ACP, MCP-native; SuperGrok or xAI API-key auth). Running the two models as distinct arms isolates the harness contribution from the model contribution on the same stack.

• Dark horse #1 — candidate: Kiro CLI (AWS). Headless via KIRO_API_KEY, subagent crews defined in .kiro/agents/*.json, speaks Agent Client Protocol (ACP). Fits the two-tier adapter model (native + generic ACP) directly. Signup + docs: https://kiro.dev.
• Dark horse #2 — candidate: Junie CLI (JetBrains). Model-agnostic BYOK including OpenRouter, reports ~78.4% on Terminal-Bench in its own results. Open-source; adapter should be a mechanical port. Repo: https://github.com/JetBrains/junie.
• Dark horse #3 — candidate: Factory Droid. Top-10 on Terminal-Bench 2.0 with published receipts; headless-first product. Docs: https://docs.factory.ai.
• Dark horse #4 — candidate: Grok Build (grok-4.5). Tier-2 beta arm. Headless via streaming-JSON stdout; speaks Agent Client Protocol (ACP); MCP-native. xAI-model-locked (arm pinned to grok-4.5). Closed-source; SuperGrok subscription or xAI API key. Adapter is a mechanical ACP port with an xAI-auth shim. Docs: https://docs.x.ai.
• Dark horse #5 — candidate: Grok Build (grok-build-0.1). Tier-2 beta arm, sibling of #4 — identical harness, distinct model. grok-build-0.1 is the xAI code-specialised checkpoint; running it as its own arm isolates harness-vs-model contribution on the same closed-source stack. Same auth (SuperGrok / xAI API key). Docs: https://docs.x.ai.

Deferred — surfaced but not admitted

Held here so the reasoning survives future revisions. Each of these came up during a harness scan and did not clear the bar for the current slate.

• Trae Agent — Held for a later cycle; not yet worth the adapter slot.
• Mistral Vibe — Pre-GA; revisit after general availability.
• Codebuff — Evals are self-reported without a public third-party baseline; deferred until independent numbers land.

Harness security caveats — containment class as a per-arm KPI

No page currently owns per-harness security posture. The findings below were measured against auggie v0.33.0 with a canary-file positive/negative control; the generalisable KPI is at the bottom. For every arm we must record a containment class, not assume one, and the bandit's reward shape must not credit an arm for speed it bought by being unsandboxed.

• `--workspace-root` is not a filesystem sandbox.
  A run scoped to a sandbox directory used `launch-process` to read and then append to a file two directories above the root; verified on disk. It bounds indexing and retrieval only. Consequence: any harness arm with write tools enabled can touch anything the OS user can, so containment must be OS-level (dedicated worktree plus a process sandbox or container), not a flag.
• `-a` / `--ask` does not prevent writes.
  Under `--ask` the agent deleted the canary via `launch-process` with `rm`, reporting "No policy blocks encountered." The CLI help claims "retrieval and non-editing tools only". It is not a safety control in 0.33.0.
• Tool-substitution footgun (upstream issue #74) reproduced in 0.33.0.
  Denying only `launch-process` caused the agent to delete the canary with `remove-files` on the first attempt. Denying one mutating family manufactures false confidence; every family must be denied together. The #74 subagent-propagation variant is genuinely fixed — under a full deny set the subagent reported the tools denied/unavailable.
• What holds.
  `--remove-tool` plus `--permission <tool>:deny` across all seven mutating families (`remove-files`, `save-file`, `str-replace-editor`, `apply_patch`, `launch-process`, `write-process`, `kill-process`) blocked the canary and propagated into subagents. `settings.json > toolPermissions` accepts the documented object form (type=deny) without warning (the bare-string form is rejected), but runtime enforcement of that path is unverified.
• Operational footgun.
  `auggie <cmd> <subcmd> --help` silently escalates into a paid agent run — it created 9 accidental sessions during probing. Any wrapper must whitelist argv.

Retrieval-ablation experiment — hold the harness constant, vary the memory stack

This inverts the page charter — same memory stack, vary the harness — to hold the harness constant and vary the memory stack. It fits the charter precisely because it isolates the retrieval axis on a stack (auggie) whose vendor claim rests on exactly that axis.

Why it is worth testing. This is Augment's own published thesis — they claim that adding only their Context Engine to Claude Code, Cursor, and Codex lifted agent performance by 30–80% (300 Elasticsearch PRs × 3 prompts = 900 attempts, five subjective dimensions: correctness, completeness, best practices, code reuse, unsolicited documentation). The study is vendor-run, has never been independently replicated, and Augment publishes no retrieval-only IR numbers at all. This ablation is therefore original work.

Three arms, identical task deck

1. (a) auggie using its own `codebase-retrieval`.
2. (b) auggie with `--remove-tool codebase-retrieval` and our MCP servers configured instead, so it must retrieve through codesearch / serena — auggie is a first-class MCP client (stdio / http / sse), which makes this a one-config change. Arm (b) is the load-bearing one: it isolates retrieval by holding harness, model, and loop constant.
3. (c) our own write router (`ozzydev-write`) with our memory stack.

What must change to make it measurable — honest gaps

• The existing write bakeoff deck was fast-apply shaped (`original + update → merged`) and file-local, so it could not detect a retrieval difference at all. Closed: the locked v2 deck is agentic-write shaped (`starting worktree`, `goal instruction`, `checks[]`) with 7 task families across 3 retrieval-difficulty strata (RE / RM / RH). RH is the load-bearing stratum — a deck without RH families collapses to RE parity and cannot discriminate between retrieval paths.
• Scoring was binary `accuracy_score ∈ {0,1}` — at an ~0.85 per-arm success rate, n = 30 only detected ≥13pp differences, underpowered for a retrieval effect. Closed: the R1–R4 continuous per-rung scorer is BUILT (nexartis-auggie-harness/probes/bakeoff/scoring/) and verified against deliberately wrong patches; retrieval-recall is computed from the MCP transcript proxy on arm-(b) and arm-(c) — the actual retrieval signal, not the binary outcome. Arm (a) is structurally `not_observable` on trajectory metrics; see §arm-a-observability.
• The v1 draft targeted n ≥ 100 cross-file tasks for a ~5pp effect. Superseded twice. Locked v2 (2026-07-29): n = 800 tiered (RH 400 / RM 250 / RE 150), MDE δ = 0.10, α = 0.05 two-sided, power 0.80, paired ρ ≈ 0.6, Holm-corrected over 10 primary tests. The intermediate n = 300 revision was itself superseded because it powered only the aggregate headline — split three ways it left ~100 tasks/stratum, below the ~255 the paired-power math requires for a standalone per-stratum conclusion. Full derivation: nexartis-auggie-harness/docs/BAKEOFF-DESIGN.md §4.

Cross-reference moe-search-router for the retrieval-side arm design, and memory-tooling-decisions (D19) for the adopt/reject decision — do not duplicate them here.

Bakeoff v2 — locked design (2026-07-29, operator-ruled)

Full spec: nexartis-auggie-harness/docs/BAKEOFF-DESIGN.md (886 lines, pre-registered). Metric contract: nexartis-auggie-harness/docs/KPI-SURVEY.md (896 lines, 43 metrics). This page summarises the load-bearing decisions and their reasoning; the design doc is the pre-registration substrate that BAKEOFF-DESIGN.md §4.4 hash-locks before any metered run.

• Deck size: n = 800, TIERED allocation RH 400 / RM 250 / RE 150. RH carries the argument; RE is a predicted null by construction (every arm scores ~100% on RE), and spending sample budget to precisely measure a predicted null is waste. The intermediate n = 300 revision powered only the aggregate headline — split three ways it left ~100 tasks/stratum, below the ~255 the paired-power math requires for a standalone per-stratum conclusion. The earlier n ≥ 100 / ~5pp target predates both.
• MDE δ = 0.10 on pass@1, α = 0.05 two-sided, power 0.80, paired correlation ρ ≈ 0.6 (verified in Phase-1 pilot; re-derived if ρ < 0.4 empirically), Holm-corrected over a family of m = 10 primary tests. Paired design cuts n ~2.5× vs unpaired. Benjamini-Hochberg FDR reported alongside as a secondary robustness check — never the primary.
• Seven task families across three retrieval-difficulty strata. RE: value-mirror (value / type variant). RM: test-driven-refactor, import-graph-completion at depth 1. RH (load-bearing): import-graph-completion at depth 3, deep-graph-navigation, semantically-similar-mislead, behavior-preserving-move, cross-package-refactor. Every RH task carries at least one lexical-overlap distractor NOT in the required-read set — a retriever picking the distractor scores zero on that rung.
• Four-source generation mix (n = 800): 400 private-repo git-log-S mining (post-cutoff commits only, ground truth = the actual commit diff verified green at commit time), 150 synthetic templates (parameters drawn from a private seed corpus that never leaves the workspace), 100 SWE-bench Verified with semantically-preserving mutation, 150 hand-crafted RH fixtures. Three-layer contamination defense — post-cutoff filter, private-corpus randomisation, held-out mutation. The (resolve@mutated − resolve@original) delta on the SWE-bench cell is itself a reported memorisation KPI: if arm-(a)'s advantage collapses under mutation, memorisation was doing the work, not retrieval.
• Phased execution — RISK control, not cost control. Phase 2a is a PROPORTIONALLY-SAMPLED 300-task tranche (RH 150 / RM 94 / RE 56, matching the 400/250/150 ratio) with a go/no-go gate; Phase 2b expands to the full 800. The tranche readout is descriptive only (point estimates + 95% CIs, no inferential p-values, no α budget consumed) and detects a design flaw or null effect before committing the remaining budget. The falsification threshold (§falsification-0pp) is NEVER post-hoc adjustable on tranche evidence — only design-flaw findings can prompt a dated amendment.
• Totals: 2,940 runs (800 × 3 headline + 300 variance-subset seeds + 100 arm-b ± retrieval ablation + 50 out-of-band arm-a recall probe + 90 Phase-1 pilot) ≈ 744 kcr across ~5 billing cycles. Cost is stated for record; it is NOT a constraint (operator ruling D1, 2026-07-29). The cost ceiling that shaped earlier n-derivations has been explicitly lifted.

KPI set — primary, secondary, diagnostic, and what we are NOT chasing

• PRIMARY (powered, Holm-corrected over m = 10): % Resolved (pass@1) per arm, cost per resolve, Retrieval-Recall@|R| (arm-(b) / arm-(c) only — arm-(a) is `not_observable`, see §arm-a-observability), edit precision (LOC not in required_edit ≤ tolerance t = max(2, 0.1 · |required_edit|)).
• SECONDARY (exploratory, 95% CIs only, no α budget): per-stratum resolve rate (RE / RM / RH), per-contamination-class resolve rate (private-post-cutoff / synthetic-private-seed / public-known-contaminated / hand-authored), per-family resolve rate (~114/family — explicitly under-powered, family-level p-values are NOT reported), continuous per-rung R1–R4 score (imports · identifiers · values · no-extraneous-edits, each ∈ [0, 0.25]).
• DIAGNOSTIC: crash rate per arm (≥ 5% triggers a crash-excluded re-derivation of the headline), wall-clock median / p90, token usage per arm, contamination-mutation delta on the SWE-bench cell, MCP tool_calls per task, retrieved-but-ignored rate (recall = 1, resolve = 0), hallucinated-symbol rate (arm-(b) / arm-(c) only).
• NOT chased, and why. HumanEval / MBPP / EvalPlus: saturated and contaminated (KPI-SURVEY §8.4). SWE-Lancer and Terminal-Bench: different task shapes, outside the retrieval-path story. nDCG@10 as a headline: non-normal variance disclaimer (KPI-SURVEY §2.9) — reported only for the 50-task variance subset with mean ± sd. Student's t on binary outcomes or on nDCG: non-normal by construction — paired bootstrap on B = 10,000 resamples throughout, permutation test as a sensitivity check.

Arm (a) observability asymmetry — not_observable, never a fabricated zero

MCP transcript shim — how arm-(b) / arm-(c) become observable

Arm (b) and arm (c) route retrieval through our own MCPs (codesearch, serena, and — for arm (c) — the full ozzydev-search router). Every MCP call is captured by an HTTP reverse proxy that transcribes JSON-RPC and text/event-stream traffic without touching the shared daemons (same PIDs before and after; verified). The wave-5 arm-b REPORT §6.1 originally called for a stdio proxy — that was the wrong shape: arm (b) wires MCP streamable-HTTP, not stdio. The corrected HTTP proxy lives at nexartis-auggie-harness/probes/arm-b/mcp-transcript-proxy.mjs and its usage contract at nexartis-auggie-harness/probes/arm-b/PROXY-NOTES.md. Retrieval-recall on arm (b) / arm (c) is computed from the per-run mcp-transcript-summary.json — the shim is what unblocked per-run retrieval-path attribution, without which no statistical arm-b / arm-c comparison was possible.

Falsification threshold — 0pp (LOCKED, never post-hoc adjustable)

Publication (D4) — INTERNAL-ONLY for now · pre-registration is mandatory

Operator ruling D4 (2026-07-29): internal writeup only for now; a public-release decision is deferred until after results are in. This carries a known risk that was flagged by the orchestrator and accepted by the operator — deciding to publish only once results look favourable is precisely the selection bias pre-registration exists to prevent, and an outside reader can reasonably infer it from a "decision made post-results" timeline.

Mandatory mitigation (Phase-1 hard blocker, not a suggestion; reinforced in three places across BAKEOFF-DESIGN.md — §5.5, §7 risk 11, §8.9): before the first metered Phase-1 pilot run, the following artifacts are committed to git and their commit hash is recorded in BAKEOFF-DESIGN.md as the pre-registration lock —

1. the hypothesis list (H0 / H1 per KPI × arm-pair),
2. the analysis notebook stub (pre-registered tests, Holm-Bonferroni correction logic, per-stratum + per-contamination-class report skeletons),
3. the 0pp falsification criterion,
4. the "What this deck did NOT prove" section boilerplate.

The commit timestamp is the evidence-of-priority that preserves the option to publish externally later without the post-hoc selection charge. No timestamped pre-registration commit ⇒ no external publication, ever.

This page is INTERNAL research documentation — served on internet-exposed Workers behind Sentinel with roles: ['admin'], ungated only on loopback. External publication of the numbers, methodology, or results is a separate gated decision downstream of the pre-registration commit, per D4.

Data model — Phase-2 experiments schema

KPI system — Ed25519 receipts + dual evaluation

Every turn emits a signed receipt: run ID, arm, task, orchestrator/subagent turn, tool calls, artefact hashes, wall-clock, tokens, cost. The receipt is the anchor for both evaluation tracks — the agentic evaluator and the human raters read the same artefact. Nothing that lacks a signed receipt is scored, because we cannot verify what happened.

• Agentic evaluator (rubric-driven) — A KYM-registered evaluator agent scores each orchestrator/subagent turn against the shared rubric. The evaluator is itself versioned as a KYM agent — same publication and receipt loop as the harnesses under test. Rubric drift is a versioned event, not silent.
  Mechanism: One receipt per turn, one score per rubric item, one overall score. Score payload is signed alongside the turn receipt.
• Human evaluations (ozzydev UI) — Human raters score the same turns in the ozzydev UI: 1–10 gradient per rubric item + an overall 1–10 per orchestrator/subagent turn. Quick filter drops the rater into the descendant subagent turns of any parent turn without leaving the view.
  Mechanism: Same rubric as the agentic evaluator. Human scores anchor the agentic evaluator; disagreements become the training signal for the next evaluator version.

Rubric-design process — its own ozzydev UI

The rubric IS the reward function; the rubric-design UI is therefore a first-class surface in ozzydev, not a settings page.

1. Rubric items are drafted in the ozzydev rubric-design UI (its own surface, not buried inside the run view).
2. Each rubric item carries: name, one-sentence intent, scoring anchors for 1 / 5 / 10, and a link to the receipt fields it grades.
3. A rubric version is published to KYM the same way an agent version is — with a signed manifest.
4. Evaluator versions are pinned to rubric versions. A rubric bump implies an evaluator retrain or, at minimum, a re-anchoring pass on frozen human-scored turns.
5. The rubric IS the reward function. Bandit reward comes from the rubric-scored receipts; the rubric is therefore the most consequential artefact on this page.

Evaluator versioning loop

The agentic evaluator is itself a KYM agent: it has a version, a manifest, and receipts of its own. When rubric v(n) publishes, evaluator v(n) is trained (or re-anchored) against the frozen human-scored turn set for rubric v(n-1) → rubric v(n) drift. Disagreement between the agentic evaluator and human raters is the signal that trains the next evaluator; a shrinking disagreement rate over evaluator versions is itself a published metric.

Reward signal

• Base reward: overall 1–10 turn score (agentic evaluator, anchored to human raters).
• Cost penalty: normalised $ per completed task, subtracted at a rubric-defined weight.
• Latency penalty: normalised wall-clock per completed task, subtracted at a rubric-defined weight.
• Receipt-completeness gate: turns without a signed receipt are excluded from the bandit reward pool. Missing artefact ⇒ zero reward, not a penalty — because we cannot verify the outcome.
• Task-class bucketing: reward is aggregated per task class (search, edit, refactor, docs, config). The bandit picks a winning arm per class, not a single global winner.

Receipt boundary — external-harness arms

Ozzydev integrations required

• Runner (nexartis-parallel-waves-agent-harness) — Executes each arm; every engine adapter emits the shared receipt schema and stores the raw run artefacts alongside.
• Receipt store — Ed25519-signed receipts land in the ozzydev receipt store, indexed by run, arm, task, turn. Immutable; agentic evaluator + human raters read from here.
• Rubric-design UI — First-class ozzydev surface for authoring, versioning, and publishing rubrics to KYM. Not a nested settings page.
• Human-eval UI — 1–10 gradient per rubric item + overall 1–10 per turn; quick filter to descendant subagent turns; keyboard-first.
• Bandit dashboard — Per-task-class arm-selection log, cumulative reward, exploration/exploitation state, and the current winning arm — with the receipts that justify it one click away.

End-state — parallel-waves SDK as the KYM developer-program starter

1. nexartis-parallel-waves-agent-harness ships an SDK: the basic orchestrator + subagent template, plus the engine-adapter interface.
2. A KYM developer starts a program by cloning the template, keeping or replacing pieces, and publishing agent variants to KYM.
3. The bandit study runs itself on their variants against the shared rubric.
4. When a variant wins, the developer publishes it to our NANDA node — discoverable on the network — matching the KYM-vs-NANDA boundary tested in memory-tooling D11.
5. This makes the parallel-waves SDK the KYM developer-program starter: the shortest path from an agent idea to a discoverable, receipted, evaluated production agent.

Benchmark integrity — what we trust

SWE-bench Verified was invalidated in February 2026: an independent audit found that 59.4% of the hardest failing cases were broken tests, not agent errors. Every "SOTA on SWE-bench Verified" claim through that period is unsafe as a comparator for this study. Our tracking uses the SWE-bench Pro standardized-scaffold column as the honest line — the standardized scaffold strips harness-specific advantages that made Verified numbers non-transferable in the first place. When a vendor cites Verified without pointing at Pro, we treat it as a signal about the vendor, not about the model.

Phase 4 — bakeoff harness (landed 2026-07-31)

Phase 4 of ADR-013's native harness router landed on feat/harness-phase4-bakeoff (PR #13). The runner is a paired-task bakeoff over an explicit set of engine arms drawn from mcp/registry.json harnesses[]: each (task, arm) pair re-seeds a scratch git repo, runs the engine adapter, captures the diff, and runs the write-router's runLadder with the corpus-declared per-task verify commands. Design is locked to four operator rulings from 2026-07-31:

1. New runner file; the Phase-0 script stays frozen historical evidence. bench/bakeoff.mts + shared bench/lib.mts (extracted spawn / scratch / diff / experiment-upsert helpers plus a pure rotateArms) implement Phase 4. bench/run.mts imports the shared helpers, but its hardcoded round='phase0-w2' / waveSlug='harness-router-2' are preserved verbatim as the r5b defect-class evidence documented in ADR-013 §8 — never retroactively "fixed".
2. Arms are enumerated from the registry. --arms <slug,slug,…> is REQUIRED (no default-all); missing / empty / unknown slug = loud refusal listing valid slugs. The registry row picks up an optional bench_fixture data field (path relative to bench/) — data only, never a type identifier. The bakeoff reads registry.json directly so it always sees the full slate even when the harness config.json inlines a runtime subset.
3. Corpus-declared verify. bench/corpus.json entries carry verify: [{rung, cmd}] mapped onto runLadder's typecheck / lint / targeted_test / full_suite positional args. Commands are run in the scratch repo AFTER each arm's engine run — same commands, same order, every arm. {bench} and {ws} placeholders are substituted at runtime. Entries without verify record verify_step='none'.
4. Null-decision semantics. Results JSON always carries decision: null — the runner never declares a winner (analysis is downstream on the console). round is null when exactly 1 arm is selected (bakeoff-of-one is a smoke); when BENCH_EMIT=1 with ≥2 arms, --round <id> is REQUIRED (loud refusal fires BEFORE any console/signer I/O). wave_slug comes from an optional --wave arg, default null — never hardcoded (the r5b defect class does not recur).

Publication path. Every arm-run emits ONE ADR-009 signed receipt via /api/receipts/with-event (paired tool_benchmark_events row). Receipts key on paper_slug='harness-bandit-experiments' and variant_id=<arm-slug>. The experiment upsert uses slug harness-bandit-phase4-bakeoff (category 'harness-bandit'). Read-back: GET /api/experiments?paper_slug=harness-bandit-experiments.

Metered-run discipline. Fixture arms are free and run as part of pnpm run validate . Live arms require a per-arm N=1 de-risk first, a preregistration commit, then the N≥2 metered batch — never extrapolate spend from one outlier run; the batch fixes the cost model (workspace §5). Runners are tracked background_process-class jobs — never switch sessions mid-metered-run; the run's partial spend is lost if the session ends.

Phase 4 — phase4-batch-1 results (2026-08-01)

First metered, preregistered batch on the Phase-4 bakeoff runner. 80 arm-runs, 80/80 ok, 80/80 signed receipts; 40 paired (task_id, seed) units across 5 tasks × 8 rotations × 2 live arms, zero drops. Preregistration LOCKED before the run at commit fff4791 (v4, batch design in §4.8); analyzer completion + §2 condition-3 wording defect recorded at commit afa64ae (v5) under §2's design-flaw rule. Full prereg: mcp/servers/ozzydev-native-harness-mcp/bench/prereg/PREREG-phase4-2026-08-01.md.

Batch parameters (as locked in prereg §4.8)

• Round phase4-batch-1, wave harness-router-hardening-5.
• Live arms: A = cli.claude-code vs B = acp.auggie. Corpus: all 5 tasks (corpus-001…corpus-005).
• 8 sequential bakeoff.mts invocations, one literal Mulberry32 seed each: 20260811, 20260812, 20260813, 20260814, 20260815, 20260816, 20260817, 20260818 — 4 odd / 4 even, an exact arm-order counterbalance the 3-seed pilot could only approximate.
• 5 tasks × 8 rotations = 40 paired units; × 2 arms = 80 arm-runs. Every pair had both arms present; zero drops; 80/80 signed ADR-009 receipts emitted.

Result files (gitignored by design; cited by filename as the prereg §4.8.1 convention requires): bench/results-phase4-2026-08-01-msawqs7m.json (seed 20260811), -msawvhc1 (20260812), -msawy7vj (20260813), -msax15rz (20260814), -msax41hw (20260815), -msax6q3z (20260816), -msax9kfw (20260817), -msaxc7um (20260818), plus -msaxfuk3.json (the separate fixture-arm canary).

§2 verdict conditions — all three evaluated

New in this batch: all THREE §2 conditions are separately evaluated. The pilot's analyzer implemented only condition 1 and then emitted verdict: parity on the Holm p-values alone — a PARTIAL evaluation presented as a full one. The v5 amendment (§2 design-flaw rule) implements conditions 2 and 3 as separately-reported named checks; KPIs that are insufficient_n / not_observable are listed as explicit qualifiers rather than silently absorbed.

• Condition 1 — Holm-corrected p-values ≥ α = 0.05. Met on every computed paired KPI. Effective family size m = 2 (KPI (c) tokens_total dropped out per prereg §1.3 — acp.auggie reports no tokens on any pair).
• Condition 2 — bootstrap 95% CIs straddle zero. Percentile method, B = 10 000, seeded Mulberry32 PRNG, resampled statistic = MEDIAN (Wilcoxon-consistent, §3.2). Met on every computed paired KPI.
• Condition 3 — fixture-arm canary verify_pass = 1.0. The v5 amendment records the wording defect: the live arms never execute on the fixture-replay stratum (§3.5 keeps them disjoint), so a live-arms-only batch (which §4.1 locks) cannot evaluate the condition as originally written. The condition's INTENT — prove the runner/instrument is not broken — was preserved by a separate zero-metered-cost phase4-batch-1-canary invocation of cli.codex + acp.gemini-cli, which returned 10/10 verify_pass=true with deterministic fixture replay (-msaxfuk3.json). The live arms' own soundness is reported separately and directly: 80/80 live arm-runs verify_pass=true.

Paired-KPI results (§3.2 tests, Holm-corrected)

• KPI (a) verify_pass — McNemar exact. Status insufficient_n, reason no_discordant_pairs: 0 discordant pairs in 40. This is the pre-stated §3.2 outcome, NOT a p-value of 1.0 that could be misread as "tested and confirmed null".
• KPI (b) latency_ms — Wilcoxon signed-rank, two-sided. n = 40 pairs; p_raw = p_holm = 0.4638, q_bh = 0.4638; median paired difference (A − B) −1.5 ms; 95% percentile bootstrap CI [−478.5, +769.0] ms (B = 10 000, seeded Mulberry32) — CI straddles zero.
• KPI (c) tokens_total. Status not_observable: acp.auggie reports no tokens (or a null endpoint) on every pair, so the KPI drops out of the family and the effective m collapses to 2 per §1.3.

KPI (d) dollars_micro — single-arm descriptive on A. n = 40; mean 71,940 micro-dollars ($0.07194 / arm-run); median 68,980. Arm B classified not_observable (reports_cost=false) per §3.1; no cross-arm dollars claim is made (§5 boilerplate item 4, below, is non-optionally attached).

KPI (e) output equivalence — reproducible corpus-003 divergence

Per-task sha256(diff.patch), normalised per §3.1 (git index <hash>..<hash> header lines stripped): 4 of 5 tasks byte-identical across arms (corpus-001-add-function, corpus-002-add-subtract, corpus-004-json-edit, corpus-005-bash-script). corpus-003-python-edit DIVERGES — this is the SAME task that diverged in the pilot, so the divergence is a reproducible task-specific finding, not seed noise. Both arms pass their verify commands (KPI (a) unaffected); the divergence is descriptive-only per §1.2 KPI (e), and it is a candidate seed for a Phase-5 real-repo follow-up on the Python edit pathway.

Cost, wall-clock, and caps

• cli.claude-code leg: $2.8774 actual vs $2.878 pre-registered projection (40 × $0.071944 from the pilot's mean, prereg §4.8.5). Projection landed within $0.0006 of actual — the pilot's own variance sized the batch correctly, exactly as §4.3 requires.
• Wall-clock: ~22 min across the 8 sequential invocations vs the 45-min hard cap (§4.8.5) — well inside the ~1.8× headroom.
• $6 hard ceiling never approached; the running dollars_micro sum was ~$2.88 at completion.
• acp.auggie leg reports no dollars (reports_cost=false) and its under-the-hood credit burn is NOT recoverable from the receipt — the operator's Augment account statement remains the sole ground truth for that leg (§4.8.5, §5.4).

Order-of-operations honesty callout

What phase4-batch-1 did NOT prove (§5 boilerplate — LOCKED, verbatim)

Reproduced verbatim from PREREG-phase4-2026-08-01.md §5, per the §8.9 mandatory-mitigation lock inherited from the retrieval-probes HYPOTHESES.md §5.1. A Phase-4 result writeup without this section is invalid.

1. Corpus is 5 tiny synthetic tasks. Each is a single-file edit with a trivially-verifiable outcome (tsc, python3 -m py_compile, python3 -m json.tool, bash -n). It is NOT representative of real-repo work — no multi-file refactor, no test-suite reasoning, no failure-mode search, no cross-package contract. Any result generalises to "can an engine complete a trivial edit", nothing further.
2. Two live arms, one engine each. cli.claude-code and acp.auggie. No third live arm, no across-vendor triangulation, no head-to-head with our own harness. Absence of a significant difference between two arms on this corpus is NOT evidence for engine substitutability at scale.
3. Single model pin per engine. cli.claude-code uses whatever model the engine's default surface selects (observed: claude-sonnet-4-6 at the de-risk point). acp.auggie does not report its model. No model-mix, no version sweep, no temperature/params grid.
4. No cross-arm dollars. acp.auggie reports_cost=false; dollars are single-arm descriptive on cli.claude-code only (§1.2 KPI d). The operator-side auggie account statement is required for any cross-arm cost claim; this analysis makes none.
5. Fixture arms are excluded from headline. cli.codex and acp.gemini-cli are fixture-replay canaries only (§1.1, §3.5). Their outcomes are asserted to be deterministic; they are not evidence about their real engines' behaviour.
6. No generalisation to real-repo tasks. The corpus is scaffolded into fresh git repos with a single seed file. Real-repo tasks involve pre-existing context, dependent files, and non-trivial verify ladders (targeted_test, full_suite). A Phase-5 batch on real-repo tasks is the appropriate follow-on; this batch does not proxy for it.

Phase 5 — the bandit picker (LANDED)

What shipped. ADR-013 Phase 5 landed on feat/harness-phase4-bakeoff (PR #13, wave-3a, 2026-07-31): a Thompson-Sampling engine picker for task_class='auto' backed by per-(engine × task_class) Beta-Bernoulli posteriors, a deterministic named-rule classifier in config data, and a receipt-coupled reward composite ingested on every production run. Bandit picks ENGINE only; containment stays the deterministic 'auto' -resolver (only inproc / worktree are implemented — nothing to learn yet). Full spec: ADR-013 §7 Amendment 2026-07-31.

Algorithm choice — Thompson Sampling

Why Thompson. For Bernoulli-armed problems Thompson Sampling is asymptotically optimal (matches the Lai–Robbins lower bound; Kaufmann–Korda–Munos 2012, Agrawal–Goyal 2013) and beats UCB1 in every regime we care about (Chapelle & Li 2011). Practically it takes ONE Beta sample per arm per pick and needs no confidence-bound tuning, no epsilon schedule, and no hand-set exploration knob — exploration EMERGES from posterior widening on early data. That is the cold-start (OPERATOR RULING R4): uniform Beta(1,1) per (engine × class); as observations arrive the posterior tightens; there is no separate rubric-to-bandit switch or minimum- observations threshold.

Contextual variable. The context is the effective task_class (from the ADR-013 TASK_CLASSES enum). We keep one posterior per (engine, task_class) tuple; picks condition on the class. Contextual bandits with hand-crafted context slots are strictly more sample-efficient than unconditional ones when the arm ordering flips across contexts — which it does here: a CLI engine that wins on edit is not guaranteed to win on research.

Samplers. Marsaglia–Tsang for Gamma(α≥1), boost trick for α<1, Beta via ratio-of-gammas, Box–Muller for standard normals. All node stdlib (crypto.randomInt for seeding, Math.* for everything else) — no new npm dependencies. RNG is INJECTABLE (mulberry32 seedable for tests; a fresh 32-bit crypto draw per pick in production). Mean convergence is a tested property: 10 000 draws from Beta(2, 5) sit within ±0.02 of the analytic mean 2/7 (test/bandit.test.ts).

Reward shape v1 — linear-normalised composite

Composite (OPERATOR RULING R3). composite = clamp01(w_v · verifyPass − w_c · norm(cost) − w_l · norm(latency)). Ship weights are {verify:1.0, cost:0.5, latency:0.25} — verify pass dominates (the whole verify ladder exists to gate composites at 0.5); cost gets a half weight because we measure it on some engines and tombstone it honestly on others; latency gets a quarter weight because it's a proxy for user-perceived responsiveness but a weak proxy for task quality. Weights live in config.bandit.weights so re-tuning is a config change, not a code change.

Normalisation anchors. Anchors are computed per task_class cohort from the same reward history the posteriors load from. Three anchor states:

• 'cohort' — ≥ 2 observable samples in the cohort; norm(x) = x / cohortMax clamped to [0, 1].
• 'not_observable' — for cost, when the engine reports cost_source = 'engine_unreported' (OPERATOR DECISION B, 2026-07-28). Penalty stays 0, anchor recorded as not_observable. Never fabricated — the honest tombstone shows up in receipts and roll-ups.
• 'degenerate_cohort' — < 2 observable samples in the cohort. An anchor at cohort-max with n=1 has no statistical meaning as an upper bound, so we refuse to pretend and set the penalty to 0.

The cohort we anchor against is fetched via GET /api/harness-runs?include=rewards&task_class=<class> — a small, bounded read with a hard timeout. Fetch failure is LOUD-classified (harness_engine_unavailable) — the bandit path doesn't fall back silently on a cold console (workspace §2). Anchors, weights, and notes are recorded on the run row's detail.reward_anchors and echoed into the decision inputs_json so a downstream analyst can reproduce the composite from the persisted evidence.

'auto' IS the opt-in — the shape-of-the-seam ruling

Locked scope decision. The Phase-2 refusal on task_class='auto' is removed; that string IS the caller's bandit opt-in. Declared task_classes keep the Phase-2 deterministic rubric UNCHANGED (byte-identical decision reason strings + rubric row + alternates_considered=[]). This preserves the two contracts our downstream analytics rely on:

1. Existing operators can keep asking for what they know they want — the harness never fights the caller. There is no coercion of a declared class into 'auto'.
2. The bandit surface is a strictly-additive change to the decision-making seam. Signed receipts, receipt schema, and D1 row shapes are unchanged for declared-class runs — the only new field on those rows is the composite reward, which was already reserved by the Phase-1 schema.

Classifier design. config.classifier_rules[] — an ordered array of named rules. Each rule has {name, task_class, intent_patterns[], path_patterns[]}. Patterns are case-insensitive substrings by default, promoted to RegExp with a re: prefix. First-match wins in array order. Structurally bad rules (empty intent AND empty path patterns — the silent-default-class trap) fail LOUD at router construction time. No-match on 'auto' = LOUD harness_request_invalid naming rules_considered — the operator sees exactly why the classifier refused. There is NO LLM anywhere in this path. The shipped table covers the 8 TASK_CLASSES with 1–4 patterns each; the table is DATA, not code, and evolving it is a config-only change.

Alternates_considered — the contract

Every bandit run's decision.alternates_considered carries the full pick evidence per the ADR-013 §1 schema {engine, score, skipped}:

• Every non-chosen live arm: its Beta-sample as score + `skipped:'lower_sample'`.
• Every excluded registry row: score 0 + one of 'fixture_replay_provenance' (the Phase-4 fixture-replay rows are NEVER arms), 'not_live', or 'class_unsupported'.
• The chosen arm never appears in alternates_considered — it's named in decision.engine and its sampled score is echoed in decision.reason: classify:<rule-name> → thompson chose <slug> (sample=<n>). rubric_row is null on the bandit path.

This is the format downstream regret / lift / churn analytics consume without pre-processing.

How this page stays current

Living methodology. The harness slate, rubric versions, evaluator versions, and per-task-class winning arms all update here in place. Numbers replace claims as they arrive; the version at the top bumps on every material change.

Changelog

• v1.1 · 2026-08-01 — Phase 4 phase4-batch-1 LANDED and analyzed on feat/harness-phase4-bakeoff. 80/80 arm-runs ok, 80/80 signed ADR-009 receipts, 40 paired (task_id, seed) units, zero drops; seeds 20260811–20260818 (4 odd / 4 even ⇒ exact arm-order counterbalance). Verdict: parity on this corpus — all three §2 conditions met — with mandatory power framing: preregistered MDE 890 ms at power 0.80 (Holm α/m = 0.025, m = 2, Wilcoxon ARE 0.955), so "no latency difference ≥ ~890 ms detected", never "engines equivalent"; the pilot's dz = −0.125 was rejected as a sizing target (≈503 pairs). KPI (b) latency p_holm = 0.4638, median paired diff −1.5 ms, 95% percentile bootstrap CI [−478.5, +769.0] ms. KPI (a) insufficient_n (0 discordant pairs). KPI (c) not_observable (acp.auggie reports no tokens). KPI (d) descriptive on A only: mean 71,940 μ$ / median 68,980 μ$. KPI (e): 4/5 byte-identical; corpus-003-python-edit reproducibly diverges (same task as in the pilot). Cost: cli.claude-code $2.8774 actual vs $2.878 preregistered projection — pilot variance sized batch within $0.0006. Prereg locked at fff4791 (v4) BEFORE the run; analyzer completion + §2 condition-3 wording defect at afa64ae (v5) under §2's design-flaw rule; §5 "What this batch did NOT prove" boilerplate carried verbatim per §8.9 mandatory-mitigation lock. Full prereg: mcp/servers/ozzydev-native-harness-mcp/bench/prereg/PREREG-phase4-2026-08-01.md.
• v1.0 · 2026-07-31 — Phase 5 bandit picker LANDED on feat/harness-phase4-bakeoff (PR #13, wave-3a). Thompson Sampling with per-(engine × task_class) Beta–Bernoulli posteriors (src/bandit.ts — Marsaglia–Tsang gamma + Beta-via-gamma-ratio, node stdlib only, no new deps); deterministic named-rule classifier for task_class='auto' in config.classifier_rules[] (src/classify.ts); reward composite v1-linear-normalised (weights {verify:1.0, cost:0.5, latency:0.25}) with cohort anchors (cohort / not_observable per OPERATOR DECISION B / degenerate_cohort); cold-start uniform Beta(1,1) — exploration emerges. Bandit picks ENGINE only; containment stays the deterministic 'auto'-resolver. decision.alternates_considered populated per schema; declared-class path stays byte-identical. POST /api/harness-runs accepts optional reward object; atomic 3-row batch; GET ?include=rewards left-joins the reward row for posterior loading. config_version → harness-router@0.0.4-phase5. Selftests: test/bandit.test.ts, test/classify.test.ts, test/router.bandit.test.ts, console/src/lib/server/harness-runs.test.ts — harness-mcp 82 → 115; workspace 947 → 1127 (2 skipped) green.
• v0.9 · 2026-07-31 — Phase 4 bakeoff harness LANDED on feat/harness-phase4-bakeoff (PR #13). New bench/bakeoff.mts + shared bench/lib.mts. Four operator rulings locked: (1) new runner file — bench/run.mts stays frozen Phase-0 evidence; (2) registry-driven --arms <slug,…> required, no default-all, with a new bench_fixture data field on mcp/registry.json harnesses[] rows; (3) corpus-declared per-task verify executed via runLadder with {bench}/{ws} placeholders — three new heterogeneous tasks (Python py_compile, JSON json.tool, bash -n) land alongside the TypeScript entries; (4) null-decision semantics — decision:null always, round null when N=1, --round required only for N≥2 EMIT runs, wave_slug from --wave never hardcoded. Publication path: per-arm-run ADR-009 receipt + paired tool_benchmark_events row via /api/receipts/with-event; paper_slug='harness-bandit-experiments'; experiment slug harness-bandit-phase4-bakeoff. Selftests: test/bakeoff.test.ts covers happy path, N=1 null-decision, unknown-slug + missing---arms + --round-required refusals (pre-I/O), Phase-0 regression, and rotateArms unit math — hermetic, no network, no vendor keys.
• v0.7 · 2026-07-30 — Retrieval-path bakeoff v2 LOCKED per operator rulings D1–D4 (2026-07-29). Deck sizing raised from n = 300 to n = 800 tiered (RH 400 / RM 250 / RE 150) with proportionally-sampled 300-task Phase 2a tranche + go/no-go gate then expansion to 800 — phasing is RISK control, not cost control. Statistical design at MDE δ = 0.10, α = 0.05 two-sided, power 0.80, paired ρ ≈ 0.6, Holm-corrected over m = 10 primary tests (paired bootstrap on B = 10,000 resamples; the intermediate n = 300 revision was itself superseded — it powered only the aggregate headline). Falsification threshold tightened from ≤ +2pp to 0pp, locked and never post-hoc adjustable. Publication ruled INTERNAL-ONLY for now with a mandatory pre-registration commit before the first metered run — no timestamped commit ⇒ no external publication, ever. Added §bakeoff-locked-design, §bakeoff-kpis (43-metric KPI set from KPI-SURVEY), §arm-a-observability (structural not_observable), §transcript-shim (the HTTP proxy that unblocked arm-b/c retrieval visibility — REPORT §6.1 called for a stdio proxy but arm-b wires MCP streamable-HTTP), §falsification-0pp, and §pub-preregistration. Phase 0 status flipped from "unbuilt" to BUILT (manifest schema + four generators executed with patch round-trip proven + R1–R4 continuous scorer + retrieval-recall + ground-truth verifier with import-graph reachability + crash-vs-fail triage; all verified against deliberately wrong patches). Cost ceiling lifted; ~744 kcr / ~5 billing cycles stated for record only. Zero credits spent in session 6; balance 177,564. Full spec: nexartis-auggie-harness/docs/BAKEOFF-DESIGN.md; metric contract: nexartis-auggie-harness/docs/KPI-SURVEY.md.
• v0.6 · 2026-07-27 — Promoted the auggie row from "Adapter pending" to measured v0.33.0 empirics (§auggie-empirics: working baseline, 34 default tools + 7 subagent tools, MCP-server surface = codebase-retrieval only, JSON envelope characterised, daemon = Cosmos on-prem worker, native AGENTS.md rules interop). Added §harness-security-caveats introducing containment class as a per-arm KPI (workspace-root not a sandbox, --ask does not prevent writes, tool-substitution footgun #74 reproduced). Added §retrieval-ablation as the headline experiment — hold harness constant and vary the memory stack, with honest gaps on cross-file tasks and continuous / retrieval-recall scoring. Added §receipt-boundary-external documenting the honest-attribution limit for external-harness arms (tokens + latency recoverable; cost + model not) and the Stop / PostToolUse hook bridge.
• v0.5 · 2026-07-16 — Confirmed v1 harness archived at nexartis-parallel-waves-archived (frozen); all wave launches on v2. Restated the implementation-wave gate as the machine-checked --tool-profile implementation --implementation-approved pair. Tightened the architecture-rule callout for publication.
• v0.4 · 2026-07-15 — Added the architecture rule: harness = operational code only; orchestrator/subagent definitions live in KYM as Agent Cards + versioned skills. Introduced the migration ledger and KYM-surface facts.
• v0.3 · 2026-07-13 — Tier-2 dark-horse slate finalised at five arms (Kiro, Junie, Factory Droid, Grok Build ×2). Bottom-line callout switched to 15 arms + one control.
• v0.2 · 2026-07-12 — Library section aligned with memory-tooling D10 (single library + engine adapters). Repo renamed to nexartis-parallel-waves-agent-harness.
• v0.1 · 2026-07-12 — Initial methodology publication.