A Common Experimental Framework for Agent Tooling

A unified category × KPI × experiment-mode matrix for agent tooling: six categories, the KPI triad, five experiment modes (including speculative-parallel N-worktree fan-out — a literature gap), one common recording schema landing in the console D1 `tool_benchmark_events` table, and two concept registrations for the ozzydev-native MCPs that operationalise it. The v0.3.0 revision folds the measurement discipline made concrete by the auggie-harness bakeoff work (pre-registration + falsification-criterion locking + power-analysis-before-sample-size + not_observable + continuous scoring + scorer verification + crash-vs-failure separation) into the framework itself, so every future experiment inherits it.

Problem — the missing unified framework

The literature has strong per-category evaluation frameworks — retrieval benchmarks for search, SWE-bench for code writes, MemoryBench for longitudinal memory — but no unified category × KPI × experiment-mode matrix that lets a workspace run apples-to-apples experiments across tooling categories. This paper opens that ground.

Category × KPI matrix

Every tooling category has a primary KPI, a secondary KPI (usually latency and cost), and an explicit verify pairing. Verify is never merged into accuracy — it earns its own column because its own optimisation surface (the verify ladder) is one of the experiment modes below.

The KPI triad — recorded every time

• Latency — Wall-clock from request to usable result. Recorded per stage where useful (e.g. write_ms vs verify_ms).
  Everything the agent does eats a turn budget. Latency is what the agent feels.
• Accuracy — Category-specific — accuracy@k for search, applies+validates for write, task-success for scripts, error-catch-per-latency for verify, retrieval@k for memory, wall-clock-to-green for orchestration.
  The correctness axis; without it, latency and cost optimise for the wrong outcome.
• Cost — Recorded as (tokens_in, tokens_out, dollars). Dollars is the fair comparison across models and vendors when token counts diverge.
  The economic axis. Turn budgets are dollars; agents that pass validate on the cheapest path win over time.

Measurement discipline — the framework requirements

A KPI matrix and a recording schema are necessary but not sufficient. What makes an experiment defensible — as opposed to merely recorded — is a set of measurement disciplines that bind the analysis before the data exists. The auggie-harness bakeoff work (nexartis-auggie-harness/docs/KPI-SURVEY.md and nexartis-auggie-harness/docs/BAKEOFF-DESIGN.md) made these concrete in one instance; the requirements below are the framework itself and apply to every future experiment.

Observation-derived KPIs

The rubric is a ratchet (workspace AGENTS.md §5). When a session surfaces a measurable failure pattern — empty subagent reports, sequential-edit spam, lost worktree work, and their kin — it is promoted into a KPI here so every future experiment measures it. The list grows; entries do not silently leave.

• empty-report rate — Fraction of subagent dispatches whose final message ends on a tool call or is otherwise report-less. Target: 0.
  Reports lost to a trailing tool call are indistinguishable from success at the orchestrator; the rate quantifies the crash-log C3 pattern.
• sequential-edit spam — Count of consecutive single-line native edits to the same file within one wave when the write-rubric routes to a batched tool (morph / weave). Target: 0.
  Direct rubric violation for D8 categories (b) and (d); measurable per session from the tool-event stream.
• lost-worktree-work rate — Number of orphan worktree artifacts at wave close (unpushed commits, `git stash` entries, untracked files) per wave. Target: 0.
  Parallel fan-out is only cheap if merge-and-GC is disciplined; drift here is invisible until a rerun is needed.

Five experiment modes

Modes are opt-in per dispatch; defaults never enable them silently. Speculative-parallel N-worktree fan-out is the mode we believe is a literature gap — search and generation have published speculative-parallel patterns, but write / skills / orchestration lack a common framing.

1. baseline — Single tool, current defaults. Establish the honest floor before touching anything. Every category has one before A/B numbers land.
2. A/B tool swap — Swap one tool for one alternative, keep everything else fixed. The classical experimental control shape.
3. speculative-parallel N-worktree fan-out — Fire N candidates (writes, skills, orchestration plans) in parallel worktrees; a verifier picks the winner. This is a literature GAP — search and generation have speculative-parallel patterns published; write / skills / orchestration do not have a common framing yet. Builds on worktree-mcp (D15) + sub-second sandboxes (Cloudflare Code Mode).
4. verify-ordering pareto — Systematically vary the order of the verify ladder (typecheck first vs lint first vs targeted test first) and plot error-catch-per-unit-latency against total verify_ms. Produces a pareto frontier the routing rubric can consume.
5. progressive-disclosure ablation — Ablate what the agent sees before opening a skill body (name-only vs name+description vs name+description+when-to-use). Measures how much discovery quality depends on the always-loaded slice — direct signal for the skills library index design.

Common recording schema

Every experiment row has this shape. The schema is the contract that makes cross-category aggregation possible; without it, results are prose.

• experiment_id — Stable id for this experiment definition. Lets us pool across runs.
• category — One of Search / Write / Scripts / Verify / Memory / Orchestration.
• mode — One of baseline / a-b-swap / speculative-parallel / verify-ordering / progressive-disclosure.
• variant — Free-form label for what changed in this run vs baseline (e.g. "codesearch → moe-router").
• inputs — The exact input(s) the experiment ran against (task id, gold-set slice, seed).
• latency_ms — Wall-clock per stage plus a total. Never a single number without a stage decomposition when it exists.
• accuracy — Category-specific accuracy score with the metric name explicit (e.g. "accuracy@5 = 0.82").
• cost — { tokens_in, tokens_out, dollars, model }. Dollars mandatory for cross-vendor comparability.
• verify_result — The paired verify outcome — which verify step caught what, or "clean". Never merged into accuracy; verify is its own column.
• receipt_id — Signed receipt reference (same store as harness bandit + moe-search-router). Ties the row back to an inspectable artifact.
• notes — Free-form. Where the anomaly explanation goes; not required for the aggregate to be valid.

Datastore realisation: this shape lands in the console D1 datastore as tool_benchmark_events (canonical spec: docs/SCHEMA-DESIGN-console-d1.md §3.2.7). Experiments are keyed by paper_slug — read-back via GET /api/experiments?paper_slug=…, event read-back via GET /api/telemetry/events?session_id=…. The harness-bandit study and the search-router enrichment flywheel both read and write this table; a single row is the atomic unit of evidence across every experiment mode above. The bakeoff / IR-benchmark experiment shape is production-proven — write-mcp rounds r2–r5 and the search-mcp IR harness (ranx-scored) both emit against this schema.

Write-KPI insight — write is a WRITE + VERIFY pair

The write category deserves its own subsection because its KPI is a pair, not a scalar, and the verify half is where most of the optimisation lives.

• Writes are measured as WRITE + VERIFY pairs, never in isolation.
  A write that applies but fails validate is not a success. A write that fails to apply but the agent recovers on the next turn is not the same as one that never applied. The pair captures both.
• The interesting question is which verify step catches most errors per unit latency.
  The verify ladder (typecheck → lint/AST → targeted test → full suite) has increasing latency and increasing coverage. The ratio (errors caught / verify_ms) is where the routing rubric earns its rank. Verify-ordering pareto (mode above) is designed to produce this frontier.
• CodeAct execution-loop is the strongest single verify signal.
  Yang et al. 2402.01030 showed +20% task success / −30% turns from making execution the loop feedback. When latency budget allows it, executing the change and observing the result beats every static analyser.
• LLM-as-a-Verifier gives continuous scores for ranking parallel candidates.
  arXiv 2607.05391 formalises LLM-as-Verifier over continuous scores rather than binary judgements. In speculative-parallel mode the verifier RANKS candidates; a continuous score is what makes "pick the winner" well-defined.

Concept registrations — the ozzydev-native MCPs

Two components operationalise this framework. Both live in the ozzydev-native surface.

• ozzydev-native-search-mcp — ozzydev-native-search-mcp — the MoE search router.
  Scope: Search algorithmic optimisation. This is where the search-side experiments live: classifier evolution, expert weighting, provenance-tier tuning, and the "one expert suffices" branch of the router.
  Discipline: Cite-all-prior-art discipline — every experiment cites the prior work it is building on (RRF, WarpGrep, FAS, Sourcegraph MCP, LangChain routers, MoE literature). Findings that duplicate prior art without extending it are labelled reproductions, not novel results.
• ozzydev-native-write-mcp — ozzydev-native-write-mcp — multiplexed write + verify across parallel worktrees.
  Scope: Write algorithmic optimisation. Multiplexes writes across parallel worktrees, pairs each write with a verify step, and lets the orchestrator run sprints of pure write experiments to measure latency + accuracy against the write-strategy rubric (D8).
  Discipline: Builds on worktree-mcp (D15) + sub-second sandboxes (Cloudflare Code Mode). Experiment mode: the orchestrator runs sprints of pure write experiments measuring latency + accuracy across the rubric categories (a)–(h). Findings feed the rubric itself.

Core-workflow rule — experiments out of band

The whole framework is worthless if it slows down the agent doing user-visible work. The following rule is load-bearing:

1. Experiments run OUT of band from the core workflow. The agent doing user-visible work is not the agent running an A/B swap on itself.
2. No latency-adding operations in the hot path — experiments feed the receipt store, receipts feed the tuner, the tuner adjusts defaults offline. The agent-in-flight sees the current defaults, not the experiment overhead.
3. The two ozzydev-native MCPs are the ONLY components allowed to run experiments during a user session, and only in speculative-parallel mode where the verifier picks the winner before the user-visible result lands.
4. Every experiment mode above is opt-in per orchestrator dispatch. Defaults never enable them silently.

Companion pages

• MoE Search Router — the search-side counterpart
• Write Strategies — the write-side rubric (D8)
• Harness Bandit Experiments — the bandit + receipt infrastructure we reuse
• Scripts as a First-Class Tool Category — the skills library that produces the discovery-side experiments
• Memory Tooling Decisions — the decision log (D15 worktree-mcp foundation)

Changelog

• v0.3.0 (2026-07-30) — Measurement-discipline section added, made concrete by the auggie-harness bakeoff (KPI-SURVEY + BAKEOFF-DESIGN): pre-registration as framework requirement (hypotheses / analysis plan / falsification criterion / "what this did NOT prove" hash-locked BEFORE first metered run — commit timestamp is evidence of priority); falsification criteria strictest-defensible and never post-hoc adjustable (0pp precedent when we authored the deck and one arm); power analysis BEFORE sample size (paired design, argued p₀ + MDE, α, power, correlation, Holm-Bonferroni / BH — a cost-shaped n is not a statistical result and is re-derived when the constraint disappears); the un-computable is structurally un-representable (explicit not_observable, never a coerced zero); continuous scoring in [0,1] over binary pass/fail; scorer verified against deliberately-wrong input at Phase-0; infrastructure crashes separated from genuine failures in every success-rate denominator. Observation-derived KPIs intro now references AGENTS.md §5 explicitly. The bakeoff work is one INSTANCE of this framework; these requirements are the framework itself and apply to every future experiment.
• v0.2.0 (2026-07-16) — framework in production use. Bakeoff / IR-benchmark shape proven (write-mcp r2–r5, search-mcp IR harness with ranx). Added observation-derived KPIs as a rubric ratchet. Datastore paragraph names the canonical spec (docs/SCHEMA-DESIGN-console-d1.md §3.2.7) and the paper_slug read-back contract.