MoE for Search — a Structured Search-Router MCP

One question in, one fused answer out. A single MCP that classifies the question, fans out to the right search experts in parallel, fuses ranked evidence with per-item provenance, and emits a receipt of who fired for tuning. MoE applied to search tooling: only the relevant experts run, and the interface itself teaches agents to decompose queries.

Problem

Agents burn turns making sequential, single-tool searches. Our workspace already exposes 6+ distinct search systems — semantic (chunkhound), hybrid (codesearch), typed symbols (serena), typed graph (SCIP/dora), polyglot graph (CoreGraph), text (grep), and product docs (external-docs / cloudflare-docs / svelte) — each optimal for a different question class. Today the routing rubric lives in agent instructions (see the D13 rubric in memory-tooling-decisions), which is better than nothing but still means every turn pays a decision-and-latency tax and every agent re-derives the same rubric.

The pathology is not that any one tool is weak; the pathology is that a good answer often needs corroboration across tools, and that corroboration lives in the agent's head instead of in a receipt. We lose signal we already have.

Proposal — a single Search-Router MCP

"One question in, one fused answer out." The agent submits a structured request; the router does the composition and the fusion; the agent gets one ranked, provenanced answer plus a receipt.

Structured request

• question (required) — The natural-language question. Not a query string — the agent submits intent, the router expands to expert-specific queries.
• intent (optional) — Optional enum: conceptual · identifier · typed-symbol · cross-repo-impact · docs · history · unknown. Hints let the classifier skip work; unknown is a valid answer.
• scope (optional) — Optional { project?, group?, path?, language? }. Missing scope = workspace-wide fan-out; specifying scope prunes experts before they fire.
• budget (optional) — Optional { latency_ms?, max_experts?, min_precision_class? }. Turn-level backpressure so the router can decline experts that will not answer within budget.
• expected_shape (optional) — Optional enum: ranked-evidence · single-file · symbol-list · answer-with-citations. Shapes the fusion output, not the fan-out.

The expected_shape field is deliberately narrow. Search should return evidence, not conclusions; conclusions are the agent's job. A "single-file" or "symbol-list" shape is a hint about how to rank + present, not a licence for the router to summarise.

Pipeline

1. 1 · Classify — Cheap local rules first: intent hint (if provided) + a small ruleset over the question (regex/quoted-identifier → lexical; camelCase/snake_case token → typed-symbol; "who calls / who consumes" → cross-repo-impact; library-name → docs; "when did … land" → history). LLM escalation only when local rules are ambiguous, and only on request via a budget flag.
   Failure semantic: Classifier abstains → routes to a documented default expert set (chunkhound semantic + regex + codesearch). Abstention is reported in the receipt; the agent sees it, the tuner sees it.
2. 2 · Select experts (MoE gating) — Given the class + scope + budget, the gate picks the subset of experts to fire. This is the MoE analogy that actually holds: only relevant experts run. Weights are per-class priors, not learned yet.
   Failure semantic: If the gate would pick zero experts (e.g. malformed scope), it fails loudly with a structured error; it does not silently pick "all".
3. 3 · Fan out in parallel — Every selected expert query runs concurrently. Timeouts are per-expert against the budget; a slow expert does not block a fast one.
   Failure semantic: Per-expert error (timeout, connection, index unavailable) is captured and reported in the receipt. The expert contributes zero results, never fake ones. No retry-with-degradation.
4. 4 · Score per-rubric — Each result gets (a) an expert-native relevance score, (b) a precision-class score derived from the source tool (typed > hybrid > semantic > lexical), (c) an index-freshness score (last successful index + branch match), (d) a scope-match score.
   Failure semantic: A missing sub-score is a missing sub-score — the result carries a nuance flag ("freshness unknown") rather than being dropped or filled with a fake value.
5. 5 · Fuse — Reciprocal Rank Fusion baseline (RRF, k=60) across experts, weighted by per-class expert priors. Dedup by (path, symbol, chunk-hash). Cross-expert corroboration boosts confidence (e.g. serena + chunkhound-regex + scip agreeing = "typed-graph confirmed" tier).
   Failure semantic: Fusion output annotates each item with its source set. Nothing appears in the fused list without a documented origin.
6. 6 · Return — Ranked evidence list, each item carrying: content, path/range, source expert(s), precision-class tier, freshness, and a plain-language nuance line ("typed-graph confirmed" vs "lexical guess" vs "semantic-only, verify"). The agent sees quality at a glance.
   Failure semantic: Empty result set is a valid answer — reported with the expert set that ran.
7. 7 · Receipt — Signed receipt of what fired: experts selected, per-expert latency, per-expert result counts, classifier decision, budget consumption, fusion weights used, error set. The receipt is the tuning signal.
   Failure semantic: Receipt is mandatory. A request that cannot emit a receipt returns an error; there is no unlogged path.

Provenance tiers — the nuance is the product

Every returned item carries a provenance tier and a plain-language nuance line. This is the main way the router creates value the underlying experts cannot create alone: it turns cross-tool agreement into a first-class trust signal.

• typed-graph confirmed — Result appears in ≥2 typed sources (serena, SCIP/dora, CoreGraph) with agreement. Highest trust.
• hybrid-confirmed — Codesearch hybrid + at least one other expert agree. High trust.
• lexical-exact — Regex/grep exact match; may or may not be the semantically relevant hit. High precision, unknown recall.
• semantic-only — Only chunkhound-semantic returned it. Useful lead, requires verification. Marked "verify" in the nuance line.
• docs — Sourced from a docs expert (external-docs — context7 today; cloudflare-docs, svelte). API-current claim.
• stale-index — Result comes from an index whose last successful build is older than the current branch HEAD. Freshness flag surfaces this to the caller.

Search as an enrichment flywheel

• Cross-modality cross-pollination — When semantic returns file F and symbol S resolves to the same file, the router writes a "semantic↔symbol confirmed" hint back into a cross-index hint store (path + symbol → provenance tier). Next-turn queries that touch F or S skip a modality if a corroborating hint is already recorded.
  Every turn silently makes the router smarter. The hint is a first-class row, not an in-memory heuristic.
• Turn-scoped cache (positive + negative) — Fused answers keyed by (question-hash, scope-hash, config-version). Negative-result caching too: if the fan-out returned nothing for a given (question, scope), the next identical query short-circuits with a cache-hit receipt.
  Sequential agent turns often ask the same question with tiny wording variants. Caching the fused answer + provenance saves the whole fan-out.
• Session-scoped bandit warmup — Within a session, per-class expert priors get incremental updates from receipts. The bandit reward comes from downstream turn success; the router promotes the arm that keeps winning inside a session before persisting to the global prior.
  Session-local adaptation without waiting for a global retrain — the fastest path from "who wins here right now" to actual behaviour.
• Benchmark-event emission (measurement substrate) — Every fan-out writes one row into `tool_benchmark_events` (the Phase-2 experiments schema — canonical spec: `docs/SCHEMA-DESIGN-console-d1.md`). Fields: run_id, arm/expert_id, task_class, question_hash, latency_ms, result_count, precision_class, hit/miss, config_version. The same store the harness-bandit study reads: one substrate, two consumers.
  Enrichment is a claim; a benchmark row is evidence. This is how "the router got smarter" becomes a measurable KPI delta.
• Recursive self-improvement loop — Receipts + hints + cache + bandit priors feed each other: a receipt tells the bandit which expert won; the bandit adjusts priors; adjusted priors shift the fan-out set on the next turn; the new fan-out writes new hints; the hints prune future fan-outs. The loop closes without human intervention.
  Search is every-turn work; a self-improving search router compounds across every turn every agent makes.

Target flow — sequence diagram

The diagram traces one structured request from an agent through the router: cache lookup by (question, scope, config) short-circuits on hit; otherwise the bandit gates a subset of experts that fan out in parallel, results are RRF-fused with cross-expert corroboration into provenance tiers, and the agent receives the fused answer plus a signed receipt. In parallel, fire-and-forget enrichment writes push the answer into cache, benchmark rows into tool_benchmark_events, cross-index hints into the hint store, and reward signal into the session bandit — closing the loop so the next turn is cheaper and more accurate.

sequenceDiagram
    autonumber
    participant Agent
    participant Router as Search-Router MCP
    participant Cache
    participant Bandit as Session bandit priors
    participant Experts as Expert pool (semantic · lexical · symbol · docs)
    participant Fusion as RRF + provenance
    participant Store as tool_benchmark_events + cross-index hints (D1)

    Agent->>Router: structured request (question, intent?, scope?, budget?)
    Router->>Cache: lookup (question-hash, scope-hash, config-version)
    alt cache hit
        Cache-->>Router: fused answer + provenance
        Router-->>Agent: fused answer + receipt (cache-hit)
        Router-)Store: benchmark event {hit=true, latency, arm=cache}
    else cache miss
        Router->>Bandit: gate → subset of experts (per-class priors)
        par parallel fan-out
            Router->>Experts: fan out (only relevant experts run)
            Experts-->>Router: ranked results + per-expert latency/errors
        end
        Router->>Fusion: RRF + cross-expert corroboration → provenance tiers
        Fusion-->>Router: ranked evidence + nuance lines
        Router-->>Agent: fused answer + signed receipt
        par async enrichment writes (fire-and-forget)
            Router-)Cache: put (question-hash → fused answer)
            Router-)Store: benchmark event row (arm, latency, precision_class, ...)
            Router-)Store: cross-index hints (semantic↔symbol confirmations, path/symbol edges)
            Router-)Bandit: reward signal from downstream turn success
        end
    end

    Note over Store,Bandit: Next-turn improvement:<br/>hints prune fan-out, cache short-circuits repeats,<br/>bandit shifts priors toward the arm that keeps winning.

Measurement substrate: tool_benchmark_events is defined in the Phase-2 experiments schema — canonical spec docs/SCHEMA-DESIGN-console-d1.md; see also experimental-framework for the wider recording schema. The same table backs the harness-bandit study.

Architecture sketch

The MCP is small on purpose. Every part is inspectable, versioned, and receipts-visible.

• Classifier — Local rules first, LLM escalation on budget flag. Publishes its decision + confidence into the receipt so tuning can see when rules are winning and when the LLM is worth its cost.
• Expert adapters — One thin adapter per underlying MCP client (chunkhound, codesearch, serena, SCIP/dora, CoreGraph, grep, external-docs [context7 today], cloudflare-docs, svelte). Adapters normalise results into the shared evidence schema and never hide errors. Note: the chunkhound expert is off pending D4; the semantic slot is filled by claude-context (Zilliz) + Sourcebot in the interim.
• Fusion layer — RRF + per-expert-prior weights + dedup + cross-expert corroboration → provenance tier. Deterministic; the same inputs produce the same output. Configurable weights live in a versioned config file; changes are receipts-visible.
• Provenance/scoring model — Maps (expert-native score, precision-class, freshness, scope-match) → final rank + nuance line. Documented, inspectable, human-readable.
• Receipts + tuner — Ed25519-signed request/response receipts stored alongside our existing receipt store. A tuner reads them to compute per-class expert priors and (later phase) bandit weights. Same infrastructure as the harness bandit study.
• Config surface — Versioned config for expert set, per-class priors, budget defaults, fusion weights. Config bumps are receipts-visible; the receipt records the config version that ran.

Expert pool — the current line-up

The router fronts the experts we already run. Each expert exposes what it is good at and what precision class its answers carry — the classifier + gate use exactly these fields.

Failure semantics — no hidden behaviour

The workspace no-defense-in-depth principle applies literally: an expert failing is a classified, reported failure, not a silent degradation. The receipt is the contract.

• Expert timeout → captured in receipt, zero results from that expert, no retry-with-degradation. The caller sees which expert did not answer.
• Expert unavailable (daemon down, index missing) → same treatment; the receipt names the missing expert.
• Stale index (last successful index < branch HEAD commit) → results still returned, tagged "stale-index" in the nuance line. The caller decides whether to trust.
• Classifier abstention → routes to the documented default expert set; abstention is a receipt field, not a hidden fallback.
• Zero-expert gate → structured error; never silently expands to "all experts".
• Empty fused result → valid answer; the receipt shows what fired and returned nothing. Empty is signal.
• Config drift (weights or expert set changed since last run) → receipt records the new config version so tuning is not confused.

Prior art — honest comparison

The novel bits are (a) MoE-style parallel fan-out with typed vs lexical vs semantic provenance, (b) receipt-driven per-class expert priors, (c) failure semantics as a first-class output. RRF is the fusion floor; the rest is stack-native.

• Reciprocal Rank Fusion (RRF) — Baseline fusion function. codesearch already does RRF inside a single tool (BM25 + vector). Our router extends RRF across tools with different precision classes and adds cross-expert corroboration → provenance tiers. RRF is the floor, not the ceiling.
• Morph WarpGrep + Relace Fast Agentic Search (FAS) — RL-trained search subagents. They optimise the search LOOP — an agent that keeps searching until it finds the answer. We optimise the FAN-OUT — one call issues the parallel expert set. Complementary, not competing: a WarpGrep-style subagent could be a client of this router and benefit from structured multi-expert answers per iteration.
• Sourcegraph MCP + cheap-model result — Sourcegraph published that a cheap model routed through an MCP with the right tools beats a strong model without. That is the case for cheap classifiers + strong experts, which is exactly the shape here. We generalise from "one strong search tool via MCP" to "structured MoE fan-out via MCP".
• LangChain router patterns — Router chains pick one tool from many. Our router picks a SUBSET and fuses. The receipt+tuning loop is also new: LangChain routers rarely publish signed evidence of what fired.
• MoE literature (Shazeer 2017; Mixtral 2024) — Analogy limits: MoE gates in models are learned soft-max weights over dense experts sharing a common representation. Our experts are heterogeneous tools with different result schemas and different precision classes; the "gate" is a rule set + prior table, not a learned soft-max (yet). The analogy is useful for the FAN-OUT + SELECTION intuition; do not oversell it as literal MoE.
• CodeRAG-Bench — Benchmark for retrieval-augmented code generation across question classes — a candidate evaluation harness for this router. Its per-class breakdown maps cleanly onto our per-class expert priors.
• RAGSearch (arXiv 2604.09666) — Recent work on structured multi-source retrieval for RAG. Informs our provenance schema; we go further by making per-item source-tool nuance a first-class output rather than an internal signal.
• AIRCoder (multi-signal fusion) — Multi-signal RRF fusion for code retrieval. Prior art for our cross-expert corroboration step; we add the failure-semantic and receipt layers they do not.

Open questions

• Latency budget per turn — Fan-out is only a win if the slow expert does not blow the turn budget. Need per-class p95 latency data (chunkhound vs serena vs CoreGraph) before we lock defaults. Instrument first, tune second. (The chunkhound expert is benched pending the D4 bakeoff, so the semantic-slot latency baseline is claude-context/Zilliz + Sourcebot.)
• When does fan-out beat one good expert? — For a known-scope typed-symbol question, serena alone is probably enough. For a conceptual cross-repo question, three experts fused is worth the cost. The classifier needs a "one expert suffices" branch, not just an expert-set branch.
• Dedup across heterogeneous experts — A codesearch chunk, a serena symbol, and a chunkhound region that all point at the same code need one canonical identity. Path + byte range + symbol name is a starting key; edge cases (renamed symbols, moved files across branches) need care.
• How the structured interface teaches query decomposition — The agent submits one question; the router expands to N expert-specific sub-queries. Publishing the sub-queries back in the receipt shows the agent how a good router decomposes — this is a training signal for the agent, not just for the router. Explicit design question: which sub-queries are shown, and how they map back to the original.
• Per-expert confidence weights — v0.1 uses per-class priors from the D13 rubric. v0.2+ tunes them from receipts via a bandit — same infrastructure as the harness bandit study. Reward = downstream agent success on the turn, not fusion-internal metrics.
• Source reputational scoring (later phase) — Operator noted this as a later phase. Beyond precision-class and freshness, per-source reputation (e.g. a repo whose type coverage is thin gets its SCIP results down-weighted) is a natural extension once we have receipt volume to justify it.
• Bandit-tuned expert weights via KYM receipts — Fold into the harness bandit study infrastructure. The router is one more thing the bandit tunes — reward comes from the same receipt store, per task class.
• Failure of the MoE analogy — MoE models share a representation; our experts do not. Naming it "MoE for search" is a communication choice; the actual mechanism is structured multi-source retrieval with provenance. Do not let the name over-promise learned gating we have not built yet.

Positioning

1. Candidate ozzydev-native MCP component: sits alongside chunkhound/codesearch/serena in mcp/registry.json as a router that fronts them. Agents call the router; the router calls the experts.
2. Open-source opportunity: the router + adapter interfaces are generalisable beyond our stack. Publishing the reference implementation is a defensible position (structured provenance + receipts) that no current MCP-in-the-wild ships.
3. Commercial opportunity (operator approves publishing): the router-as-a-service surface is a natural product line — organisations get one MCP that fronts their whole search stack with quality nuance built in.
4. Sequencing: v0.1 architecture (this page). v0.2 reference implementation on our own expert set. v0.3 publish paper + open-source repo. Any external publication needs operator sign-off first per workspace convention.

Reclaim targets — the founding inspiration

Augment's discontinued VS Code extension is the founding inspiration for this platform. The operator used it every day and lost it when it was sunset (marketplace notice: extension sunset 2026-07-01, enterprise-only maintenance thereafter). What follows is therefore not a competitor scoreboard — it is a capability we are reclaiming, written as concrete, falsifiable numbers we own hitting. See large-codebases for the engine architecture and the extension's history; not duplicated here.

Latency & freshness — a separate axis from relevance

Retrieval quality and retrieval freshness are different axes. Our benchmark harness scores the first well and the second not at all. Augment's architecture is genuinely ahead on this axis — seconds-latency incremental indexing at "thousands of files/sec" and near-instant branch switches via per-developer indices — and the honest gap is that we do not currently measure most of what would make our own claim comparable. The table names each measurement we would have to add, not a number we have.

• Index build time vs repo size (curve, not a point) — not measured — Voyage-embedding-bound; no scaling curve captured across 100k → 10M → 100M LOC
• Incremental update latency after file save — not measured — claude-context walks a Merkle diff on save, but we have no p50/p95 wall clock
• Branch-switch reindex wall clock — not measured — the workspace tree is watched, not per-branch snapshotted; unknown behaviour on rapid switches
• p50 / p95 query latency, cold vs warm — p95 measured only at fixture scale (n=36, 25-doc corpus); no cold-vs-warm split; no 10M-LOC probe
• RAM / memory footprint of the local index — not measured — Zilliz collections live remote, so the on-machine slice is uncharacterised

This is flagged as work, not as parity. Adding these to the ADR-007 §9 measurement set is the follow-up sprint surface for the reclaim programme.

Techniques ranked by reproduction difficulty

The signal-to-effort ranking below is condensed from the in-workspace research packet (_WORKSPACE-DOCS/RESEARCH-AUGMENT-PERF-2026-07-28.md §5). Highest value × lowest cost first; the ordering is deliberate.

1. Context Lineage — commit-diff summaries embedded alongside chunks — "when did this land / why" answered from the index itself, not by re-reading git history each turn
   Cost for us: 🟢 cheap: batch `git log -p` through a Flash-class model, embed the summary as a chunk. Highest reproducibility × user-value in the list; ship this fast.
2. Quantized ANN + full-embedding rerank + hot-set fallback — ~40% latency reduction, 8× RAM cut, ≥ 99.9% parity, real-time coverage of just-written code that has not been quantized yet
   Cost for us: 🟡 moderate: Zilliz binary quantization is off-the-shelf; the "hot set" abstraction for un-quantized new writes is our lift.
3. Code-relationship-trained embeddings — recall on call-site ↔ definition, doc ↔ code, cross-language semantic pairs that generic embeddings miss
   Cost for us: 🔴 hard: labeled pairs + GPU budget. Voyage `voyage-code-3` + `rerank-2.5` is the credible off-the-shelf substitute; the gap is narrowing, not gone.
4. Per-developer real-time index (branch-agnostic) — correct results across branch switches with no monolithic reindex; incremental in seconds
   Cost for us: 🟡 moderate on-device: fs-watcher → local embedding worker → Merkle-diff. The industrial PubSub/BigTable plumbing is unnecessary at workspace scale.
5. Proof-of-possession — zero-trust content boundary; a cryptographic hash of the file must precede any server return of file text
   Cost for us: 🟢 trivial locally (server == client); load-bearing only in a future hosted mode.

The takeaway shape: the mechanical performance envelope (100M LOC at <200 ms with >99.9% parity) is now a commodity stack — Voyage embeddings + Zilliz/Qdrant HNSW + binary quantization + BM25 fusion + AST chunking + Merkle-diff incremental reproduces it. The non-commodity pieces (code-relationship-trained embeddings and the industrial real-time pipeline) are engineering problems, not research problems, and neither is required at on-device scale. Context Lineage is the cheap high-value win we should ship first.

Hardening & evaluation plan (next sprint)

Metric set — per-class + router + latency + receipt

Four metric families measured together. No single number tells the truth about a multi-expert router; blending them into one aggregate hides the failure modes fan-out was built to surface.

• Retrieval quality per query class — nDCG@10 (primary; balances binary and graded relevance per BEIR methodology), MRR@10, Recall@10, computed per intent class (conceptual · identifier · typed-symbol · cross-repo-impact · docs · history). Reporting all three is a hedge: SWE-bench-style retrieval evals show BM25 alone fails to retrieve any relevant file for ≈40% of instances (Jiménez et al., ICLR 2024; SWE-bench Multilingual retrieval eval, 2026), so recall matters as much as ranking. Metric implementations from ranx (Numba, bundles RRF/wsum fusion optimisation + paired-t significance) — chosen over pytrec_eval because the router needs fusion tuning in the same harness. Fixtures reuse the CoIR / BEIR triplet schema (corpus, queries, qrels with graded 0–3 relevance) so nothing has to be re-shaped (Li et al., ACL 2025).
• Router-selection accuracy — a distinct KPI from retrieval quality: precision / recall / F1 on which experts fired against a gold expert-set per query class. Prior art (RouterEval, 2025; LLMRouterBench, 2026) shows the Oracle-vs-router gap in routing is driven by model-recall failures, not ranking — the KPI has to name that failure mode. Also measured: expert- order rotation delta, because an empirical 2025 study observed a ~9.5% first-tool positional bias across LLM-based selectors; our expert list is stable, so we rotate to expose the same defect if it exists.
• Latency SLO conformance — per-expert p50/p95/p99 and end-to- end fan-out p95 against ADR-007 §9 budgets, cache-hit vs miss reported separately. Distribution — not the mean — because a slow expert blowing turn budget is exactly the failure fan-out was built to fight.
• Receipt-verified emission rate — every fired request must emit an ADR-009 signed receipt; missing receipts are a loud failure. Measured as a cross-join between the fixture run and the receipt store. Fail-closed: the KPI is 100% or the wave does not exit. This is the workspace no-hidden- behavior principle turned into a testable gate.

Fixture strategy — query classes + gold labels

Fixtures follow the CoIR / BEIR triplet shape (corpus + queries + qrels with graded relevance 0–3) so ranx computes nDCG unchanged. Each query is tagged with its intent class plus a gold expert-set annotation so router- selection accuracy is measurable in the same pass.

• Query classes (10) — the six intent classes above, plus four adversarial classes drawn from RAG-robustness literature: empty (whitespace / null / zero-token), huge (>4KB input, long-line perturbation), injection-shaped (“ignore previous instructions” and its 5-attack-category taxonomy per Ramakrishnan & Balaji, 2025 — 847 adversarial cases), and non-ASCII (Unicode identifiers, RTL text, mixed CJK). The router must classify-and-decline or fail-loud on these, never silently degrade — the BestIR robustness survey (ACM TOIS 2025) documents these as the standard adversarial classes for neural IR.
• Real-agent transcript slice — a fraction of fixtures mined from actual KYM receipts (agent-observed queries) to keep the distribution realistic. Both CoIR authors and MCP-Bench (Accenture, NeurIPS 2025 Wksp) emphasise real trajectories over synthetic-only fixtures — synthetic fixtures over-fit to the router that built them.
• Gold-label bootstrapping (three-tier, private codebase) — public benchmarks can't be reused directly, so:
  • Typed-graph weak supervision — serena / SCIP typed edges treated as high-confidence positive labels for typed-symbol and cross-repo-impact classes. Snorkel-style Language-Models-in-the-Loop pattern applied to program graphs instead of LM votes — free labels wherever the language server has type information.
  • LLM-as-labeler with audit sampling — for conceptual and docs classes where typed edges do not exist, a strong model produces graded relevance labels; a stratified 10% audit by an operator gates the label set (target Cohen's κ ≥ 0.75 against the audit slice). Budget explicitly for the biases documented in eval.qa's 2026 LLM-judge survey: position (5–15pp), verbosity (~15% score inflation), self-enhancement, and sycophancy toward assertive claims. Mitigations: use a judge model different from any candidate, run pairwise comparisons twice with positions swapped and average, and validate the judge against a human-labelled seed before trusting it at scale.
  • Human-in-the-loop for adversarial + edge cases — the small hard tail (empty / huge / injection / non-ASCII) is hand-labelled once; the volume is bounded so the operator budget is hours, not weeks (Artemova et al., COLING 2025 on hybrid labeling economics).

Harness choice — adopt vs build

Workspace principle: adopt proven libraries over building. The recommendation:

• Adopt ranx for IR metrics + fusion + significance testing; CoIR's BEIR-compatible fixture I/O schema; and mcpbench (protocol-aware JSON-RPC, per-tool p50/p95/p99, stdio + HTTP/SSE, CI regression-gate compare subcommand) for MCP-server load and soak. Generic HTTP benchmarkers (k6, vegeta, wrk) cannot correlate per-tool latency or distinguish JSON-RPC error codes from transport timeouts — mcpbench is purpose-built for the shape our router actually serves.
• Build (thin) only the router-selection accuracy scorer and the receipt-verification cross-join. Nothing on the market measures our specific gold-expert-set contract because our expert taxonomy (typed-graph tier, cross-modality corroboration) is stack-native and does not appear in any public benchmark.
• Do NOT build a from-scratch retrieval eval loop. ranx + CoIR schema + pytrec-eval-terrier already cover ≈90% of the surface. Building our own would fail the workspace “adopt proven libraries” rule and we would spend the sprint reimplementing nDCG.

Soak + concurrency plan

• Concurrency sweep — mcpbench at 1 / 10 / 50 / 100 concurrent JSON-RPC clients against every expert route. The inflection where p99 starts climbing without bound is the router's throughput ceiling — standard MCP- server methodology (Mendes multi-language MCP bench, 2026 ran 3.9M requests across four language runtimes with this shape).
• Steady-state soak — 60-minute run at 50% of the measured throughput ceiling. Watches: GC-pause spikes, index-file leaks, hint-store growth (search_router_hints, migration 0003), receipt-signing key exhaustion. Regression gate: p95 must not drift >10% and p99 not >20% versus the baseline captured on wave entry.
• Chaos slice — inject per-expert failures (kill a semantic daemon mid-flight; stall codesearch for 5s; return malformed JSON from a mocked docs expert). The failure-semantic contract in the reference design mandates a loud classified failure per expert; the soak run proves it end-to- end. This is directly analogous to the standard neural-IR adversarial- robustness playbook in BestIR, applied to the MCP transport layer.

Blocked-on-key vendor arms — now unblocked

Wave 1.2's Secrets panel (ADR-006) landed the console key wiring, unblocking the vendor arms the router benchmark can now measure:

• Zilliz claude-context (semantic slot; chunkhound expert benched pending the D4 bakeoff) — with the Zilliz key wired through the Secrets panel, the semantic-slot arm is a live participant in the fan-out, not a placeholder. First vendor arm to benchmark against the shipped BM25 + typed-graph baseline; enters the fixture run on day one.
• Voyage voyage-code-3 + rerank-2.5 — already the ChunkHound embedding default; measured as a reference point once chunkhound comes off the D4 pause and re-enters behind the same expert adapter. Reported alongside Zilliz so the semantic-slot arms compare apples-to-apples.
• Context7 / Cloudflare-docs / Svelte-docs — measured on the docs class only; scope discipline is itself a test signal. A docs expert firing on a typed-symbol class is a router-selection bug, and the receipt trail is what proves it.

Exit criteria (drives roadmap wave-1-5-search-hardening)

• Fixture set with gold labels (six intent + four adversarial classes) committed under version-controlled test-data.
• ranx-based IR-metrics harness running as a CI-adjacent nightly script; per-class nDCG@10 / MRR@10 / Recall@10 emitted into tool_benchmark_events with category="search".
• Router-selection accuracy (precision / recall / F1 on gold expert-set) reported per class; oracle gap tracked release-over-release.
• Per-class p50/p95/p99 measured against ADR-007 §9 budgets; SLO breaches are classified failures, never silent warnings.
• ADR-009 receipt-verified emission rate = 100% across the fixture set; any missed receipt fails the wave.
• Results emitted as tool_benchmark_events plus a keyed experiments row with paper_slug="moe-search-router"; read-back via GET /api/experiments?paper_slug=moe-search-router.
• ADR-009 benchmark-plan template instantiated under docs/DECISIONS/ so future MCPs inherit the same discipline.

Augment Context Engine — three-arm design (MEASURED 2026-07-31; Phase-2 CANCELLED — parity)

A sibling research pass surfaced Augment's Context Engine as the next candidate arm on the semantic slot. What follows is the arm design against fixtures v1.4.0. Cross-references: large-codebases for the engine architecture; memory-tooling-decisions (D19) for the adopt/reject decision. Not duplicated here.

Arms — three standalone runs, no fusion

Fusing a dense retriever into the RRF ensemble would discard its confidence (score = 1000 - rank) and bias against it. Each arm is scored independently on fixtures v1.4.0.

• Arm A — router baseline — the shipped fused router, unchanged. This is the 0.824 / 0.861 / nDCG@10 0.821 reference point.
• Arm B — Augment Context Engine (local) — a standalone expert over the local MCP/SDK variant, which indexes the working tree in real time and therefore gives branch parity by construction. The remote variant indexes only GitHub default branches and is unsuitable. Auth: session JSON via auggie login / AUGMENT_SESSION_AUTH, sourced from a config-declared env var name exactly like CONTEXT7_API_KEY; missing ⇒ classified expert_unconfigured. Cost ≈ $0.03–$0.06 / query ⇒ ~$1.50–$3.00 per 48-query run. Availability on our Legacy Developer Plan is undocumented and must be probed first.
• Arm C — semantic-only local control — standalone zilliz-semantic so we can separate “Augment's engine vs our dense engine” from “Augment vs our fused router”. This also fills the D4 semantic-layer bakeoff slot on the same fixtures.

Required scaffolding — concrete changes before any arm runs

• run-benchmark.mts must accept --config-file <path> and --run-name <name>, and write each arm to its own results dir, so compare.mts can diff two arms' run.trec.
• Each arm gets a dedicated config file, never merged into the shipped config.json. Priors set to 1.0 for the arm's expert across all classes, so pickExperts degenerates to “the one implemented expert always fires”.
• Cache off for benchmark runs (cache.positive_ttl_ms = 0). The baseline arm re-runs warm while a new arm is always cold, which silently flatters the incumbent.
• sidecar.json must record git_head and git_branch. Today it records config_version, timestamp, and fixture dir but not the commit, so two arms measured at different commits are silently comparable.
• Index-parity preflight — a corpus-fingerprint probe asserting the Augment index returns each of the 25 corpus docs before scoring begins, failing loud below threshold. Augment exposes no way to pin or query the indexed commit, so parity is asserted, not guaranteed.
• Adversarial fixtures stay excluded (already filtered); the docs-* class stays IR-out-of-scope, which preserves parity since neither Augment nor our stack retrieves third-party library docs.

Evidence context

Augment publishes no retrieval-only IR numbers. Their headline lift (+71% Cursor+Opus 4.5, +80% Claude Code+Opus 4.5, +30% Cursor+Composer-1) is a downstream agent-quality benchmark — 300 Elasticsearch PRs × 3 prompts = 900 attempts, scored on five subjective dimensions — vendor-run and unreplicated. There is no published retrieval figure to compare against, and this arm would be original work. See large-codebases for the engine architecture and memory-tooling-decisions (D19) for the adopt/reject decision; the details are not duplicated here.

Measured results — current state (fixtures v1.4.0)

Router-selection accuracy — F1 vs gold_experts

Honest overall figures on fixtures v1.4.0 (n=36 scored queries): micro-F1 0.824, macro-F1 0.861. Per-class F1 = 1.000 on cross-repo-impact, docs, history, and typed-symbol; identifier is 0.500 (three experts fire against a single-expert gold set) and conceptual is 0.667. Router failures: 0/36. Receipts emitted: 36/36 (receipt_emission_rate = 1.0). The selection scorer has a documented sole-expert carve-out: when the router fires exactly one expert and that expert is in gold, the query scores P=R=F1=1 regardless of gold-set size — this is why per-class 1.000 is reachable and why the overall micro-F1 is the honest headline, not the per-class high. Selection stayed at the same overall level through a wrong-index adapter regression that only IR metrics caught (basename→alias collision between two workspaces; adapter now resolves scope paths against the daemon's project registry with absolute-path filter_path narrowing) — the strongest argument yet for keeping IR benchmarks in the pre-enable gate.

Conceptual class — semantic arm promoted (R3 outcome)

Restoring the ADR-007 §4.1 priors {semantic: 1.0, codesearch: 0.8} on the graded corpus lifted Recall@10 0.617 → 0.803 (+0.186) and MRR@10 0.917 → 1.000; nDCG@10 0.635 → 0.640 sits within run-to-run noise on n=6. Two clean healthy-stack runs (results/2026-07-16T03-20-18-207Z, results/2026-07-16T03-22-33-839Z) produced identical metrics vs the codesearch-only baseline (results/2026-07-16T02-55-51-229Z). Priors PROMOTED; conceptual gold_experts extended to [codesearch, semantic]; config bumped to search-router@1.4.0.

Latency — ADR-007 §9 amendment

ADR-007 §9 carries a measured amendment (2026-07-15): the cross-repo-impact and conceptual class totals move 2500 ms → 3000 ms. Per-expert caps are unchanged. Evidence: healthy-stack fixture runs measured cross-repo-impact p95 2991 ms and conceptual max 2998 ms — over the original 2500 ms totals. Root cause is fail-closed receipt/hint round-trips against remote D1; a tracked re-measure-DOWN gate follows when console ingest throughput lands. Stated loud in the amendment, never silently relaxed. Per-class p95 targets and the wider budget table live in ADR-007 §9.

Soak · chaos · adversarial

The wave shipped an mcpbench-compatible soak driver (benchmark/soak.mts) rather than adopting upstream mcpbench directly (upstream is Go-only, lacks receipt / envelope awareness). Contract: same paper_slug="moe-search-router", same fixture triplet, JSON output mirrors mcpbench's summary shape. Concurrency ladder c ∈ {1, 10, 50, 100}: the c ≥ 50 shelf is a documented vite-dev console receipt-ingest ceiling, not a router defect; recommended production concurrency c = 10 until a compiled local console or batched receipt endpoint lands. Chaos slice: dead-port serena and broken codesearch produced classified expert_unavailable / expert_internal / timeout errors, all loud, receipts 100% — end-to-end proof of the ADR-007 §7 no-hidden-behavior contract. Adversarial: 12/12 pass (empty = classified rejection; huge / injection-shaped / non-ASCII = safe + receipted). Regression gate benchmark/compare.mts exits 1 on p95 drift > 10%, p99 drift > 20%, receipt-rate < 100%, router_errors > 0, or F1 drop > 0.05.

Retrospective — the v1.2 "semantic collapse" was a scoring artifact

An earlier fixture version (v1.2.0, 2026-07-15) appeared to show a large conceptual nDCG@10 collapse (0.656 → 0.218) when the semantic arm entered the fan-out. Investigation (v0.7 deep-dive) traced this to three stacked scoring-infrastructure issues, not a retrieval failure: (i) 443 of 646 returned items were "unmapped" by the harness's suffix-match pathToDocId() — including SOH-delimited git log -S commit-record strings that were spuriously counted as unmapped paths; (ii) the fixture corpus was 20 src/*.ts files with no ADR / prose entries, so plausibly-relevant semantic hits (ADRs, research pages, tests) were silently zero-scored; (iii) the doc-adr-007 corpus title referenced a stale filename that never suffix-matched hits from any expert, zero-wiping two conceptual queries. The v0.7 remediation plan's top three items (R1 expanded corpus + graded qrels; R2 commit-record classification; R3 re-admission re-test) ran together and produced the current v1.4.0 baseline. The v1.2 "collapse" verdict is formally superseded — it measured the fixture, not the arm.

Fixture manifest v1.4.0

Full changelog: mcp/servers/ozzydev-native-search-mcp/benchmark/fixtures/fixtures-manifest.json. Summary: (a) conceptual priors restored to ADR-007 §4.1 {semantic: 1.0, codesearch: 0.8}; (b) conceptual gold extended to [codesearch, semantic]; (c) corpus expanded 20 → 25 (ADR-007, ADR-009, this research page, three verifying tests); (d) conceptual qrels upgraded to graded 0–3 (nDCG uses grades; MRR / Recall binarize on rel_lvl ≥ 1); (e) config bumped to search-router@1.4.0; (f) git-log-s commit records classified into a dedicated commitRecords sidecar field, removing 179 spurious "unmapped" counts from the history class; (g) stale doc-adr-007 filename fixed.

Environmental flakes surfaced by the run — reported, not silenced

• codesearch MDB_READERS_FULL: after four sequential benchmark runs, the LMDB reader table on ~/.codesearch/ exhausted its 126-slot ceiling; the CLI-shellout path saturated while the upstream com.nexartis.codesearch-serve LaunchAgent kept running normally. Follow-up: route the codesearch expert through the local serve HTTP surface so reader slots are pooled (captured under §New ideas).
• serena expert_internal on find_symbol: intermittent under sustained benchmark load; direct raw-MCP calls succeed. Not correlated with any router-side change.
• Cloudflare D1 receipt-persistence flake: 1–4 failures per full 36-query run under network turbulence. Loud + classified, fail-closed working as designed; the canonical clean 36/36 run under v1.4.0 remains an outstanding cosmetic — a partial 34/36 run and two clean conceptual-slice runs jointly establish the v1.4.0 numbers.

Follow-up surfaces (from the v0.7 remediation plan)

R1–R3 are landed. R2 also fixed the git-log-s commit-record classification bug. Residual items, unchanged priority:

1. R4 · KYM-transcript conceptual queries — mine real-agent transcripts for paraphrase-heavy queries with no filename leakage; push the fixture past the synthetic-only ceiling per the CoIR / MCP-Bench recommendation.
2. R5 · Optional cross-encoder rerank behind a budget flag — Voyage rerank-2.5 already ships as the ChunkHound default; wire it as an optional post-fusion adapter (top-50 → rerank → top-k). Latency risk ~100–300 ms for API rerankers.
3. R6 · Fork @zilliz/claude-context for the ignore-file bug and native Svelte AST support. Upstream issue #222 mirrors our findIgnoreFiles() root cause; the workaround (rename .prettierignore → prettier.ignore + --ignore-path) is stable, so this is low-urgency. Operator sign-off required before any external PR (workspace §2a).
4. R7 · Per-class RRF k sweep — sweep k ∈ {20, 40, 60, 80, 100} per class against the graded fixture; Bruch et al. 2022 predicts ≤ 3% relative NDCG gain even on domain-shifted corpora, so this is a floor-check, not a lever.

Not on the list, on purpose: swapping the embedding model. No 2026-vintage code-embedding model with published benchmarks beats voyage-code-3 on code retrieval; investment there is speculative until R4–R5 have moved the measurable ceiling.

New ideas — captured for future consideration

Everything below is a documented idea, not a commitment. Each is scoped small enough to fit in a future wave.

• Proactive structural-map injection (codebase-graph pattern) — broskees/codebase-graph ships a live ~2–5K-token TOON-format structural map (modules/symbols/hierarchies/deps, tree-sitter via cased/kit, <50ms incremental) injected into the system prompt on every LLM call. Not search — the complement our stack lacks: all five of our memory systems are REACTIVE (agent must ask); a standing pre-turn map could reduce search fan-outs entirely for navigation-class questions. Candidate experiment: generate the map FROM our own indexes (no new extractor), measure search-call reduction vs the per-turn token cost on real KYM-receipted sessions. Adopt the idea, not the dependency.
• Console receipt-ingest throughput as a benchmark target — the c ≥ 50 vite-dev ceiling is real. Two candidate remediations, both benchmarkable: (a) a compiled local console (no vite-dev in the hot path), (b) a batched receipt endpoint so N receipts cost one write-lock cycle. Either way, the ceiling becomes a measurable KPI, not a workaround.
• Positional-bias rotation for expert order — the 2025 empirical study flagged a ~9.5% first-tool bias in LLM-based selectors. Our current fixtures do not rotate expert order; adding a rotation slice would surface the same defect in the router if it exists.
• Route codesearch expert through codesearch serve HTTP — pool LMDB reader slots so sustained fan-out cannot saturate the CLI-embedded reader table (MDB_READERS_FULL incident).
• MILVUS_ADDRESS capture UI in the Secrets panel — auto-resolve works when exactly one cluster exists; a first-class capture field adjacent to the Zilliz key row would tighten the loud-placeholder handoff without introducing a fallback path.

Incident lesson — “committed + validated” is not “running”

Decision history

Reverse-chron, one line per material change; ADRs are the durable source of truth.

• v0.12 (2026-07-30) — Docs expert rename context7 → external-docs is LIVE and verified in the running expert registry (search-router@1.5.0; expert_registry returns codesearch, git-log-s, serena, semantic, external-docs) — the vendor-neutral-seam principle applied to the router's own type system. Recorded the two-session fail-closed incident that produced the durable “committed + validated is not running” rule: config carried external-docs while dist/ still shipped the zod enum expecting context7, so every spawn exited 1 with experts.external-docs: Invalid enum value; misdiagnosed twice as intermittent MCP-attachment. Rule: verify find src -name '*.ts' -newer dist/cli.js = ∅ for every served MCP (the older process-vs-dist check only catches stale PROCESS, not a fresh process running a stale BUILD), and hand-spawn to read stderr because a fail-closed MCP is indistinguishable from an unattached one at the harness's toolset check. Expert table + provenance-tier “docs” meaning updated to name external-docs as the generic expert with context7 configured as an instance behind it.
• v0.14 (2026-07-31) — Retrieval-bakeoff parity verdict recorded. Calibration batch calibration-20x3 (20 hard-stratum tasks × 3 arms × 1 seed, sonnet4.5 pinned, prereg lock 4f3cce5) completed: 60 runs, 13,675 cr, zero crashes. Phase-2 de-saturation gate (HYPOTHESES §5.3 v3.2, four pre-stated criteria) FAILED — criteria 1–3 fail (all arms 100% resolve; per-task composites byte-identical across arms, 0.0 pp arm spread, sd 0.1646 identical), only criterion 4 (8/20 divergent-output tasks) at threshold. Operator ruling: accept parity as headline; Phase-2 full-n (~302–430 kcr) CANCELLED. Retrieval stack does not change resolve/composite under sonnet4.5 on this deck at pilot+calibration scale (~56 kcr total). Exploratory cost signal: retrieval arms ~27% cheaper (a=277 / b=202 / c=205 cr/run mean). §Augment Context Engine three-arm design marked MEASURED with verdict callout + evidence pointer to nexartis-auggie-harness/probes/bakeoff/analysis/calibration-20x3/ @ bb8b333 / ruling 8cc56b7.
• v0.11 (2026-07-28) — Added §Reclaim targets: the discontinued Augment VS Code extension is named as the platform's founding inspiration, and its performance envelope is written up as falsifiable reclaim targets (search < 200 ms @ 100M LOC after quantized ANN, 8× RAM reduction 2 GB → 250 MB, > 99.9% ANN parity with full-embedding fallback on the 0.1% edge, seconds-latency incremental at thousands of files/sec, near-instant branch-switch via per-developer indices, $0.03–$0.06 per query) — each marked vendor-published with source and date. Strategic-finding callout: no retrieval-only IR metric (nDCG / MRR / Recall / precision) has ever been published by Augment or by any third party, so our ADR-007 numbers (micro-F1 0.824 / nDCG@10 0.821 on fixtures v1.4.0, n=36) are first-of-kind in the comparison, with the honest caveat that n=36 over a 25-doc corpus is bench-scale and a public claim needs a larger corpus. Latency + freshness listed as a separate axis where we do not currently measure most items (index build curve, incremental p50/p95, branch-switch wall clock, cold/warm split, local RAM footprint) — flagged as work, not parity. Techniques ranked by reproduction difficulty; Context Lineage (commit-diff summaries embedded alongside chunks) called out as the cheap high-value win to ship first. Sources block extended with Augment repo-scale-100M, real-time-index, Context Lineage, Context Engine MCP overview, marketplace sunset notice, and the in-workspace research packet.
• v0.10 (2026-07-27) — Claim-hygiene pass: status line and §Router-selection accuracy corrected to the honest overall figures (router-selection micro-F1 0.824 / macro-F1 0.861; IR nDCG@10 0.821 / MRR@10 0.857 / Recall@10 0.830); the sole-expert carve-out that makes per-class 1.000 reachable is called out in the honest-baseline callout and again in-section. Added §Augment Context Engine three-arm design (Arms A/B/C): headline finding that Augment's supported API returns a formatted string with no per-item scores, but rank-only scoring is valid on this harness because our runfile builder already sets score = 1000 - rank; named the “does the formatted string encode retrieval rank” open question with the raw-endpoint-as-oracle fallback; listed the scaffolding (per-arm config, --config-file/--run-name, cache off, sidecar.json git_head/git_branch, index-parity preflight) and hazards (priors-before-scoring, sole-expert incumbency, cache warmth, fusion-discards-confidence, dormant selection-F1 gate on wave-1-5-entry.json, resolveScope single-alias limit). Pending operator sign-off before implementation.
• v0.9 (2026-07-16) — Path-keyed-collection incident found + repaired: Zilliz collections hash md5(codebasePath)[0:8], so the workspace move orphaned hybrid_code_chunks_b5fa75c8 (the v0.6 collection — now HISTORICAL/dropped) and the semantic expert failed classified-but-invisible on every call since. Corpus re-keyed + rolled out multi-repo (4/47: ozzydev, knowyourmodel-ai, knowyourmodel-sdk, waves-harness) with supportedExtensions ['.svelte', '.sql']; coverage surface = console /memory Repos tab + GET /api/doctor/repos (orphan detection, ignore-slurp hazards per upstream #222). Same day: embedded core PATCHED with the upstream PR #372 allowlist (.gitignore + .contextignore only — patches/@zilliz__claude-context-core@0.1.15.patch), retiring the slurp class; corpus then rolled out fleet-wide to 45/45 workspace repos incl. the previously-blocked .dockerignore repo. ADR-007 amendment of record.
• v0.8 (2026-07-16) — R1–R3 landed; fixtures v1.4.0, config search-router@1.4.0; conceptual semantic arm promoted (Recall@10 0.617 → 0.803, MRR@10 1.000); ADR-007 §9 amended (cross-repo-impact + conceptual class totals 2500 → 3000 ms); v1.2 "semantic collapse" formally superseded as a scoring artifact.
• v0.7 (2026-07-15) — Deep-dive traced the v1.2 conceptual collapse to corpus-coverage + suffix-match + stale-title bugs; published the R1–R7 remediation plan.
• v0.6 (2026-07-15) — Zilliz semantic arm went live (aws-us-west-2, collection hybrid_code_chunks_b5fa75c8, voyage-code-3, hybrid dense+sparse); fixtures v1.2.0 admitted semantic to cross-repo-impact gold on measured evidence; conceptual re-admission deferred pending fixture expansion.
• v0.5 (2026-07-13) — First measured baseline: selection macro-F1 0.861, receipts 36/36, W3 latency optimization (single-query p50 4.4 s → ~2.5 s), W5 soak / chaos / adversarial harness shipped, wave-entry soak baseline archived.
• v0.1 (2026-07-12) — Architecture published; ADR-007 locked (expert pool, classifier, RRF k=60 + per-class priors, hint store, cache, bandit reward, failure semantics).

How this page stays current

Living architecture paper. The version at the top bumps on every material change; the decision history above is the changelog. Numbers on this page must trace to a persisted artifact (ADR or benchmark result under mcp/servers/ozzydev-native-search-mcp/benchmark/) or they get softened to qualitative.