Deliberative Search — an Open-Question Synthesis Expert for the MoE Router

Extend ADR-007 with a new intent class for open DESIGN questions. When an agent hits an unresolved design decision, the router mines internal conventions, gathers external best practices, synthesises structured options with a single recommendation, and hands the harness a signed option-picker payload. Deliberation as a first-class search capability — not a client-side prompt pattern.

Problem

Agents in this workspace routinely hit open DESIGN questions: "should we adopt library X", "which of three patterns does our house style already favour", "is there a 2026 best practice for Y that we are behind on". The current AGENTS.md convention says: after the initial audit, address open questions FIRST, using an explicit rubric (mine internal patterns · gather external best practices · synthesise · pick). Today that rubric lives in prompt-space — every agent re-derives it, every session pays the tax, and the synthesis quality varies with the agent's diligence.

The pathology is the same one MoE for Search named for routing: a good discipline that lives in agent instructions is a discipline the agent might follow. A good discipline that lives in a router with a signed receipt is a discipline that is either followed or classified-as-not-followed. Deliberative Search moves the open-question rubric into router-space.

The four-stage rubric

This is the AGENTS.md open-question rubric, unchanged in shape, moved from prompt-space to router-space. Each stage has an explicit failure semantic per the workspace no-hidden-behaviour principle; "empty" is a valid, reported outcome, never a silent degradation.

1. 1 · Mine internal conventions — Fan out to the internal experts (codesearch + serena + git-log-S + an ADR/architecture-doc expert) to extract the consistently-followed design patterns and load-bearing architecture claims that already govern this codebase. Every claim carries a path + line-range citation from a persisted artifact (an ADR, an architecture doc, a canonical file). No unsupported "we usually do X" bullets.
   Failure semantic: A convention with no artifact citation is dropped, not softened. If NO internal convention is found, the stage returns an empty set with a receipt field saying so — silence is not consensus.
2. 2 · Gather external best practices — Fan out to the external-docs experts (context7 + cloudflare-docs + svelte + a new web/best-practices adapter) for the current 2026-vintage answer to the same question. Prefer primary sources (specs, upstream code, benchmarks) over READMEs, per the AGENTS.md source-scouting directive. Each external claim carries its URL and a fetched-at timestamp.
   Failure semantic: An external claim without a URL is dropped. If no external best practice exists (novel problem), the stage returns empty and the synthesiser is told so — a green field is a valid finding, not a gap to paper over.
3. 3 · Synthesise into an option set — Deterministic synthesis stage: cluster the internal + external evidence into 2–5 distinct options, produce a brief per-option analysis (tradeoffs, risk, cost, fit-to-house-style), and emit a single recommendation with its rationale. Every option carries the citation set that supports it; every option carries an explicit dissent set (evidence that argues against it). The recommendation is a POSITION, not a hedge.
   Failure semantic: If the evidence cannot support a single recommendation, the stage says so and returns options with equal-weight tradeoffs. "Undecidable on current evidence" is a valid, reported outcome; it is not silently collapsed to "pick the first one".
4. 4 · Hand the harness a structured payload — The router returns a typed OptionPicker payload — { question, options[], recommendation, dissent, citations[], receipt_ref } — that the harness renders as an option-picker UI. The agent (or the operator) picks; the choice is written back into the receipt so the tuner can learn which recommendations survived contact with the operator.
   Failure semantic: A payload without a receipt_ref is refused; a payload whose citations do not resolve is refused. There is no "free-text answer" fallback shape.

Router integration — where the extension hooks in

ADR-007's pipeline stays intact. Deliberative Search is additive: one new class in the classifier, one new gate row, two new expert adapters, one new post-fusion stage, one new return shape, six new receipt fields. Nothing is removed; no existing class changes behaviour.

• Classifier — new class `open-question` (design-decision) — Local rules trigger on shape: interrogative + design-lexicon terms ("should we", "which pattern", "is it better to", "how do others", "tradeoff between") + no exact identifier or file path. On rules ambiguity, LLM escalation (existing opt-in path) is allowed. Confidence < 0.6 abstains to the documented default set for `open-question` (see §Failure semantics) rather than silently falling through to `conceptual`.
• Gate — new expert-set prior for the class — For `open-question`, the gate fires TWO expert bundles in parallel: (a) internal-convention miner (codesearch + serena + git-log-S + ADR/architecture-doc expert) and (b) external-best-practices bundle (context7 + cloudflare-docs + svelte + web/best-practices). The one-expert-suffices branch (§3.2) is explicitly DISABLED for this class — deliberation requires cross-source corroboration.
• Fan-out — two new expert adapters — Adds two adapters to the pool without changing existing ones: (i) `internal-convention-miner` — a thin composite that runs the internal fan-out with an ADR-aware ranker that up-weights docs/DECISIONS/**, docs/*ARCHITECTURE*.md, AGENTS.md, and canonical file paths named in those docs; (ii) `external-best-practices` — a thin composite that runs the docs bundle plus a web-search adapter guarded to primary-source domains (arxiv, official vendor docs, upstream repos).
• Fuse — new synthesis stage after RRF — RRF still runs first over the raw evidence set (the ADR-007 §5 fusion contract is preserved). A new deterministic synthesis stage then reads the fused evidence and produces the OptionPicker payload. The synthesis stage is a pure function of (evidence set, config version, class); the same inputs always produce the same output. If synthesis is model-mediated, the model call is receipts-visible and pinned by version.
• Return — new `expected_shape=option-picker` — Extends the existing `expected_shape` enum with `option-picker`. Callers who explicitly ask for `ranked-evidence` on an open-question still get evidence; callers who ask for `option-picker` (or leave it unset on `class=open-question`) get the synthesised payload plus the underlying evidence appended. No caller silently gets a different shape than it asked for.
• Receipt — new synthesis fields — ADR-009 receipt gains: `synthesis.options_count`, `synthesis.recommendation_id`, `synthesis.dissent_count`, `synthesis.citation_set_hash`, `synthesis.operator_pick` (written back post-decision). The tuner reads `operator_pick` as a first-class reward signal — the router learns which recommendations survived the operator, not which recommendations the LLM approved of.

OptionPicker payload shape

Typed, contract-first. The harness rendering is per-harness; the router's guarantee is the shape.

• question — Verbatim caller question, normalised whitespace.
• class — Always `open-question` for this payload shape; other classes never emit an OptionPicker.
• options[] — 2–5 options. Each: `{ id, title, summary, supports: Citation[], dissent: Citation[], tradeoffs: string[], cost: qualitative, risk: qualitative }`. `title` is a short label; `summary` is 2–4 sentences.
• recommendation — `{ option_id, rationale, confidence: low|med|high }`. A single option id or the string `undecidable-on-current-evidence` — never null, never empty.
• citations[] — Deduplicated citation set across all options. Each: `{ id, kind: internal|external, path_or_url, line_range?, fetched_at?, precision_class }`. `id` is stable across a config version so the payload is diffable turn-to-turn.
• evidence_ref — Opaque handle to the raw fused evidence set the synthesis consumed. Callers who want to inspect the pre-synthesis fan-out follow this ref; the router does not inline it into the option-picker payload.
• receipt_ref — ADR-009 signed receipt id. Every OptionPicker payload has one; a payload without one is a bug, not a soft failure.

Failure semantics — deliberation-specific

Inherits every ADR-007 §7 failure clause. On top of those, the synthesis stage has its own classified failure modes:

• Internal miner returns empty (no ADR/convention artifact matches) → stage 1 reports empty with a classified reason; synthesis proceeds on external evidence only and the recommendation confidence caps at `low`. Silence is not consensus.
• External bundle returns empty (novel or private problem) → stage 2 reports empty; synthesis proceeds on internal evidence only and the recommendation notes "no external best-practice found — this is stack-native reasoning".
• Synthesis cannot separate options (evidence is symmetrical) → recommendation = `undecidable-on-current-evidence`, options returned with equal weights, the receipt records the symmetry so the tuner can request more evidence next time.
• Citation fails to resolve at return-time (a URL 404s, a path was deleted mid-flight) → the option is downgraded to `low` confidence and the failed citation is retained with a `resolution_failed` flag. Never silently dropped, never silently replaced.
• Config drift between fan-out and synthesis (rare: bump mid-request) → request is failed classified. No implicit re-run.
• Operator overrides the recommendation → the override is written back into the receipt (`synthesis.operator_pick != synthesis.recommendation_id`); the tuner treats overrides as negative reward on the recommendation and positive reward on the picked option. Overrides are how the router learns.
• Escaping the class — an `open-question` request that resolves to an identifier lookup mid-fan-out (the classifier was wrong) still emits the OptionPicker shape but marks it `class_mismatch=true` in the receipt. The caller gets a payload, the tuner gets a labelled error.

Prior art — honest comparison

The novel bits are (a) mining INTERNAL artifact conventions as an equal-weight source alongside external best practices, (b) a deterministic post-fusion synthesis stage that emits a typed payload rather than free-text, (c) operator-pick as the ground-truth reward for the tuner, (d) a citation-set-hash that makes deliberations diffable turn-to-turn. Everything else is composition of well-known patterns.

• OpenAI Deep Research (2025) · Google Gemini Deep Research (2024) · Perplexity Deep Research (2025) — Multi-step web research agents that produce a synthesised report with citations. Closest external analogue for stage 2 + 3. Deltas: (a) they are single-agent report generators, we are a router extension that hands the harness a STRUCTURED option-picker, not a report; (b) they synthesise from web only, we co-mine internal artifact evidence as an equal-weight source; (c) our synthesis is a pure function pinned by config version so results are diffable, theirs is a free-form generation.
• ReAct (Yao et al., ICLR 2023) — reason + act interleaving — The reason-act loop is what an agent runs AROUND our router. Deliberative Search collapses many ReAct iterations for a single open question into one structured call: the agent reasons ABOUT which option to pick, the router does the deliberation ONCE with a receipt. arXiv:2210.03629.
• Tree of Thoughts (Yao et al., NeurIPS 2023) — branching deliberation — ToT branches candidate solutions and searches over them. Our synthesis stage is a bounded, deterministic version of that idea — options are the branches, dissent + supports are the search state, the recommendation is the picked branch. We drop the tree search (bounded to 2–5 options) and add receipts. arXiv:2305.10601.
• Self-Consistency (Wang et al., ICLR 2023) — sample & vote — Self-consistency samples many chains and votes. Analogous to our cross-expert corroboration in stage 1+2: an option supported by BOTH internal artifact evidence AND external best-practice sources gets a higher recommendation confidence than one backed by either alone. arXiv:2203.11171.
• Self-Refine (Madaan et al., NeurIPS 2023) · Reflexion (Shinn et al., NeurIPS 2023) — Iterative self-critique loops. Our synthesis stage explicitly does NOT self-refine — it emits an option set and a recommendation ONCE per receipt. Refinement is the operator picking (or overriding) and the tuner learning from that pick. Reflection stays on the caller side, not the router side. arXiv:2303.17651, arXiv:2303.11366.
• LLM-as-a-Judge / MT-Bench (Zheng et al., NeurIPS 2023 D&B) — Pairwise / graded scoring by an LLM judge. The synthesis stage uses judge-style pairwise reasoning INTERNALLY to rank options, but the KNOWN biases (position, verbosity, self-enhancement, sycophancy — eval.qa 2026 survey) are mitigated the same way the MoE router already mitigates them: position rotation across option ids, judge model distinct from any candidate that produced options, and an operator-pick reward signal that overrides judge preferences. arXiv:2306.05685.
• Debate (Irving et al., 2018) · Constitutional AI (Bai et al., 2022) — Two-agent debate + rule-based critique. Deliberative Search inherits the "pair every claim with a dissent" discipline from debate but does not spin up debating agents — the dissent set is mined from the same evidence pool as the supports, not generated. Constitutional AI informs the house-style-fit criterion (a candidate option that violates AGENTS.md conventions gets flagged as a dissent against itself). arXiv:1805.00899, arXiv:2212.08073.
• RAG for decision synthesis (2024–2026 practice) — Retrieval-augmented generation for structured decisions. Prior art for stage 3. Delta: we do not generate a free-text answer; we produce a typed payload whose shape is a contract, and we require deduplicated citation sets that resolve to persisted artifacts. Ungrounded prose is not a valid synthesis output.
• ADR auto-drafting (adr-tools, log4brains, ADR Manager, GitHub-Copilot ADR templates) — Tools that scaffold ADR markdown files. Complementary, not competing: Deliberative Search produces the DECISION content (options + recommendation + citations) that these tools then serialise into an ADR file. We do not write ADR files — we produce the payload an ADR-drafting tool consumes. See docs/DECISIONS/ADR-007 for our house ADR template.
• ArchiGPT / ArchDoc-Auto (2024–2025 architecture-doc auto-generation) — End-to-end architecture-doc generators. They aim at the final document; we aim at the DECISION MOMENT that precedes it. Our synthesis is a step earlier in the pipeline: pick the option, THEN let a doc generator write it up.

Naming — recommendation + alternatives

The operator asked for a name. Candidates weighed against the house style (descriptive titles, occasional lens name like "MoE for Search"; slugs stay hyphenated-lowercase and readable on the /research index):

• Deliberative Search (slug: deliberative-search) — Parallels "MoE for Search" cadence. "Deliberation" is the widely-known term for Tree-of-Thoughts / debate-style multi-branch reasoning, so the name teaches its own shape. The whole thing IS a deliberation (mine → gather → weigh → recommend), not just a synthesis step tacked on. RECOMMENDED.
• Open-Question Synthesis (slug: open-question-synthesis) — Task-descriptive; matches the AGENTS.md convention verbatim ("address open questions first"). Reads clearly on the /research index. Weakness: emphasises the OUTPUT (synthesis) rather than the METHOD (deliberation across sources); undersells stages 1–2.
• Decision-Support eXpert (DSX) (slug: decision-support-expert) — Acronym is memorable but the house style avoids acronyms ("MoE" is the exception because it names a real ML pattern). "Decision support" is enterprise-BI-flavoured and undersells the internal-convention mining. Not recommended.
• the Counsel expert (slug: counsel-expert) — Evocative single word — the router gives you counsel, not an answer. Fits the operator-in-the-loop framing. Weakness: as a research-page slug it is too thin and non-searchable; works better as an INTERNAL adapter id inside Deliberative Search.
• Adjudicator (slug: adjudicator) — Wrong metaphor: adjudication picks a winner from adversaries; the router synthesises complements. Reserve if we ever build an LLM-as-judge pairwise ranker as a distinct expert.
• Synthesis expert (slug: synthesis-expert) — Accurate and too generic. "Synthesis" is one of the four stages, not the whole thing.

Recommendation: Deliberative Search with slug deliberative-search. It parallels the parent paper's cadence ("MoE for Search" → "Deliberative Search"), the name itself teaches the shape (four-stage deliberation, not just a synthesis step), and it leaves the operator-facing string "open-question" as the CLASS name inside the router while the CAPABILITY carries a brandable identity. Runner-up: Open-Question Synthesis if we prefer maximal task-descriptiveness over cadence with the parent paper.

Open questions

• Is the synthesis stage model-mediated or purely rule-based? — A rule-based synthesiser (cluster by keyword + score by citation count) is deterministic and cheap but brittle on unfamiliar questions. A model-mediated synthesiser (a small pinned LLM call) generalises but adds latency + a receipt-visible model version. v0.1 recommends model-mediated with a pinned model and a rules-based fallback documented in the receipt; v0.2 measures both against a fixture of real open questions harvested from AGENTS.md audit sessions.
• How is the internal-convention miner ranker built? — The internal miner up-weights docs/DECISIONS/**, canonical architecture docs, and AGENTS.md. But architecture docs drift; ADRs supersede each other; AGENTS.md changes. Freshness signals from ADR-007 §5 apply, but we also need an ADR-supersedence-aware ranker so a superseded ADR is down-weighted automatically. Open: is that ranker part of the miner adapter or part of the fusion layer?
• External-best-practices adapter — which primary sources? — Guarded domain allow-list vs. open web. v0.1 recommends allow-list (arxiv.org, official vendor docs, upstream project READMEs pinned by commit, plus the existing context7/cloudflare-docs/svelte MCPs). Open web is deferred until we have a receipt-visible provenance model for arbitrary URLs. The AGENTS.md source-scouting directive still applies: the adapter must record NEW sources it discovered so the allow-list grows.
• Latency budget — deliberation is slower than search — Stage 1+2 are parallel fan-outs like a normal search. Stage 3 (synthesis) adds a serialised step. ADR-007 §9 latency budgets are per-class; `open-question` needs its own budget row. Draft target: p95 ≤ 6000 ms end-to-end (2× the conceptual-class budget after the 2026-07-15 amendment), justified by the deliberation cost. Measured against a fixture in the R-wave benchmark plan.
• How does the tuner reward synthesis quality? — ADR-007 §8 uses downstream turn success from KYM receipts. For an open-question, downstream success is fuzzier: the operator PICKED an option and the resulting work later succeeded. Two candidate reward signals: (a) `operator_pick == recommendation` (agreement reward), (b) later turn success given the picked option (outcome reward). Recommendation: track BOTH; the outcome reward is the ground truth, the agreement reward is a cheap proxy that becomes measurable immediately.
• Cache key for open-question turn cache — ADR-007 §7 keys the turn cache by (question-hash, scope-hash, config-version). For open-question, we need a THIRD dimension: `artifact-set-hash` — the hash of the current ADR/architecture-doc set + AGENTS.md. If AGENTS.md changed, prior deliberations are stale even if the question is identical. Draft: extend the cache key; invalidation is atomic on any tracked-artifact change.
• How does the option-picker payload render in the harness? — The harness renders it, not the router. For Kilo / Claude Code / Cursor, the OptionPicker must degrade gracefully to markdown when structured rendering is unavailable — a table with option ids + recommendation callout + citations list. The router's contract is the payload; the rendering is per-harness. Coordinate with ozzydev-console for the reference renderer.
• When SHOULD an agent NOT call Deliberative Search? — The class is `open-question` — design decisions. A well-scoped identifier lookup, a typed-symbol reference, a docs lookup: none of these should route to Deliberative Search even if the agent phrases them interrogatively. The classifier's abstain-to-default path for `open-question` explicitly routes to the standard `unknown` default set, not to synthesis. Deliberation is expensive; misfiring on trivial questions is the failure mode to prevent.

Positioning

1. Extension of ADR-007, not a replacement. The MoE router remains the primary interface; Deliberative Search is a new intent class + synthesis stage inside it. All existing classes and behaviours are unchanged.
2. Operationalises the AGENTS.md convention ("address open questions first, using this rubric") as a first-class capability. The rubric moves from prompt-space (every agent re-derives it) to router-space (one call, one signed payload).
3. Sequencing: v0.1 architecture (this page). v0.2 reference implementation behind a feature flag on the existing MCP; fixture = 20 open questions harvested from recent AGENTS.md audits with gold recommendations picked by the operator. v0.3 promotion decision based on the fixture + operator-pick reward signal; new ADR (candidate ADR-010 or an amendment to ADR-007) captures the promoted contract.
4. Publication path: same as MoE for Search — operator sign-off before any external publication. The novel bits worth publishing are (a) the four-stage rubric as a signed router capability, (b) the operator-pick reward as the ground-truth signal for LLM deliberation quality, (c) the citation-set-hash discipline that makes deliberations diffable turn-to-turn.

Next steps

1. Operator sign-off on the name (Deliberative Search vs Open-Question Synthesis) and on the extension shape (new class + adapters + synthesis stage).
2. Draft the ADR — either an amendment to ADR-007 (extension, no supersedence) or a companion ADR-010 (Deliberative Search normative). Recommendation: amend ADR-007, mint ADR-010 only if the synthesis stage grows a schema large enough to warrant its own decision record.
3. Harvest a fixture of 20 real open questions from recent AGENTS.md audit sessions; hand-label a gold recommendation per question (operator, one afternoon). This is the v0.2 measurement substrate.
4. Implement behind a feature flag in mcp/servers/ozzydev-native-search-mcp; extend the ADR-009 receipt schema with the synthesis fields; wire the OptionPicker renderer in ozzydev-console.
5. Run the fixture; report agreement-with-gold + operator-pick reward; promote or iterate. Same discipline as the MoE-for-Search fixture-driven promotion path.