Telemetry & Observability — the Measurement Substrate

The recursive self-improvement loop needs a place to record what happened. Migration 0002 landed the six-table core (experiments · variants · runs · turns · tool_benchmark_events · session_reports); 0003 added the ADR-009 receipts pair, rubrics/evaluations, and event_rollups_daily; 0004 closed the receipt_id reconciliation slice; the 2026-07-30 workspaceId attribution wave closed a misattribution vector that had been silently binding rows to the wrong tenant. This paper narrates the substrate, who reads back what through paper_slug, the general principle the attribution wave codified (emit identity you know; never let a downstream guess), and the shared receipt contract that binds every terminal MCP path.

Purpose

• Every other research page proposes an experiment. Without a shared recording surface each one would invent its own log format and none would compose. The substrate fixes that: one table shape for the loop (experiment → variants → runs → turns → events), one companion for the per-session wrap-up, one signed-receipt table anchoring every terminal MCP path.
• Agents emit tool events + session reports → D1 → research papers read them back via paper_slug. The write path is fire-and-forget POSTs; the read path is scoped GETs keyed to the paper that owns the experiment.
• Loud, classified failure at ingest (§2 of AGENTS.md): batch ingest validates every row before touching the DB and throws telemetry_batch_invalid with row indexes on any bad row. No silent drops, no partial writes, no defense-in-depth fallbacks.
• Non-goals of the substrate itself: it does not evaluate, it does not rank, it does not decide. Those live in the experiment consumers (harness-bandit, moe-search-router, write-strategies, …). The substrate only makes their answers durable and comparable.
• Per-MCP telemetry pattern (operator decision 2026-07-17, NORMATIVE for every future native MCP): each MCP under development self-emits tool-specific telemetry — (a) signed receipts on every terminal path, (b) tool-taxonomy events (category · tool · operation) to the shared D1, (c) classified loud failures, (d) config_version discipline so behavior changes are receipts-visible. The local D1 substrate is the deliberate early PROTOTYPE of the future agentic telemetry: when a tool gains an LLM helper it becomes agentic, registers in KYM, and only then does receipt forwarding + run-registry reputation optimization begin. Until that gate, all optimization is MCP/tool-level inside Ozzy Dev, tuned from this local data — the first tool set (ozzydev-search, ozzydev-write, telemetry_emit) establishes the pattern the rest inherit.

The landed schema

• experiments — The experiment definition. Long-lived; a research paper owns one or more of these.
  columns (excerpt): ulid, name, category (harness-bandit | tool-benchmark | write-strategy | search-router | embedding-bench | context | skills | orchestration | voice | other), status (draft | running | paused | completed | archived), paper_slug, decision_slug, created_at, …
• experiment_variants — The arms of a bandit / cells of a matrix. Immutable once a run references them.
  columns (excerpt): ulid, experiment_ulid → experiments, name, config_json, created_at
• experiment_runs — One end-to-end pass of an experiment with a fixed variant. Aggregates its turns.
  columns (excerpt): ulid, experiment_ulid, variant_ulid, session_id, machine_id, workspace_id, started_at, ended_at, outcome (success | fail | error | skipped | running)
• experiment_turns — One agent turn inside a run — the context self-monitoring anchor.
  columns (excerpt): ulid, run_ulid, turn_index, agent_role (orchestrator | subagent | benchmark | evaluator | router), context_utilization_pct, quality_note (crisp | terse | degraded | empty), compaction_events, truncation_events, tokens_in, tokens_out, latency_ms
• tool_benchmark_events — Central capture row for every tool invocation. The KPI-triad table the experimental-framework paper specified.
  columns (excerpt): ulid, session_id, category (search | write | scripts | verify | memory | orchestration | …), tool, operation, latency_ms, ok, error_class, tokens_in, tokens_out, dollars_micro, accuracy, verify (json), receipt_id (rc-prefixed; app-layer contract, no hard FK — §3.4.5), machine_id, occurred_at
• session_reports — Absorbs the manual TOOL-USAGE-TRACKING.md doc: one row per session with the wrap-up JSON blobs.
  columns (excerpt): ulid, session_id, machine_id, workspace_id, tools_summary_json, incidents_json, wrong_tool_picks_json, lessons_json, context_utilization_pct, quality_note, created_at
• run_receipts + run_receipt_payloads — ADR-009 Ed25519-signed provenance. One row pair per terminal MCP call (search / write), including cache hits — a call that cannot emit a receipt is a classified receipt_* error, never an unlogged path.
  columns (excerpt): run_receipts(id, category, paper_slug, config_version, schema_version, run_id, turn_id, workspace_id, machine_id, signer_pubkey (32B), signature (64B), payload_hash, issued_at_ms, kym_status, …); run_receipt_payloads(receipt_id → run_receipts, payload_json)
• event_rollups_daily — Pre-aggregated (day_utc × category × tool × agent_role × workspace) rows so dashboards never scan raw tool_benchmark_events. Idempotent hourly recompute inside one D1 batch; rollup_* classified errors on failure.
  columns (excerpt): day_utc, category, tool, agent_role, workspace_id, event_count, ok_count, error_count, latency_p50_ms, latency_p95_ms, tokens_in_sum, tokens_out_sum, dollars_micro_sum, computed_at

Write paths

• POST /api/receipts/with-event — Single-transaction endpoint: verifies the signed receipt (hash + signature + envelope↔common cross-check) and commits run_receipts + run_receipt_payloads + the paired tool_benchmark_events row in ONE D1 batch. Both native MCPs (emitSearchReceipt / emitWriteReceipt) go through the shared @nexartis/ozzydev-receipts helper — the orphan window is closed by construction.
  D1 batch() is atomic per the Cloudflare contract; partial success is impossible. event.receiptId must equal receipt.receiptId — mismatch is invalid_request (400).
• POST /api/telemetry/events — Legacy batch ingest for tool_benchmark_events (telemetry-emit CLI, soak driver, bakeoff harness), ≤500 rows per call. Validates every row up front; either all land or none do (telemetry_batch_invalid with row indexes).
  Ingest-time contract: receiptId must match the rc- prefix or the row is rejected. Existence is not checked here — the reconcile surface below handles the race for legacy callers.
• POST /api/telemetry/reconcile — Audit path for legacy dual-POST flows. Counts (receiptIdRows, orphanCount, deletableCount, inFlightCount) always populated; optionally deletes orphans older than a safety window (default 60s so a legitimate in-flight pair is never swept). Emits receipt_orphan (400) on deletion.
  Belt-and-suspenders for callers that still POST /api/receipts and /api/telemetry/events independently. MCP-driven emission does not need it — with-event closes the window.
• POST /api/telemetry/session-report — One row per session; the tools_summary/incidents/wrong_tool_picks/lessons blobs replace the hand-maintained TOOL-USAGE-TRACKING.md workflow.
  Written at end-of-session by the orchestrator (or the subagent, when scoped). Idempotent on (session_id, agent_role, wave_slug).
• POST /api/experiments (+ /variants, /runs, /turns) — CRUD for the experiment lifecycle. Mutations go through auditInsert (subject_kinds registered in workspaces.ts AUDIT_ALLOWLIST); telemetry events do NOT audit-log (they ARE the telemetry).
  Lifecycle mutations are audited; per-event ingest is not. The distinction is deliberate — auditing every event would blow up audit_log.
• GET /api/experiments?paper_slug=… — The read-back key. Every research page whose experiments are recorded here can list its own runs, variants, and aggregate outcomes without knowing DB internals.
  paper_slug is indexed (idx_experiments_paper_slug WHERE paper_slug IS NOT NULL); decision_slug is the sibling for ADR-anchored experiments.
• telemetry_emit MCP tool (ozzydev-search server) — SHIPPED 2026-07-17 — the planned per-turn MCP entry point. Kinds: events (≤500 rows, client-side cap), session-report, experiment. Posts straight to the console D1 with a 30 s deadline (generous-baselines directive); classified failures telemetry_console_unreachable / telemetry_invalid / telemetry_internal. KYM sync deliberately deferred until after dogfood-release hardening — data accrues locally for the self-improvement loop.
  Loud-failure contract: no retries, no buffering, no silent skip. Orchestrators emit; subagents report to the orchestrator.
• bin/telemetry-emit.mjs + bin/ozzydev-checkpoint.mjs — Zero-dependency CLIs (pnpm run telemetry:emit <events|session-report|experiment> <file|->; ozzydev-checkpoint for per-turn context self-monitoring) — the fallback for non-MCP contexts now that telemetry_emit is the primary path (workspace AGENTS.md §5). Both resolve workspace_id via --workspace-id if given, otherwise walk up from CWD to kilo.json and POST {path} to /api/workspaces (register-or-reanimate, idempotent by path) — the same resolution the native MCPs perform through @nexartis/ozzydev-receipts.
  Loud-failure contract: no retries, no buffering, no silent skip — if the console is down or workspace_id cannot be resolved the wave deliverable carries the failure (receipt_workspace_unresolved). The CLIs deliberately DUPLICATE the resolveWorkspaceId shape inline to preserve zero-dependency portability; any contract change to the resolver MUST be mirrored into both scripts.

What each paper's experiments consume

• /research/experimental-framework — Recording schema = tool_benchmark_events rows. The KPI triad (latency / accuracy / cost) is literally latency_ms / accuracy / dollars_micro + tokens.
• /research/harness-bandit-experiments — experiment_runs + experiment_turns + experiment_variants for the MAB arms; rubrics + evaluations for the ranking; run_receipts for signed provenance on every arm pull.
• /research/moe-search-router — tool_benchmark_events with category="search" for expert-level accuracy@k and latency; search_router_hints for the per-question-class expert selection cache (0003).
• /research/write-strategies — tool_benchmark_events with category="write" and the verify column populated (typecheck / lint / test result JSON). Powers the a–h rubric measurement.
• /research/context-engineering — session_reports + experiment_turns.context_utilization_pct / quality_note / compaction_events / truncation_events — the self-monitoring loop lives here.
• /research/scripts-as-tools — tool_benchmark_events where tool = "skill:<slug>@<version>". Skill success rates and turns-saved ratios roll up from that pattern.

Landed in migrations 0003 & 0004

• run_receipts + run_receipt_payloads (§3.4.1) — Ed25519 envelope, JCS payload hash, tagged search/write union, anchor-exclusive CHECK. Leads ADR-009 §5.
• rubrics + rubric_items + evaluations + evaluation_scores (§3.4.2) — immutable rubric versions, target-exclusive CHECK, verdict enum. Turns runs into ranked outcomes.
• event_rollups_daily (§3.4.3) — day × category × tool × agent_role × workspace grain, idempotent recompute inside one D1 batch, rollup_* classified errors. Dashboards no longer scan raw events.
• search_router_hints (§3.4.4) — leads ADR-007 §6; hint_kind discriminator, corroboration-edge variant, twin partial-unique identity indexes.
• Migration 0004 (§3.4.5) — receipt_id reconciliation: one-shot orphan scrub on migrate, ingest-time rc- prefix check, POST /api/telemetry/reconcile surface. No hard FK (D1 enforces PRAGMA foreign_keys = ON; a hard FK would fail-close the parallel-POST race legacy callers still take).
• @nexartis/ozzydev-receipts (mcp/packages/receipts) — shared receipt core consumed by both native MCPs: JCS canonicalisation, macOS Keychain custody, Ed25519 signing envelope, and the POST /api/receipts/with-event helper. Collapses the previous dual-POST paths in emitSearchReceipt / emitWriteReceipt.

The workspaceId attribution wave — a misattribution vector, closed

The substrate shipped and started collecting. What it collected was silently attributed to the wrong tenant for a window that only closed after the emitters were fixed at their source. This section is the durable record — the incident is instrumental, the principle it exposes is the payload.

• The defect: null workspace_id at emission → machine-global fallback → silent misattribution
  All three native MCPs (ozzydev-search, ozzydev-write, telemetry_emit) and the bin CLIs (ozzydev-checkpoint.mjs, telemetry-emit.mjs) emitted workspace_id: null. Three separate console ingest paths — POST /api/receipts/with-event, POST /api/telemetry/events, POST /api/telemetry/session-report — then fell back to a machine-global "active workspace" pointer. Receipts, tool_benchmark_events, session_reports, and harness_runs rows were silently attributed to whatever workspace happened to be flagged active, in practice a DIFFERENT workspace entirely. Nothing threw; the loop was measuring the wrong tenant.
• The durable lesson: when identity is knowable at emission time, EMIT it — never let a downstream guess
  A machine-global "active X" pointer is a misattribution vector for every fallback consumer, not just this one. The pointer is where the bug bit us; the principle generalises to session_id, agent_role, wave_slug, model_id, and any other identity a downstream sink might otherwise "helpfully" infer. Inference-at-ingest is defense-in-depth under a friendlier name (§2 AGENTS.md); the honest path is loud-fail-close on the emitter.
• The fix: resolveWorkspaceId hoisted into @nexartis/ozzydev-receipts, cached per process, threaded into BOTH ingest body AND signed common.workspace_id
  Operator chose emitter-side everywhere over console-side path inference. The receipts package POSTs {path} to /api/workspaces (register-or-reanimate, idempotent by path); the resolved ULID is cached per process and threaded into the ingest body AND the signed receipt envelope's common.workspace_id, so the two never disagree. Loud fail-close on every failure mode (receipt_workspace_unresolved) — no retry, no buffering, no degraded path. The console was left UNCHANGED — the router is not the place to paper over an emitter defect.
• CLI contract: --workspace-id flag or walk-up-from-CWD to kilo.json, then POST — zero-dependency preserved by inline duplication
  The bin CLIs (ozzydev-checkpoint.mjs, telemetry-emit.mjs) take --workspace-id when given; otherwise they walk up from CWD until they hit a workspace root marker (kilo.json), then POST to /api/workspaces the same way. The zero-dependency CLI contract was deliberately preserved (no runtime import of @nexartis/ozzydev-receipts from the bin scripts), which means intentional inline duplication of the resolveWorkspaceId shape. Any contract change to the resolver MUST be mirrored into both scripts — this is a maintenance tax that buys the CLIs' portability across repos without a workspace toolchain.
• Historical rows: LEFT and ANNOTATED, not mutated (operator decision — agents do not mutate D1)
  Rows emitted while the active pointer was wrong carry the OTHER workspace's ULID. They are not rewritten; a dated annotation in this page and in the D1 schema notes marks them as attribution-suspect for any downstream aggregation. The rule generalises: an agent that quietly rewrites history is an agent whose measurements cannot be trusted.
• Verified live: canary receipt landed under the correct workspace ULID end-to-end after the rebuild + host restart
  Emitter (native MCP) → /api/receipts/with-event → run_receipts.workspace_id and run_receipt_payloads.common.workspace_id both match the resolved ULID; the paired tool_benchmark_events row inherits it. Config_version bumped so the receipt population itself distinguishes pre-fix (attribution-suspect) from post-fix (verified) rows. This is the receipt-driven verification pattern working as designed — the substrate proves its own fix.

Known rough edges — the ingest contract the tool description doesn't tell you

Surfaced by the workspaceId wave and adjacent measurement work. Every future emitter will hit these; recording them here means the next author fixes the docs, not the symptom.

• telemetry_emit events kind
  Per-row category (search | write | scripts | verify | memory | orchestration | …) and ok (0|1) are BOTH required by ingest but are not called out in the tool description. Missing either → telemetry_batch_invalid with a row index but no field hint. Documented here until the CLI/help text catches up.
• session_reports upsert key = (session_id, agent_role, wave_slug)
  MEASURED 2026-07-27: subagent dispatches that omit --wave all upsert into ONE row — ~14 dispatches collapsed to 3 rows, with distinct agents returning the IDENTICAL row id and per-subagent telemetry lost silently. wave_slug is the discriminator; it is mandatory in every dispatch prompt, and the dashboard separates rows by agent_role. This is a second, independent instance of the same principle as the workspaceId wave: an identity that IS knowable at emission time must be emitted, or upserts collide.
• events waveSlug is a foreign key
  An unregistered waveSlug in an events batch → bare D1 500 with no classified error body. The wave must exist in D1 (via POST /api/waves or a prior session_reports upsert) before events referencing it land. Registering waves lazily inside the events endpoint was rejected — it would hide the FK failure that this contract is designed to surface.
• /api/doctor accepts workspace by PATH only
  GET /api/doctor?workspace=<ULID> → 403 forbidden. The endpoint keys off the resolved workspace PATH, not its ULID (an intentional consequence of the register-by-path model above). Callers that already have the ULID must round-trip it through GET /api/workspaces to recover the path — the ULID-first ergonomic will be added when the workspace registry gains a reverse-lookup index.

Live read-back — this page reads its own substrate

Fetched client-side from GET /api/experiments and GET /api/telemetry/events?limit=20 against the datastore this console runs on. On the Workers demo deploys these routes classify loudly (requires_local_runtime, admin_auth_required, or forbidden) and the error is shown verbatim below — the paper never fakes data it cannot read.

Loading live rows…

Corpus roadmap — the research-surface gaps

Inventory of research pages the platform still needs. Each becomes a paper_slug when it lands and immediately gets a read-back view through GET /api/experiments?paper_slug=….

• model-provider-routing — Which model for which category × operation, with cost/latency/accuracy evidence. Sibling of moe-search-router at the model layer.
• agent-definition-lifecycle — Agent cards live in KYM; how versions land, get benchmarked, and get promoted needs its own paper.
• agentic-regression-testing — How we lock in behavior once measured: fixture query sets, replay against tool events, guardrails against silent regression.
• agent-runtime-sandboxing — Cubicle host boundaries + Workers isolate. What each layer prevents, what only the host can, and where scripts-as-tools has to live.
• cost-budgets — dollars_micro is in the schema for a reason. Per-session/per-wave budgeting, alerting, and the operator-in-the-loop escape hatch.
• telemetry-observability — CLOSED (this paper) — This paper. CLOSED — substrate (0002), receipts + rubrics + rollups + hints (0003), and receipt_id reconciliation (0004) are shipped; consumers read back through paper_slug.
• pr-wave-methodology — The wave-scoped sprint / QA-gate-1&2 / atomic-promote convention deserves a published paper of its own, with measured wave outcomes.
• documentation-lifecycle — DRY-for-docs, canonical vs planning vs research, the assess→refactor→compress loop for AGENTS.md.
• local-runtime-operations — requires_local_runtime vs Workers-friendly boundary; doctor probes; the on-machine console operational surface.
• multi-machine-console — How multiple operator machines share the same D1 datastore while keeping machine-scoped state honest (migration 0001 groundwork).
• session-handover-continuity — Handover-as-checkpointing (context-engineering §Repair) formalized: schema for the handover doc, verifier probes, round-trip fidelity metrics.
• operator-in-the-loop — Where humans stay decisive: PR approval / merge, prod credentials, budget breach, Linear ticket mutations. The consent surface across the whole loop.

Changelog

• v0.7 (2026-07-30) — workspaceId attribution wave: three native MCPs and both bin CLIs were emitting workspace_id: null, and three ingest paths silently fell back to a machine-global "active workspace" pointer, misattributing receipts + tool_benchmark_events + session_reports to whichever workspace was flagged active. Cured by hoisting resolveWorkspaceId into @nexartis/ozzydev-receipts (POST {path} to /api/workspaces, cached per process, threaded into ingest body AND signed common.workspace_id), with loud fail-close (receipt_workspace_unresolved) on every failure mode. Console left UNCHANGED; historical rows LEFT and annotated (operator decision — agents do not mutate D1). Verified live end-to-end. Recorded here as the durable lesson: emit identity you know at the source; a fallback pointer is a misattribution vector for every consumer that trusts it. Second, independent finding: session_reports is keyed (session_id, agent_role, wave_slug), so subagent dispatches that omit --wave collapse into one row (measured ~14 → 3 with duplicate ids and lost per-subagent telemetry) — wave_slug is the discriminator and is mandatory. Rough-edges section added for the undocumented events category/ok requirement, the waveSlug FK-500, and /api/doctor\'s path-only workspace parameter.
• v0.4 (2026-07-16) — Migrations 0003 + 0004 shipped: receipts pair, rubrics / evaluations, event_rollups_daily, search_router_hints, and the receipt_id reconciliation slice (no hard FK; loud reconcile surface). Shared @nexartis/ozzydev-receipts package + POST /api/receipts/with-event collapse dual-POST paths for both native MCPs. ADR-009 kept normative; page narrates.
• v0.3 (2026-07-12) — ADR-009 locked as the single normative home for the shared receipt + benchmark contract; corpus roadmap entry closed.
• v0.2 — Emission loop closed: agents post via bin/telemetry-emit.mjs (workspace AGENTS.md §5 convention); this page reads its own rows back.
• v0.1 — Migration 0002 six-table substrate + batch ingest documented; corpus roadmap seeded.