Two-Way Voice Control of Agentic Systems — UX Research

Voice-first agentic sessions use a speak channel for agent-to-operator progress updates. This page scaffolds the UX research agenda around two-way voice control of parallel agentic waves: what works today, where voice beats screen, and the open questions on two-way command and multi-orchestrator disambiguation.

Context

• Voice-first sessions: agents speak progress via the nexartis-remote-control (NRC) speak channel; a human practitioner listens ambient while a wave runs.
• Parallel-wave orchestrations produce field observations about which voice affordances help, which get in the way, and which product ideas emerge.
• Intended integration surface: DevFlow wizard step 2 (voice + agent flow).
• Scope: two-way voice control of agentic systems only. General TTS/STT infrastructure choice lives elsewhere.

Known patterns so far

• Voice updates as session narrative — Short, number-light summaries at wave boundaries, before/after validation, and before commit/push. Concise sentences beat dense telemetry. Numbers, hashes, IDs are omitted per the workspace voice-first rule (never speak secrets, hashes, PIDs, deploy IDs).
  Narrative can be absorbed at ambient volume; numeric detail requires screen focus. Narrative → screen only when narrative flags something.
• Queued updates, non-interrupting playback — Voice updates queue and play sequentially; a new update does not interrupt the current utterance. The queue is the pacing mechanism.
  Interruption is jarring and destroys situational trust. Queueing keeps the narrative coherent.
• When voice beats screen — ambient monitoring during parallel waves — Multiple orchestrator sessions running concurrently; the practitioner is away from the terminal; voice enables monitoring several waves without eye-time.
  The load-bearing use case. Everything else derives from it.
• When voice fails — Long lists of file paths, diffs, PR URLs, exact numeric SHAs — anything that must be read back or copied. Voice for these degrades to "there is a thing, look at the screen" (which is the correct pattern).
  Voice is a signal channel, not a transport for data. Attempting to speak data is the anti-pattern.

Open UX questions

• Two-way control — voice commands in — Today voice is agent-to-operator only. Operator-to-agent voice (interrupt, redirect, approve, halt) is the obvious next step. Open questions: wake word vs push-to-talk vs terminal-focus gating; command grammar (freeform vs closed set); disambiguation across concurrent sessions.
• Interruption semantics — If the practitioner speaks mid-agent-utterance, does the agent stop talking? Does it acknowledge and defer? Does it stop the underlying work? Different answers for "correction" vs "halt" vs "just checking in".
• Multi-orchestrator disambiguation — Parallel waves = multiple speaking agents = multiple voices. Options: per-session voice assignment (different ElevenLabs voice per orchestrator), session-tag prefixes on updates, single-active-speaker routing.
• Voice + wizard flows in ozzydev — DevFlow wizard (step 2) is the intended integration point. What does a voice-first wizard step look like? Progressive disclosure via voice (agent narrates the choice; practitioner picks by voice or click)?
• Confirmation gates — Destructive actions currently require typed approval. Voice confirm-then-proceed may be admissible for some class; others stay typed-only. The workspace hard rule (agents never merge PRs to dev/prod) does not change; the question is which non-merge gates voice can carry.
• Cognitive load calibration — How much voice is too much? Ambient monitoring depends on updates being rare-enough to notice. Empirical tuning by session length + wave count is open.

Future work

Product research placeholders for later revisions. DevFlow wizard step 2 is the intended integration point.

• DevFlow wizard step 2 — voice + agent flow.
• Per-session voice assignment scheme (multi-orchestrator UX).
• Voice command grammar (closed-set MVP → open-set follow-up).
• Interruption semantics decision matrix (correct / halt / defer / clarify).
• Session-tag prefix format for spoken updates in multi-orchestrator mode.
• Confirmation-gate taxonomy — which gates accept voice, which stay typed-only.
• Structured capture of practitioner observations from live voice-driven sessions.

Prior-art scan planned

• OpenAI Realtime API — server-side voice agent primitives (interruption, function calling)
• ElevenLabs Conversational AI — realtime agent with voice + tool call
• Anthropic Claude — voice mode UX (turn-taking, interruption behaviour)
• Voice-first developer tools (Whispering, SuperWhisper, Wisprflow) — dictation vs command
• CLI voice control prior art (Talon Voice, Cursorless) — closed-grammar command sets
• Multi-agent voice UX (nothing published we consider adequate — likely a gap to fill)

Next-revision expansion plan

1. Capture structured field observations from voice-first sessions.
2. Draft the voice command grammar MVP (closed set) — halt, approve, status, switch-session, repeat.
3. Decide multi-orchestrator disambiguation approach (per-session voice vs prefix).
4. Enumerate confirmation-gate taxonomy: which gates voice-approve, which stay typed.
5. Sketch DevFlow wizard step 2 with the voice + agent flow explicit.
6. Prior-art scan (voice-first dev tools, realtime agent APIs) with links and specific affordances borrowed.
7. KPI registration on the experimental-framework matrix — voice-updates-per-wave, operator-interrupt-rate, gate-latency-with-voice.

How this page stays current

Living outline. The current revision scaffolds context, known patterns, and open questions. Later revisions land the command grammar MVP, the multi-orchestrator disambiguation decision, and the DevFlow wizard step 2 sketch.