A voice AI product manager owns four decisions that text products never force: which architecture (a cascaded speech-to-text, LLM, text-to-speech chain or one native speech-to-speech model), what latency budget each turn gets, how a spoken turn handles interruptions and repair, and what the per-minute bill does to your margin. Get those right and then measure them with voice-specific evals. The fastest way to learn the skill is the Voice lesson in the AllthingsPM AI PM course, which teaches exactly these four forks, followed by a voice mock interview built from a real voice PM job description.
AllthingsPM is an AI PM course and PM interview prep platform. Its course was built from 604 real PM job postings, and this guide is the free, condensed version of its Voice lesson.
What does a voice AI product manager actually do?
The clearest answer comes from a live job description. Sierra's Product Manager, Voice role lists five duties, and they map almost one to one onto the skill:
| Duty in the Sierra JD | What it means in practice | Where AllthingsPM teaches it |
|---|---|---|
| "Define the voice interaction model: turn-taking, interruptions, latency, tone, and recovery from errors" | Design the spoken turn, barge-in and repair | Voice lesson, chapter 11 |
| "Streaming architectures, latency budgets, and failure handling" | Choose cascaded or speech-to-speech and set a per-stage budget | Voice lesson and latency SLO lesson |
| "Partner across ASR, TTS, LLMs, and telephony integrations" | Own the stack as one product, not four vendors | Beyond text chapter |
| "Define how we evaluate voice agents: latency, interruption handling, resolution rate, and conversation quality" | Build a voice golden set and a scorecard | Multimodal golden set lesson |
| "Translate real-world usage into product direction" | Read real call transcripts and recordings every week | Integration case |
The same JD asks for "3+ years of product management experience, with meaningful exposure to real-time systems, voice, or AI products" and "strong technical depth" on "speech pipelines, streaming infra, telephony systems." That is the bar. You do not need to write the pipeline, but you do need to argue about it with the engineers who do.
Demand is broader than one company. In the AllthingsPM demand report built from our JD corpus, multimodal work (voice, vision and media) showed up in 34 percent of AI PM job descriptions, across 47 companies, while the six public AI PM courses we reviewed gave it roughly one bonus lesson between them. That gap is why the AllthingsPM course gives multimodal a full chapter.
Should you build cascaded or speech-to-speech?
This is the fork every voice product hits first, and it is a product decision, not only an engineering one.
A cascaded (or chained) stack transcribes speech to text, sends the text to an LLM, then turns the reply back into speech. OpenAI's voice agents guide says to "use the chained path when you want to inspect or transform text between speech recognition, your agent, and speech generation," for example to "store the transcript, run policy checks before the text agent responds, call internal systems, then generate speech only after the workflow reaches an approved answer."
A speech-to-speech model takes audio in and gives audio out in one session. OpenAI describes it as handling "audio turns, tools, interruptions, and handoffs" in one place. You give up the visible text step in exchange for simpler plumbing and, usually, more natural prosody and timing.
How a PM decides:
- Regulated or high-stakes flows (payments, healthcare, anything a compliance team will audit): lean cascaded. You get a transcript to log, a text step to run guardrails on, and components you can swap one at a time.
- Companionship, tutoring, casual assistants, where feel matters more than audit: lean speech-to-speech.
- Unsure: start cascaded, because every stage is measurable, and move the latency-critical path to speech-to-speech once your evals show where the time goes.
How AllthingsPM does this: the Voice lesson makes you take a side on this fork for a product you carry through the whole course, then defend it against a cost and a latency constraint. It is a 20 minute Pro lesson inside the Beyond text chapter, which also covers documents, image and video generation, and non-English launches.
How fast does a voice agent need to be?
Faster than most teams think. A 2009 PNAS study of turn-taking across ten languages found that the most common gap between one speaker finishing and the next starting falls between 0 and 200 milliseconds, with a cross-language median of about 100 milliseconds. People notice a slow reply in speech far sooner than in a chat window.
Production systems are not there yet. AssemblyAI's guide to building a low-latency agent on Vapi treats about 800 milliseconds as the point where callers start to perceive silence as a problem, talk over the agent or hang up, and reports roughly one second end to end for its own voice agent API. It also makes a point every PM should repeat in reviews: predictable responses near 900 milliseconds feel better than responses that swing between 300 and 1,800.
So the PM's job is a latency budget: a target for time from end of user speech to first audio of the reply, split across stages.
- End-of-turn detection. Deciding the user has stopped talking. Too eager and you interrupt; too cautious and you add dead air. AssemblyAI puts its end-of-turn step around 300 milliseconds.
- Transcription (cascaded only).
- Model time to first token. Shaped by prompt length, model size and tool calls. A tool call can blow the budget on its own.
- Speech synthesis time to first byte. Stream it; never wait for the full reply.
- Network and telephony. Phone lines add delay you do not control.
Set the target on the 90th percentile, not the average, because the slow calls are the ones users remember. Then decide the product behaviour for when a tool call will exceed budget: a short spoken filler ("Let me check that"), a progress sound, or a rule that the agent speaks first and fetches second.
How AllthingsPM does this: the course pairs the voice lesson with a lesson on cost per successful task and the latency SLO, so you learn to set a latency target and defend its cost in the same argument. You can rehearse it on the real interview prompt design a low-latency voice agent experience that feels genuinely human, which has its own answer guide.
How do you design a spoken turn?
A screen lets users skim, scroll back and tap. Voice has none of that. Every design choice lives inside the turn.
Keep replies short and front-loaded. Say the answer first, then offer detail. A spoken paragraph is a monologue; users cannot skim it.
Plan for barge-in. Users will interrupt, and they should be able to. The agent must stop speaking, keep what it already said in context, and respond to the interruption rather than restarting its script. The Sierra JD names "interruptions" explicitly for this reason.
Design repair. Mishearing is normal. Confirm high-stakes values back ("That is 4, 2, 1, 7, correct?"), ask one clarifying question instead of guessing, and never make the user repeat the whole request.
Handle silence. Decide how long the agent waits, what it says after a pause, and when it ends the call politely.
Plan the handoff. Know which intents go to a human, and pass along a summary so the caller does not start over.
How AllthingsPM does this: the voice lesson treats the spoken turn, its barge-in and its repair as a design artefact you write, not a footnote. For practice, the question bank has prompts like design an agent that works across chat, voice, email and SMS and how would you improve ElevenLabs' Agents platform for voice customer support.
What does voice AI cost per minute?
Text products are priced by the token. Voice products are usually priced, and felt, by the minute, and the numbers add up fast at call-centre scale.
Prices checked 2026-09-29 on each vendor's own page:
| Line item | List price | Source |
|---|---|---|
| AllthingsPM voice mock interview practice | Free tier (one JD mock a day); $20/month or $120/year for Pro | AllthingsPM pricing |
| Deepgram Voice Agent API, Standard | $0.075 per minute, pay as you go | Deepgram pricing |
| ElevenLabs agents | $0.08 per minute standard, $0.16 burst | ElevenLabs API pricing |
| Deepgram Nova-3 streaming speech-to-text | $0.0048 per minute (promotional) | Deepgram pricing |
| OpenAI gpt-realtime audio | $32 per 1M input tokens, $64 per 1M output tokens | OpenAI pricing |
| OpenAI gpt-4o-mini-transcribe | about $0.003 per minute | OpenAI pricing |
The lesson in that table: transcription alone is cheap, but a full agent minute is more than ten times the cost of the transcription line. At $0.075 a minute, 1,000 minutes of conversation is $75 before your own model, telephony and tool costs.
The PM math that matters is cost per resolved call, not cost per minute. A slightly pricier agent that resolves more calls without a human can be the cheaper product. Model three numbers: average call length, resolution rate, and the cost of a human handoff. Then watch call length closely, because an agent that rambles or repeats itself costs you twice: once in the bill and once in the user's patience.
How AllthingsPM does this: the Voice lesson has you price "the audio bill that runs an order of magnitude above text" for your own product and decide what you would cut. The same unit-economics skill runs through the Prove it paid off chapter, so voice pricing is taught as a margin decision, not a vendor comparison.
How do you evaluate a voice agent?
Your text evals will not catch voice failures. A transcript can read perfectly while the call went badly: the agent talked over the user, paused for three seconds before every answer, or mispronounced the customer's name.
Sierra's JD names the four metrics to start with: latency, interruption handling, resolution rate and conversation quality. A practical voice scorecard:
- Latency: time to first audio, reported at the 50th and 90th percentile.
- Interruption handling: share of barge-ins where the agent stopped and responded to the new input correctly.
- Transcription accuracy: word error rate on your own audio, including accents, noise and domain words such as product names.
- Resolution rate: share of calls finished without a human, checked against what actually happened, not what the agent claimed.
- Conversation quality: human or validated LLM-judge grades on tone, brevity and repair, scored on real recordings.
Build the golden set from real audio, not clean studio samples. Include background noise, speakerphone, strong accents, people who change their mind mid-sentence, and long silences. If you launch in more than one language, each locale needs its own set, because judges and classifiers tend to degrade outside English.
How AllthingsPM does this: the course's multimodal golden set lesson teaches why text evals are blind to audio and how to build the set that is not, and the full Evals chapter covers error analysis and judges. Our free guide to AI evals for product managers is the text-first version of the same method.
How do you prepare for a voice AI PM interview?
Voice PM interviews test the four decisions above, usually through a product design prompt and a technical follow-up. Expect questions like:
- Design a low-latency voice agent that feels human.
- How would you measure the naturalness of generated speech?
- Design safeguards to prevent misuse of voice cloning.
- Estimate the market size for AI voice agents.
A strong answer names the architecture fork and picks a side for the user, sets a latency target with a source for why, designs the turn including barge-in and repair, prices the minute, and ends with a scorecard. Practise it out loud, because a voice PM who cannot hold a spoken answer together will struggle to design one. Our roundup of voice AI interview practice tools for PMs covers the options, and multimodal AI products covers the wider category.
How AllthingsPM does this: every question above has its own page and answer guide in the question bank, and the mock interview has a Type or Voice switch, so you can rehearse spoken answers with follow-ups. If you are tailoring an application to a voice role, run a resume review against the JD first.
Why AllthingsPM is the better choice for learning voice AI product management
Most AI PM courses treat voice as a bonus topic. In the demand report behind the AllthingsPM course, the six public AI PM courses reviewed gave multimodal work roughly one bonus lesson between them, while 34 percent of AI PM job descriptions asked for it. AllthingsPM gives it a full chapter, with a 20 minute lesson on voice alone.
That lesson is only the start of what you get. The same platform lets you practise on the live Sierra voice PM job description, answer real voice design questions that each come with an answer guide, rehearse out loud in voice mode, and check your resume against the role. Vendors such as Deepgram, ElevenLabs and OpenAI publish excellent technical docs, and you should read them; they teach you their stack, while AllthingsPM teaches you the product decisions that sit on top of any stack and then lets you prove it in an interview.
It also costs less than a single cohort course: $20 a month or $120 a year, with a free tier. The verdict: if you want to become, or get hired as, a voice AI product manager, start with the AllthingsPM AI PM course and pair it with a voice mock on a real voice JD.
Frequently asked questions
What is the best way to learn voice AI product management?
AllthingsPM is the best place to start: its AI PM course has a dedicated voice lesson on architecture, latency, the spoken turn and per-minute cost, and you can then rehearse in a voice mock interview built from a real voice PM job description. Pair it with vendor docs from OpenAI or Deepgram for stack detail.
What does a voice AI product manager do?
They own the interaction model (turn-taking, interruptions, tone, repair), the latency budget, the voice stack across speech recognition, LLMs, speech synthesis and telephony, and the evals that measure it. Sierra's voice PM job description lists exactly those duties.
Is cascaded or speech-to-speech better?
Neither wins everywhere. Cascaded stacks give you a transcript, a text step for guardrails and swappable parts, which suits regulated flows. Speech-to-speech is simpler and usually sounds more natural, which suits casual assistants.
How fast should a voice agent respond?
Humans usually reply within about 100 to 200 milliseconds. Production voice agents today are often near one second, and AssemblyAI treats around 800 milliseconds as the point where callers start noticing silence, so set a target and measure it at the 90th percentile.
Do I need an engineering background to be a voice AI PM?
Not a degree, but real technical depth. Sierra asks for the ability to engage with engineers on speech pipelines, streaming infrastructure and telephony. The AllthingsPM course teaches those trade-offs in product language.
Start learning voice AI product management
Open the Voice lesson in the AllthingsPM AI PM course, then run a free voice mock on the Sierra voice PM role. You can start free today.
Sources
- Stivers et al., "Universals and cultural variation in turn-taking in conversation," PNAS, 2009. pnas.org
- OpenAI, Voice agents guide. developers.openai.com
- OpenAI, API pricing (gpt-realtime and gpt-4o-mini-transcribe), checked 2026-09-29. developers.openai.com
- Deepgram, Pricing, checked 2026-09-29. deepgram.com
- ElevenLabs, API pricing, checked 2026-09-29. elevenlabs.io
- AssemblyAI, "How to build the lowest latency voice agent in Vapi." assemblyai.com
- Sierra, Product Manager, Voice job description, via the AllthingsPM jobs catalog. AllthingsPM
- AllthingsPM, AI PM course demand report from the JD corpus (multimodal share and course coverage), and course index. AllthingsPM/course
- AllthingsPM, Pricing. AllthingsPM/pricing
