Speech AI for telephony: speech recognition, synthesis and language understanding

Speech AI for telephony is not the same problem as speech AI in a mobile app or a meeting tool. Phone audio is narrowband, compressed and often noisy; callers talk over prompts, pause mid-sentence and spell names; and every hundred milliseconds of silence is noticeable. Voxivra integrates speech-to-text (STT), text-to-speech (TTS) and natural language understanding (NLU or LLM-based dialogue) into Asterisk and SIP platforms, with the telephony engineering that makes these components work on real calls.

The voice pipeline on a phone call

A conversational voicebot, also called a callbot, chains several components in real time:

1. Media access: call audio is streamed out of the telephony platform, for example with Asterisk ARI ExternalMedia, to a media gateway.
2. Voice activity detection (VAD) and endpointing: deciding when the caller is speaking and when they have finished their turn.
3. Streaming STT: transcribing audio as it arrives, with partial results long before the caller stops talking.
4. Understanding and dialogue: extracting intent and entities (NLU), or letting an LLM interpret the request and decide the next step, with tool calls into your CRM, calendar or billing system.
5. TTS: generating the answer as speech, streamed back into the call as soon as the first audio is ready.
6. Barge-in control: stopping playback the moment the caller interrupts.

Each stage is a component you can choose and tune. The architecture around them is what we build. For a step-by-step explanation aimed at a broader audience, see our article on how AI voicebots work.

Speech recognition (STT) on phone audio

Most phone calls still reach the platform as G.711 at 8 kHz, which removes high frequencies that recognition models rely on for consonants. Wideband codecs such as G.722 or Opus help when the whole path supports them, but you cannot count on it for calls from the public network.

What we do to get reliable transcription:

• Benchmark on your audio. We measure word error rate and, more importantly, task success (correct account number, correct date) on a sample of real calls, in each language you serve.
• Custom vocabulary and hints: product names, street names, medication names, internal codes.
• Constrained recognition where it matters. For account numbers, postal codes or dates, we combine speech with DTMF, use grammars or digit-specific models, and always read back critical values for confirmation.
• Tuned endpointing. Too aggressive and the bot cuts callers off; too lax and every turn has dead air. We adjust silence thresholds per question type.

Speech synthesis (TTS)

Neural TTS now sounds natural in both French and English, including Canadian voices. The engineering points are latency and control:

• Streaming synthesis, so playback starts on the first audio chunk instead of waiting for the full sentence.
• Sentence-level chunking of the dialogue output, so the bot starts speaking while the rest of the answer is still being generated.
• SSML or equivalent controls for pronunciation of numbers, amounts, dates, acronyms and proper names.
• Caching of static prompts, which removes TTS latency entirely for greetings and fixed messages.

Understanding: NLU, LLMs and guardrails

Classic NLU (intents and entities) is predictable and easy to test. LLMs handle free-form, multi-intent requests much better and can phrase natural answers. In production we usually combine them:

• the LLM interprets the caller’s request and fills structured fields;
• business logic in code validates those fields, calls your APIs and decides what is allowed;
• sensitive steps (identity verification, payment, consent) run as deterministic flows. Card numbers, for instance, are best collected by DTMF with masking rather than by voice, which also limits what reaches the AI providers. Obligations under PCI-DSS or privacy laws depend on your context and should be validated with your compliance advisors.

We also design the escalation path: when confidence drops, the caller repeats themselves, or the request is out of scope, the call goes to a human with the transcript and collected data.

Latency budget: where the milliseconds go

Perceived responsiveness is the difference between a voicebot callers accept and one they hang up on. The delay the caller hears is the sum of:

Our levers: stream every stage, run independent steps in parallel, host components close to each other (and to your callers, when data residency allows), pre-fetch customer data at call start, and use short filler phrases only when a slow backend call is unavoidable. We measure each stage per turn in production, so latency regressions are visible immediately.

Data protection and hosting

Voice data is personal information, and transcripts often contain more sensitive details than the audio’s metadata. We help you decide which components can be cloud-hosted, which should run in Canada or in your own infrastructure, what is stored, for how long, and how it is redacted. The rules that apply (Law 25 in Quebec, PIPEDA, GDPR in Europe, sector-specific requirements) depend on your situation, and we recommend confirming them with legal counsel.

When Voxivra recommends speech AI, and when not

Speech AI is a good fit when callers’ requests are varied and hard to map to a menu, when you want to automate common requests end-to-end, or when you want to transcribe and analyze calls at scale. It is less useful when a short DTMF menu already serves callers well, or when audio conditions are consistently poor. In those cases, a well-designed IVR may be the better investment.

On real projects, the results come from the whole system: for example, our healthcare appointment voicebot automates 80% of calls. To explore a voicebot for your own use case, see our AI voicebot service or book a free 30-minute consultation.