How neural TTS works

Modern TTS systems are end-to-end neural models that learn a direct mapping from text (or phoneme sequences) to audio waveforms. The dominant architecture in 2026 combines a text encoder, a duration predictor, a mel-spectrogram generator, and a neural vocoder (such as HiFi-GAN or EnCodec) that converts the spectrogram to a time-domain audio signal. Voice cloning systems add a speaker encoder that extracts a speaker embedding from a reference audio clip — allowing the model to reproduce any voice from as little as 3–10 seconds of sample audio.

from elevenlabs import ElevenLabs, Voice, VoiceSettings

client = ElevenLabs(api_key="YOUR_API_KEY")

# Generate speech using a preset voice
audio = client.generate(
    text="Welcome to LumiChats — premium AI at coffee prices.",
    voice=Voice(
        voice_id="21m00Tcm4TlvDq8ikWAM",  # Rachel — natural, conversational
        settings=VoiceSettings(
            stability=0.5,       # 0 = expressive, 1 = consistent
            similarity_boost=0.75,  # how closely to match reference voice
            style=0.2,           # speaking style exaggeration
        )
    ),
    model="eleven_multilingual_v3",
    output_format="mp3_44100_128",
)

# Save to file
with open("output.mp3", "wb") as f:
    for chunk in audio:
        f.write(chunk)

Voice cloning and consent: Voice cloning can reproduce anyone's voice from a short audio sample. The legal landscape in 2026: the US federal NO FAKES Act (pending) and Tennessee's ELVIS Act (in force) protect individuals against non-consensual voice cloning for commercial use. The EU AI Act classifies high-quality voice cloning as a biometric system subject to transparency requirements. Always obtain explicit consent before cloning anyone's voice for any use.

Evaluation: what makes TTS good

Engine comparison: language coverage beyond English

MOS scores are measured overwhelmingly on English, so they tell you almost nothing about the question most teams actually face: does this engine speak my users' language well? Coverage differs by an order of magnitude between vendors, and — critically — within a single vendor between model tiers.

The Telugu trap, and the deeper one behind it: Telugu on ElevenLabs exists only on Eleven v3 — it is not in Multilingual v2's 29-language list, nor in the low-latency Flash/Turbo models. So a real-time Telugu voice agent is not simply "slower" on ElevenLabs; it is unavailable. Worse, ElevenLabs' phoneme tags are documented as compatible only with eleven_flash_v2 — an English-only model. The consequence is stark: there is no deterministic pronunciation control for Telugu or Chinese on ElevenLabs at all. Its v3 alternative is inline IPA, which the docs themselves describe as achieving "80–90% pronunciation consistency" — vendor-admitted non-determinism. If a name or term must be pronounced correctly every single time in a non-English language, that is an architectural dealbreaker, not a tuning problem.

Chinese TTS: polyphony, and what the benchmarks actually say

Mandarin's hard problem is polyphonic characters (多音字): one character carries several pronunciations depending on usage, so the engine must disambiguate from context before it can speak. The surname 仇 is read qiú, but the same character as a common noun ("hatred") is read chóu. Get it wrong and you have mispronounced a customer's name. Grapheme-to-phoneme polyphone disambiguation has been an active research problem for two decades and remains one — recent work still attacks it with BERT-based and LLM-augmented disambiguation.

This is exactly where SSML earns its keep. Azure documents a targeted control for it: <say-as interpret-as="name">仇先生</say-as> forces the surname reading qiú instead of chóu. Azure also exposes phoneme tags and custom lexicons (with a sapi phone set defined for zh-CN), so a fixed vocabulary of names and product terms can be pinned deterministically. ElevenLabs offers no equivalent for Chinese.

Honest answer on "Azure vs ElevenLabs for Chinese": A frequently searched comparison — and no objective head-to-head benchmark exists. Every "Azure vs ElevenLabs Chinese quality" page you will find is vendor-authored or anecdotal. The closest genuine evidence is CN-NewsTTS Bench (arXiv, June 2026), which auto-evaluated seven Chinese TTS systems — and ElevenLabs was not among them. In it, Azure resolved every test target with zero majority-voted errors; its headline number is depressed by scorer coverage gaps rather than mistakes, so it should not be read as an error rate. Two further caveats: that benchmark is a single-author v0.1 preprint, and it measures text normalization (numbers, units, abbreviations) — not polyphony. The defensible conclusion is about capability, not scores: Azure gives you deterministic tools to fix Chinese pronunciation; ElevenLabs asks you to trust the model.

Is HiFi-GAN still used? (And the ElevenLabs SDK change that breaks old code)

Yes — the classic two-stage pipeline is not a historical footnote. Azure Neural TTS runs Text Analyzer → neural acoustic model (FastSpeech/Uni-TTS lineage) → neural vocoder, where the vocoder is HiFiNet, Microsoft's HiFi-GAN-derived model; HiFiNet2 is what upgraded 400+ voices to 48 kHz. Even the end-to-end alternative doesn't escape it: VITS removes the external vocoder, but its decoder is a stack of transposed convolutions built "much in the same style as the HiFi-GAN vocoder." HiFi-GAN didn't die — it got absorbed.

On ElevenLabs' architecture: ElevenLabs does not publish it — no papers, no model cards, no technical docs on internals. Blog posts claiming specific architectures ("latent diffusion + transformer") trace to no primary source and should be treated as rumor. The only defensible inference comes from documented behavior: their handling of markup and their IPA consistency caveat are consistent with end-to-end training on real speech rather than a rule-based phoneme pipeline.

# ❌ OLD (pre-v2.0) — this now raises AttributeError, it does not warn
# from elevenlabs import generate
# audio = client.generate(text="Hello", voice="Rachel")

# ✅ CURRENT (SDK v2.0+, since ~May 2025)
from elevenlabs.client import ElevenLabs

client = ElevenLabs(api_key="...")        # class is ElevenLabs, not ElevenLabsClient

audio = client.text_to_speech.convert(
    text="Hello world",
    voice_id="21m00Tcm4TlvDq8ikWAM",     # voice_id — NOT voice=
    model_id="eleven_v3",                 # v3 for Telugu/Gujarati; v2 lacks them
    output_format="mp3_44100_128",
)

# Instant voice cloning also moved:
#   old: client.clone(...)      → removed
#   new: client.voices.ivc.create(...)

Why your old ElevenLabs snippet fails: The v2 upgrade guide states plainly that generate and clone were removed from the SDK — not deprecated with a warning. Any tutorial using client.generate(...) predates v2.0 (May 2025) and will error outright on a current install (v2.58.0 as of July 2026). Three migration breaks catch people: the method is now client.text_to_speech.convert, the parameter is voice_id rather than voice, and cloning moved to client.voices.ivc.create.

Use cases by sector in 2026

• Content creation: YouTube creators, podcast producers, and audiobook publishers use TTS to generate narration in minutes rather than booking studio time. ElevenLabs is the market leader for creator-focused voice generation.
• AI customer service: Voice AI agents using TTS for real-time speech synthesis are replacing IVR (Interactive Voice Response) systems. Latency below 300ms is now achievable, enabling natural conversation flow.
• Accessibility: TTS enables screen readers, reading assistance for dyslexia, and audio description for visually impaired users across all languages — Google Cloud TTS covers 220+ languages.
• Language learning: Natural TTS pronunciation models are integrated into Duolingo, Babbel, and dedicated pronunciation training apps.
• Indian language support: Google Cloud TTS Chirp HD and Microsoft Azure Neural TTS cover all major Indian languages including Hindi, Tamil, Telugu, Kannada, Malayalam, Bengali, Marathi, and Gujarati with natural prosody.

Practice questions

1. What is the difference between concatenative TTS, parametric TTS, and neural TTS? (Answer: Concatenative TTS: splice together recorded speech segments from a large audio database — high naturalness for covered phrases, robotic for novel combinations. Parametric TTS: model acoustic features (mel spectrograms) with HMMs or early neural networks — smoother but robotic. Neural TTS (WaveNet, Tacotron, VITS): end-to-end neural generation from text to waveform or via intermediate mel spectrogram. Near-human naturalness. Current state-of-the-art is neural: ElevenLabs, Azure Neural TTS, Google WaveNet, Amazon Polly Neural.)
2. What is voice cloning and what are the ethical concerns? (Answer: Voice cloning: train a TTS model on as little as 3–60 seconds of a target speaker's voice, enabling generation of arbitrary speech in that voice. Applications: accessibility (restore lost voices), personalization, entertainment. Ethical concerns: deepfake audio for fraud (fake CEO calls authorising wire transfers), non-consensual content (generating fake statements), political disinformation (fake politician speeches). Many jurisdictions are legislating consent requirements for voice cloning. ElevenLabs requires users to confirm ownership or consent for cloned voices.)
3. What is the role of the vocoder in neural TTS systems like Tacotron 2? (Answer: Tacotron 2 architecture has two parts: (1) Sequence-to-sequence model: converts text to mel spectrogram (acoustic features). (2) Vocoder (WaveNet/WaveGlow/HiFi-GAN): converts mel spectrogram to audio waveform. The vocoder's job is to synthesise the raw audio sample-by-sample from the abstract mel spectrogram representation. Early neural vocoders (WaveNet) were too slow for real-time (1 second audio took 2 minutes). HiFi-GAN achieves real-time synthesis at 100× speed.)
4. What is prosody in TTS and why is it hard to get right? (Answer: Prosody = the patterns of stress, intonation, rhythm, and emphasis in speech. 'I never said she stole the money' has 7 different meanings depending on which word is stressed. TTS systems trained on flat, neutral speech may correctly pronounce words but place stress incorrectly or use monotone intonation. Modern approaches: (1) Explicit prosody control via markup (SSML tags). (2) Emotion/style conditioning (train on diverse emotional speech). (3) In-context TTS (ElevenLabs): provide a short reference audio clip to match prosody style.)
5. What are the key differences between ElevenLabs, Azure TTS, and Coqui/open-source TTS for production deployment? (Answer: ElevenLabs: highest quality, most realistic voices, voice cloning in 1 minute of audio, multilingual. Cost: ~$0.24/1000 characters. Latency: 200–500ms. Azure Neural TTS: enterprise SLA, compliance certifications (HIPAA, GDPR), 400+ voices, custom neural voice. Cost: ~$0.016/1000 characters (much cheaper). Coqui TTS/XTTS (open source): free, self-hosted, privacy-preserving, high-quality voice cloning. Cost: infrastructure only. Latency depends on hardware. Best for: ElevenLabs=quality; Azure=enterprise; Coqui=privacy+cost.)
6. You need a real-time Telugu voice agent with sub-300ms latency. Why does ElevenLabs fail this requirement? (Answer: Telugu exists only on Eleven v3 — it is absent from Multilingual v2 (29 languages) and from the low-latency Flash/Turbo v2.5 models (32 languages). So the language and the latency tier do not intersect: you cannot get Telugu on a Flash model. Azure offers te-IN-ShrutiNeural / te-IN-MohanNeural, and Google Chirp 3 HD also supports te-IN.)
7. A customer surname must be pronounced identically on every call, in Chinese. Which engine can guarantee it, and why? (Answer: Azure. It supports full SSML — phoneme tags, custom lexicons (PLS), and specifically which forces the surname reading of polyphonic characters (仇 as qiú, not chóu). ElevenLabs cannot: its phoneme tags are documented as compatible only with eleven_flash_v2, an English-only model, so there is no deterministic pronunciation control for Chinese; its v3 IPA alternative is described by its own docs as achieving 80–90% consistency.)
8. Is the acoustic-model-plus-vocoder pipeline obsolete in 2026? (Answer: No. Azure Neural TTS ships it at 600-voice scale: Text Analyzer → neural acoustic model (FastSpeech/Uni-TTS lineage) → HiFiNet vocoder, a HiFi-GAN derivative; HiFiNet2 raised 400+ voices to 48kHz. Even end-to-end VITS, which drops the external vocoder, builds its decoder from transposed convolutions in the HiFi-GAN style. The architecture was absorbed, not replaced.)
9. Legacy code calls client.generate(text=..., voice="Rachel") against the ElevenLabs SDK and crashes. Why? (Answer: generate() and clone() were REMOVED — not deprecated — in SDK v2.0 (~May 2025), so the call raises an error rather than warning. Current usage: client.text_to_speech.convert(text=..., voice_id=..., model_id=...), with instant voice cloning at client.voices.ivc.create(). Note the parameter is voice_id, not voice — a frequent migration break.)