The Complete Map of AI Audio Models

Categories, Models, Capabilities, Costs, and Licensing

Saeed Sheikh Amiri · November 2026 · v1.0

---

Before you read

I wrote this report because the audio AI market has become almost impossible to navigate from the outside. Vendor pages compare their own best model to everyone else's worst, benchmark tables measure English read speech and call it "multilingual," and license terms are buried three clicks away from the download button. What follows is my attempt to put the whole field on one page: eight functional categories, the models that matter in each, what they actually cost, and where the legal tripwires are.

A few notes on method. The comparison is organized by function rather than architecture, because that is how build-versus-buy decisions actually get made. Prices are list prices as of November 2026, converted to US dollars, and they move constantly — treat every number here as a starting point for a quote, not as a quote. Where I could verify a figure against a vendor page I did; where I could not, the number is an estimate and should be checked before it goes into a budget. Model versions churn monthly, so a name in this report may already have a successor by the time you read it.

I have tried to be honest about the parts that are genuinely uncertain: latency figures depend on region and payload, WER depends on domain, and "supports 99 languages" usually means "supports 99 languages badly, except English." Where the data is thin, I say so.

---

1. The short version

1. There are eight functional categories: text-to-speech (TTS), speech-to-text (ASR), voice conversion and cloning, music generation, sound-effect generation, singing synthesis, audio understanding, and audio processing/restoration.
2. The real dividing line is cloud versus open weights. Cloud gives you quality, uptime, and someone else's on-call rotation; open weights give you control and a fixed GPU bill, at the cost of maintenance and uneven quality.
3. TTS has stratified into three tiers: high-expression models (ElevenLabs v3, Hume Octave), low-latency agent models (Cartesia Sonic, Deepgram Aura-2, ElevenLabs Flash), and small open models (Kokoro, Piper, Chatterbox, CosyVoice).
4. Whisper remains the default "good and cheap" ASR. On English accuracy it has been overtaken by Parakeet/Canary, Nova-3, and AssemblyAI — but for Persian, Whisper plus Azure or Google is still the realistic stack.
5. Music splits into consumer products (Suno, Udio, Riffusion) and open models (ACE-Step, Stable Audio Open, MusicGen, YuE). Full-song vocals remain a commercial advantage.
6. Sound effects grew fastest: text-to-SFX is now routine, and video-to-Foley (MMAudio, HunyuanVideo-Foley) is close behind.
7. Audio understanding has moved to "Omni" models — GPT-4o/5 audio, Gemini, Qwen3-Omni, Kimi-Audio. These are no longer transcription tools; they are voice assistants with ears.
8. Cost should be measured per unit of work: TTS per million characters, ASR per minute, music per generation, self-hosting per GPU-hour. Rule of thumb: below roughly 500 hours of audio per month, APIs win; above it, dedicated GPUs do.
9. Licensing is the main trap in open weights: CC-BY-NC, community licenses, and "open code, closed weights." Always check the license on the weights, not the GitHub repo.
10. Legal risk is the most serious issue in the field: voice cloning without consent, and ownership of generated music. For a commercial product, the safe path is a model with an explicit commercial license plus a supplier contract.

---

2. The map

Two secondary axes are useful. By signal domain: speech, music, and general/environmental sound. By model direction: generative (TTS, music, SFX) versus analytical (ASR, tagging, speaker recognition). Omni models do both.

---

3. How I compared them

Every model was assessed against the same twelve criteria, so the tables are at least internally consistent.

---

4. Text-to-speech

4.1 The state of the market

Three generations of TTS are running in production at the same time:

• Generation 1 (concatenative and parametric): Tacotron, FastSpeech, VITS. Fast, small, medium quality, still common in embedded products and low-resource languages.
• Generation 2 (diffusion and LLM-based): XTTS, F5-TTS, CosyVoice, Chatterbox, Orpheus, Dia. Zero-shot cloning and high expressiveness.
• Generation 3 (full duplex): CSM, Moshi, GPT-4o audio, Gemini. TTS as one component of a conversational model that understands context.

4.2 Commercial services

A note on price. Measure TTS by the cost of one hour of audio, not by the headline per-character rate. One hour of speech is roughly 90,000 characters. So $15 per 1M characters is about $1.35 per audio hour; $30 per 1M is $2.70; ElevenLabs on a low-tier plan can reach $10–20 per hour.

4.3 Open-weight models

Bottom line on TTS. Best absolute quality: ElevenLabs v3, Hume Octave, MiniMax in commercial; Higgs Audio v2, IndexTTS-2, Chatterbox in open. Best quality per dollar in open: Kokoro (no cloning) and CosyVoice (with cloning). Best for real-time agents: Cartesia, Deepgram Aura-2, ElevenLabs Flash; on the open side, CosyVoice-2 and Orpheus. Best for Persian: Azure Neural/Custom Neural, Google Chirp 3, ElevenLabs (test it), Polly; open options are Piper fa_IR and community VITS/Coqui models trained on ManaTTS.

---

5. Speech-to-text

5.1 Commercial

5.2 Open weights

Bottom line on ASR. Best accuracy: Nova-3, Universal-3, Scribe v2, Canary-Qwen-2.5B. Cheapest cloud: Soniox, Gladia, Speechmatics batch. Best open self-host: Whisper large-v3-turbo for breadth, Parakeet for speed on European languages, Canary for accuracy. Persian: Whisper large-v3 (WER ~10–15% depending on domain), Azure fa-IR, Google fa-IR, Speechmatics, Vosk fa for lightweight use, and NVIDIA NeMo Persian if you have training data.

---

6. Voice conversion and cloning

What matters here. Cloning is a legal risk, not just a technical one. Under the EU AI Act and several US state laws, disclosing synthetic voice and obtaining consent from the voice owner is mandatory in many contexts. Take watermarking (e.g., Perth in Chatterbox) and consent logging seriously. RVC is the best cost/quality route to a custom voice; Seed-VC is the zero-shot option. For singing, RVC and so-vits-svc remain the community standard, while full-song generation is easier through Suno or ACE-Step.

---

7. Music generation

7.1 Commercial and consumer

7.2 Open weights

Bottom line on music. For release-ready songs, Suno and Udio. For legally safe commercial background music, Firefly, Stable, or ElevenLabs Music. For local deployment and control, ACE-Step (best open) and DiffRhythm (fastest).

---

8. Sound effects and Foley

What matters here. For video, the metric is temporal sync, not just audio quality. Video-to-audio models (MMAudio, HunyuanVideo-Foley) beat text-to-audio on that metric.

---

9. Singing synthesis

What matters here. Singing from scratch (lyrics + notes) and voice conversion (a real singer → another timbre) are different problems. For studio-grade results Synthesizer V still wins; for speed and cost, Suno.

---

10. Audio understanding (Audio LLMs)

Where these get used: voice assistants, call analytics, factory monitoring, music description, accessibility. The benchmarks that matter are MMAU, AudioBench, and LongAudioBench.

---

11. Processing, separation, and restoration

---

12. The cost question

12.1 Cost per unit of work (approximate, November 2026)

One hour of speech is about 90,000 characters; one hour of audio is 60 minutes. Do not mix the two.

12.2 Self-hosting TCO

The key sentence. Whisper large-v3-turbo on a single L4 transcribes at roughly 30–60× real time — so one GPU-hour equals 30–60 hours of audio, or about $0.01–0.04 per audio hour. At scale (above roughly 500 hours per month) that undercuts any cheap API. What it does not include is engineering time and operational risk.

Break-even. If you produce or transcribe more than 300–500 hours of audio per month and have engineering capacity, self-hosting deserves serious consideration. Below that, APIs are cheaper and far less painful.

12.3 Hidden costs

• Egress and bandwidth on bulk audio delivery.
• Retries and quality: TTS error rates (mispronunciation, truncation) cause regeneration and double billing.
• Version lock: cloud models change without notice; yesterday's output is not today's. Pin versions or cache outputs.
• Data governance: voice is biometric data. In some sectors (health, banking), sending it to the cloud is prohibited, which forces self-hosting.

---

13. Licensing: a decision table

The golden rule: the license on the repository is not the license on the weights. F5-TTS has MIT code and CC-BY-NC weights; Fish Speech has open code and non-commercial weights. Read the LICENSE file attached to the weights on Hugging Face.

---

14. Choosing by scenario

---

15. The Persian situation

Persian ASR. Whisper large-v3 works on Persian (WER around 10–15% on clean speech, higher in casual or noisy audio). Common failure modes: numbers, the zero-width non-joiner, Arabic versus Persian letter forms, and proper nouns. Azure fa-IR and Google fa-IR have official support and are solid on formal or news-style speech. Speechmatics supports Persian; Vosk fa is light but inaccurate; MMS covers it with lower quality. For high accuracy on conversational Persian, fine-tuning Whisper on labeled in-domain data is the most reliable path.

Persian TTS. Azure and Google are the stable official options. ElevenLabs covers Persian in its multilingual models, but pronunciation quality needs testing on your own vocabulary — numbers, idioms, names. On the open side: Piper fa_IR (fast, light, robotic) and community VITS/Coqui models trained on ManaTTS and Common Voice fa. The quality ceiling depends on data and training time.

The real gap. No open multilingual model at the current state of the art covers Persian at English-level quality. For a Persian product, a fine-tuned open model plus a pronunciation dictionary is more realistic than expecting a general model to handle it out of the box.

Engineering notes for Persian: text normalization (digits, half-spaces, alef variants), a pronunciation lexicon for proper nouns, and evaluation on an internal test set rather than public MOS numbers.

---

16. Benchmarks worth looking at

A warning. Public benchmarks are not your domain. Always build a small test set from your own data (50–200 samples) and compare models on it. That is the only metric you can actually trust.

---

17. Risks and legal considerations

1. Voice cloning requires explicit consent, consent logging, and disclosure that the audio is synthetic. The EU AI Act and several US states (Tennessee, California) impose transparency requirements.
2. Music copyright. Suno and Udio reached agreements with publishers in 2025, but open models trained on protected data still carry risk. For commercial products, choose models with transparent data licensing.
3. Biometric data. Voice is sensitive personal data under GDPR and CCPA; storing or transferring it requires a DPIA.
4. Vendor lock-in. Dependence on one API means exposure to price and version changes. Keep an abstraction layer and a swap path.
5. Safety and hallucination. Audio models can generate harmful speech; output filtering and logging are necessary.
6. Low-resource languages. Quality drops sharply versus English. Manage expectations accordingly.

---

18. A practical evaluation checklist

• ☐ 50–200 real samples from your own data (including noise, accents, terminology).
• ☐ Numeric success criteria: target WER, MOS, TTFA, pronunciation error rate.
• ☐ Blind A/B testing between the three finalists.
• ☐ Cost per unit of work including retry rates.
• ☐ Weight license review for commercial use.
• ☐ Stability test: same input, ten runs, measure variance.
• ☐ Streaming/latency test on real networks, not just benchmarks.
• ☐ An exit plan: export format and model replacement path.

---

19. Sources

Leaderboards and benchmarks:

• Artificial Analysis — Text to Speech Leaderboard
• Hugging Face — Open ASR Leaderboard
• Hugging Face — TTS Arena

Pricing and comparison (for verification):

• Deepgram Pricing
• ElevenLabs Pricing
• OpenAI API Pricing
• AssemblyAI Pricing
• Google Cloud TTS Pricing
• Azure Speech Pricing
• AWS Polly Pricing
• Suno Pricing
• Future AGI — TTS comparison 2026
• Future AGI — STT comparison 2026

Open models (selection):

• [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M)
• Chatterbox
• CosyVoice
• F5-TTS
• ACE-Step
• Demucs
• Qwen3-Omni
• Whisper

---

Appendix: ten quick picks