Top 10 Open Source AI Audio Models in 2026

Open-source audio models now cover speech recognition, text-to-speech, real-time voice conversations, and music generation, and several of them run on a single consumer GPU. Qwen3-ASR transcribes 30 languages and 22 Chinese dialects, Kokoro generates speech with just 82 million parameters, and ACE-Step writes full songs on less than 4 GB of video memory (VRAM).

For this guide, I selected 10 of the best open-source and open-weight AI audio models by combining 3 factors: GitHub repositories with high star counts, widely downloaded models on Hugging Face, and top-performing models from Artificial Analysis benchmarks. I covered 4 categories: speech recognition, text-to-speech, real-time conversations, and music generation.

Each model includes a quick-start example based on its official documentation. Some examples need specific hardware, dependencies, or extra configuration, so check each model's requirements before you run them.

A quick note: This is a curated list, not a strict benchmark ranking. Open-weight means the trained weights are downloadable, but the license can still restrict how you use them, so I flag licenses in each section and in the comparison table at the end. The examples come from official documentation, but I have not personally tested the code, so your setup may need additional dependencies or configuration.

In a Nutshell

• Speech recognition: Qwen3-ASR for multilingual accuracy, Nemotron 3.5 ASR Streaming for low-latency live transcription, and Voxtral Small when you also want summaries and answers about the audio.
• Text-to-speech: Kokoro 82M for lightweight local use, Fish Audio Drama 3 for expressive multi-speaker dialogue, and Breeze TTS 2 for voice design (non-commercial use only).
• Real-time conversations: NVIDIA PersonaPlex for audio-only voice agents, and MiniCPM-o 4.5 when you also want video input.
• Music generation: ACE-Step 1.5 for full songs with vocals, and MusicGen Large for instrumentals (non-commercial use only).

Best Open-Source Speech Recognition Models

Speech recognition models, also called automatic speech recognition (ASR) models, convert spoken audio into text. They are useful for meeting transcription, subtitles, podcasts, voice assistants, and audio analysis. Modern models also recognize multiple languages, process live conversations, and some can reason about what is being discussed.

1. Voxtral Small

Voxtral Small 24B is an open-weight speech understanding model from Mistral AI, released under the Apache 2.0 license. It supports 8 languages (English, Spanish, French, Portuguese, Hindi, German, Dutch, and Italian) and can transcribe recordings, summarize conversations, answer questions about audio, and follow spoken instructions.

Source: Voxtral Small 24B

Its biggest advantage is that one model handles both speech recognition and language reasoning, so you don't need a separate transcription and summarization pipeline. That suits meeting analysis, interviews, and voice assistants. The catch is size: the 24-billion-parameter model needs approximately 55 GB of GPU memory in BF16 or FP16, which means high-memory or multi-GPU hardware.

If you only need transcription, Mistral's newer Voxtral Mini 4B Realtime is much lighter. Mistral released it in February 2026 under Apache 2.0, with streaming latency configurable down to sub-200 ms.

Quick start: Install vLLM and the required Python packages.

pip install -U "vllm[audio]" "mistral-common[audio]" openai huggingface_hubPowered by

Start the vLLM server. This configuration uses two GPUs.

vllm serve mistralai/Voxtral-Small-24B-2507 \
  --tokenizer_mode mistral \
  --config_format mistral \
  --load_format mistral \
  --tensor-parallel-size 2 \
  --tool-call-parser mistral \
  --enable-auto-tool-choicePowered by

Open another terminal and run the Python example to transcribe a sample audio recording.

from mistral_common.protocol.transcription.request import TranscriptionRequest
from mistral_common.protocol.instruct.messages import RawAudio
from mistral_common.audio import Audio
from huggingface_hub import hf_hub_download
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

model = client.models.list().data[0].id

audio_file = hf_hub_download(
    "patrickvonplaten/audio_samples",
    "obama.mp3",
    repo_type="dataset",
)

audio = RawAudio.from_audio(
    Audio.from_file(audio_file, strict=False)
)

req = TranscriptionRequest(
    model=model,
    audio=audio,
    language="en",
    temperature=0.0,
).to_openai(exclude=("top_p", "seed"))

response = client.audio.transcriptions.create(**req)
print(response.text)Powered by

The script downloads a sample recording, sends it to your local Voxtral server, and prints the transcription.

2. Qwen3-ASR

Qwen3-ASR is an open-source speech recognition model family from Alibaba's Qwen team, available in 0.6B and 1.7B parameter versions under the Apache 2.0 license. It supports 30 languages and 22 Chinese dialects, detects the spoken language automatically, and can transcribe speech over background music, including singing. It works well for multilingual transcription, subtitles, podcasts, and interviews.

Source: Qwen3-ASR

Qwen3-ASR balances accuracy, speed, and flexibility. The 1.7B model posts a mean word error rate (WER) of 5.76 on Hugging Face's Open ASR Leaderboard, compared with 6.62 for Voxtral Small (lower is better). The 0.6B version trades some accuracy for efficiency.

Both sizes support offline and streaming inference, although streaming currently requires the vLLM backend (pip install -U qwen-asr[vllm]). The optional Qwen3-ForcedAligner-0.6B model generates word-level timestamps for subtitles in 11 supported languages.

Quick start: Install the official Python package.

pip install -U qwen-asrPowered by

Load the 1.7B model and transcribe the official sample audio.

import torch
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "Qwen/Qwen3-ASR-1.7B",
    dtype=torch.bfloat16,
    device_map="cuda:0",
)

results = model.transcribe(
    audio="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav",
    language=None,
)

print(results[0].language)
print(results[0].text)Powered by

The model downloads automatically on first use. It detects the spoken language and prints the transcription.

You can replace the sample URL with your own audio file. This example uses an NVIDIA GPU.

3. NVIDIA Nemotron 3.5 ASR Streaming

NVIDIA Nemotron 3.5 ASR Streaming 0.6B is a lightweight speech recognition model built for real-time transcription. It has 600 million parameters and supports 40 language locales, including English, Arabic, Hindi, Spanish, and Japanese. Of these, 32 transcribe out of the box and 8 need fine-tuning first.

Source: NVIDIA Nemotron 3.5 ASR Streaming 0.6B

Its speed comes from a cache-aware streaming architecture. Instead of reprocessing an entire recording, it processes new audio chunks while reusing cached encoder context, with chunk sizes as small as 80 milliseconds. On a single H100, NVIDIA reports roughly 17x more concurrent streams than its buffered Parakeet RNNT 1.1B model at the 80 ms setting (240 versus 14).

That makes it a strong choice for live subtitles, meetings, and low-latency voice assistants. The weights use the OpenMDW-1.1 license, which NVIDIA's model card describes as ready for commercial use, and the model runs locally through NVIDIA's NeMo-Speech.cpp runtime.

Quick start: Install NeMo-Speech.cpp on Linux or macOS.

curl -fsSL https://github.com/NVIDIA/NeMo-Speech.cpp/raw/main/scripts/install.sh | shPowered by

Transcribe a WAV file. The CLI downloads the default Nemotron 3.5 model on first use.

nemo-speech transcribe audio.wavPowered by

To test streaming transcription, which feeds the file through the streaming recognizer in 160 ms chunks, run:

nemo-speech transcribe audio.wav --streamPowered by

For real-time microphone transcription on supported builds:

nemo-speech transcribe --livePowered by

Replace audio.wav with your own recording. The runtime supports CPU and compatible GPU backends, so you do not need a large GPU just to get started.

Best Open-Source Text-to-Speech Models

Text-to-speech (TTS) models turn written text into natural-sounding audio. Modern models can clone voices, express emotion, generate multi-speaker dialogue, and speak multiple languages. Here are 3 open-weight models worth exploring.

4. Fish Audio Drama 3

Fish Audio Drama 3 is an expressive text-to-speech model built for multi-character scenes and long-form content. It uses a mixture-of-experts architecture with 35 billion parameters, but activates only about 3 billion per token. It supports voice cloning, natural-language voice control, and dialogue with up to 16 speakers, which suits audiobooks and character-driven content.

Source: Fish Audio Drama 3 Technical Report

What stands out is the emotional range. You can place inline tags such as [sighs] and [laughs] directly in the text, assign up to 16 speaker tags in a single script, and stream long passages chunk by chunk. Fish Audio reports a time to first audio of about 89 ms for a single stream on an H200 GPU, and the model runs through SGLang-Omni and vLLM-Omni.

The catch is size and licensing. The FP8 checkpoint needs 37 GB for weights alone, and a 4-bit GPTQ version cuts that to 20.8 GB. The weights use the Fish Audio Community License, so read its terms before commercial use.

Quick start: Install SGLang-Omni and its Fish Audio dependencies on a supported Linux system with an NVIDIA GPU.

pip install -U uv huggingface_hub
git clone https://github.com/sgl-project/sglang-omni.git
cd sglang-omni
uv venv
source .venv/bin/activate
uv pip install --prerelease=allow -v -e ".[fishaudio]"Powered by

Download the model and start the server:

hf download fishaudio/drama-3

sgl-omni serve --model-path fishaudio/drama-3 --port 8000Powered by

Open another terminal and generate speech:

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Hello, how are you?"
  }' \
  --output output.wavPowered by

The model generates speech and saves it as output.wav. You can change the input text or explore voice cloning and multi-speaker generation using the official Drama 3 cookbook.

5. Breeze TTS 2

Breeze TTS 2 is a 3-billion-parameter text-to-speech model for English and Chinese. It supports voice cloning from a reference recording, voice design from a plain-language description, and natural-language instructions for tone, emotion, and delivery.

Source: Breeze TTS 2

BreezeBlue's model card says it ranks first among open-weight models on the Artificial Analysis TTS leaderboard, with under 40 ms time to first audio on an H100 using the fast path. It also supports inline vocal events such as (laugh), (sigh), and (cough).

That makes it a good fit for interactive voice applications, but its self-hosted weights and outputs are restricted to research and non-commercial use. Only the source code is Apache 2.0.

Quick start: On Linux with Python 3.10 or newer and a CUDA-compatible NVIDIA GPU, clone the official repository and install the dependencies.

git clone https://github.com/breezeblue-ai/breeze-tts.git
cd breeze-tts

python -m pip install -r requirements.txt
python -m pip install -U huggingface_hubPowered by

Download the model checkpoint:

hf download BreezeBlue/Breeze-TTS-2 \
  --local-dir ../breeze-tts-2Powered by

Generate speech using the official voice-design example. This does not require reference audio.

python infer.py ../breeze-tts-2 \
  --text "(sigh) Welcome aboard. Your journey begins now." \
  --instruction "A warm, thoughtful young woman with a clear voice and a calm, reflective delivery." \
  --cfg-scale 4 \
  --output outputs/voice_design_en.wavPowered by

The model generates speech using the voice description and saves it as outputs/voice_design_en.wav. The default inference mode needs approximately 7.7 GiB of GPU memory (14.4 GiB with the --fast-all option), and a 12 GB GPU is the minimum recommended. You can also try voice cloning and streaming through the official Breeze TTS 2 documentation.

6. Kokoro 82M

Kokoro 82M is a lightweight text-to-speech model with just 82 million parameters. Its v1.0 release offers 54 voices across 8 languages. It is one of the more accessible options for developers who want to experiment with speech generation without a high-end GPU.

Source: Kokoro 82M

Kokoro's main advantage is efficiency. It suits offline narration, accessibility tools, and voice assistants, and its Apache 2.0 weights mean you can use it in commercial projects.

Quick start: You can run Kokoro directly in Google Colab. First, install the required packages:

!pip install -q "kokoro>=0.9.2" soundfile

!apt-get -qq -y install espeak-ng > /dev/null 2>&1Powered by

Generate speech using the official Python interface:

from kokoro import KPipeline
from IPython.display import display, Audio
import soundfile as sf

pipeline = KPipeline(lang_code='a')

text = "Hello! Welcome to the world of open-source text-to-speech models."

generator = pipeline(text, voice='af_heart')

for i, (gs, ps, audio) in enumerate(generator):
    display(Audio(data=audio, rate=24000))
    sf.write(f'{i}.wav', audio, 24000)Powered by

The model generates speech using the af_heart American English voice, plays the audio in Colab, and saves it as a WAV file. You can change the text, voice, or language using the official Kokoro model card.

Best Open-Source Real-Time Audio LLMs

Real-time audio large language models (LLMs) listen to spoken questions and generate natural voice responses. Instead of chaining separate speech recognition and text-to-speech systems, they handle continuous conversations, interruptions, and simultaneous listening and speaking in one model.

7. MiniCPM-o 4.5

MiniCPM-o 4.5 is a 9-billion-parameter multimodal model from OpenBMB that understands images, video, audio, and text and responds with speech. It supports real-time voice conversations in English and Chinese, voice cloning, and full-duplex interaction. Full-duplex means the model can listen, watch, and speak at the same time instead of waiting for each turn to finish.

Source: MiniCPM-o 4.5

Its distinguishing feature is the combination of visual understanding and voice interaction. You can build assistants that answer spoken questions about a live camera feed, describe their surroundings, or respond to conversations while processing video. The weights are Apache 2.0 licensed, and the developers provide a ready-to-use web demo, although full-duplex streaming is still experimental.

Quick start: The official MiniCPM-o 4.5 web demo supports live audio and video conversations. You need Linux, Docker Compose, NVIDIA Container Toolkit, and an NVIDIA GPU with more than 28 GB of VRAM.

First, clone the official demo repository:

git clone https://github.com/OpenBMB/MiniCPM-o-Demo.git
cd MiniCPM-o-DemoPowered by

Download the model checkpoint:

hf download openbmb/MiniCPM-o-4_5 \
  --local-dir ./MiniCPM-o-4_5Powered by

Generate the local HTTPS certificate:

mkdir -p certs data
openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout certs/key.pem -out certs/cert.pem -subj "/CN=minicpm-o"Powered by

Start the web demo using Docker Compose:

MODEL_HOST_PATH=/path/to/MiniCPM-o-4_5 docker compose up -d --build
docker compose logs -f gateway
docker compose logs -f worker-backend-0Powered by

Replace /path/to/MiniCPM-o-4_5 with the absolute path to the downloaded checkpoint, and edit docker-compose.yml to match your GPU count. Open the HTTPS web interface on port 8006 and allow microphone and camera access.

Select Audio Full-Duplex to have a live voice conversation, or Omnimodal Full-Duplex to interact using both video and audio. You can ask questions, interrupt the model, and experiment with continuous conversations.

8. NVIDIA PersonaPlex

NVIDIA PersonaPlex is a 7-billion-parameter speech-to-speech model designed for natural, real-time conversations. Built on the Moshi architecture, it supports full-duplex communication, so it can listen and speak simultaneously. It handles interruptions and overlapping speech without separate transcription and speech generation systems.

Source: NVIDIA PersonaPlex

PersonaPlex lets you control the assistant's voice and personality, so developers can customize its conversational role and speaking style for customer service assistants, virtual tutors, and other voice agents. It focuses on audio conversations and has no visual input, so choose MiniCPM-o if you need video. The weights are released under the NVIDIA Open Model License.

Quick start: Clone the official repository and install the dependencies on a supported Linux system with an NVIDIA GPU.

git clone https://github.com/NVIDIA/personaplex.git
cd personaplex

sudo apt install libopus-dev
pip install moshi/.Powered by

Accept the model license on Hugging Face and configure your access token:

export HF_TOKEN="YOUR_HUGGINGFACE_TOKEN"Powered by

Start the official PersonaPlex server:

SSL_DIR=$(mktemp -d); python -m moshi.server --ssl "$SSL_DIR"Powered by

If your GPU has limited memory, add the --cpu-offload flag, which requires the accelerate package.

Open https://localhost:8998 in your browser and allow microphone access. You can now speak directly with PersonaPlex, ask questions, interrupt its responses, and test different voice personalities.

Best Open-Source Audio and Music Generation Models

Audio generation models create original music, instrumentals, and melodies from text prompts. Some can also generate complete songs with vocals, edit existing recordings, or produce music based on a reference melody.

9. ACE-Step 1.5

ACE-Step 1.5 is an open-source music generation model that creates complete songs from text descriptions. It supports vocals, custom lyrics, multiple musical styles, and songs up to 10 minutes long. You can also control tempo, musical key, and duration, or use existing recordings to create covers and edit sections of a song.

Source: ACE-Step 1.5

ACE-Step stands out for how accessible it is compared with larger music generation models. It runs on NVIDIA, AMD, Intel, and Apple Silicon hardware, and its technical report says it can run on GPUs with less than 4 GB of VRAM. The project's README claims under 2 seconds per full song on an A100 and under 10 seconds on an RTX 3090.

It includes a Gradio web interface and a REST API. The XL variants, which use a 4B-parameter DiT decoder, trade memory for audio quality and need at least 12 GB of VRAM with offloading or 20 GB without. The project is MIT licensed.

Quick start: Install the uv package manager on Linux or macOS:

curl -LsSf https://astral.sh/uv/install.sh | shPowered by

Clone the official repository and install the dependencies:

git clone https://github.com/ACE-Step/ACE-Step-1.5.git
cd ACE-Step-1.5

uv syncPowered by

Launch the music generation interface:

uv run acestepPowered by

Open http://localhost:7860 in your browser. The application downloads the required model files on first launch and automatically selects a configuration for your hardware.

To generate music, enter a description of your desired song, such as an upbeat electronic track with energetic drums and synthesizers. You can also provide custom lyrics, adjust the duration, or select an instrumental track. Click the generation button to create your music, listen to the result, and download the audio file.

10. Meta MusicGen Large

MusicGen Large is a 3.3-billion-parameter text-to-music model developed by Meta as part of its AudioCraft project. It generates instrumental music from text descriptions covering genres, moods, instruments, and musical styles. It is the largest text-to-music checkpoint in the original MusicGen release.

Source: MusicGen Large

MusicGen Large produces detailed background music, soundtracks, and instrumental clips from a few words, and it works with Hugging Face Transformers and AudioCraft. It is also the oldest model on this list. Meta trained it in April and May 2023 on 20,000 hours of licensed music, and it cannot generate realistic vocals.

A GPU is recommended. The pretrained weights use the CC BY-NC 4.0 license, which restricts commercial use, while the code is MIT licensed.

Quick start: Install Transformers and SciPy.

pip install --upgrade transformers scipyPowered by

Use the official Hugging Face example to generate music from a text prompt:

from transformers import pipeline
import scipy

synthesiser = pipeline("text-to-audio", "facebook/musicgen-large")

music = synthesiser(
    "lo-fi music with a soothing melody",
    forward_params={"do_sample": True}
)

scipy.io.wavfile.write(
    "musicgen_out.wav",
    rate=music["sampling_rate"],
    data=music["audio"]
)Powered by

The model downloads automatically on first use, generates an instrumental track based on the prompt, and saves it as musicgen_out.wav. You can change the text to experiment with different music genres, instruments, and moods.

Which Open-Source AI Audio Model Should You Choose?

The best model depends on whether you want to transcribe speech, generate realistic voices, build a real-time assistant, or create music. This comparison covers capabilities, model size, and license for each use case.

Final Thoughts

Open-source audio models are improving quickly across speech recognition, text-to-speech, real-time conversation, and music generation. They are getting more accurate, faster, and easier to run, so developers can build audio applications without relying entirely on expensive commercial services.

Personally, I use Voxtral through Spokenly for almost all my transcription work, and I love it. The accuracy is impressive, and although the open-weight model is huge, I call Mistral's hosted API, which has been free for my usage. It is a good example of how these models are becoming practical for everyday tasks.

What excites me even more is the progress in conversational AI. We already know how natural voice conversations with ChatGPT can feel, and open-source models like NVIDIA PersonaPlex and MiniCPM-o 4.5 bring similar capabilities to developers who want to build their own assistants. Meanwhile, ACE-Step makes it possible to generate complete songs, and lightweight options like Kokoro make speech generation more accessible.

There is no single best model for everything, and that is what makes this space interesting. Whether you want accurate transcription, realistic voices, natural conversations, or generated music, there is probably an open-source model worth trying.