Higgs Audio V2 Project Details

Project Overview

Higgs Audio V2 is a powerful open-source audio foundation model from Boson AI, pre-trained on over 10 million hours of audio data and diverse text data. Despite no post-training or fine-tuning, Higgs Audio V2 excels in expressive audio generation, thanks to its deep understanding of language and acoustics.

Core Features

1. Powerful Speech Synthesis Capabilities

Supports expressive speech generation
Multi-language audio generation support
Natural multi-speaker dialogue generation
Automatic prosody adaptation for narration
Melodic humming with cloned voices
Simultaneous generation of speech and background music

2. Exceptional Performance

In the EmergentTTS-Eval benchmark, the model achieved win rates of 75.7% and 55.7% in the "Emotion" and "Question" categories, respectively, surpassing "gpt-4o-mini-tts". It also achieved industry-leading performance in traditional TTS benchmarks such as Seed-TTS Eval and the Emotional Speech Dataset (ESD).

3. Unique Emergent Capabilities

The model exhibits capabilities rarely seen in previous systems:

Generating natural multi-language, multi-speaker dialogues
Automatic prosody adaptation during narration
Melodic humming using cloned voices
Simultaneous generation of speech and background music

Technical Architecture

Core Technical Innovations

Automated Annotation Pipeline: Utilizes multiple ASR models, sound event classification models, and internal audio understanding models to clean and annotate 10 million hours of audio data (dubbed AudioVerse).
Unified Audio Tokenizer: A unified audio tokenizer trained from scratch, capable of capturing both semantic and acoustic features.
DualFFN Architecture: Enhances the LLM's ability to model acoustic tokens while maintaining minimal computational overhead.

Environment Setup

Recommended Docker Environment

#  NVIDIA Deep Learning Container
docker run --gpus all --ipc=host --net=host --ulimit memlock=-1 --ulimit stack=67108864 -it --rm nvcr.io/nvidia/pytorch:25.02-py3 bash

Standard Installation Methods

Using Git + pip

git clone https://github.com/boson-ai/higgs-audio.git
cd higgs-audio
pip install -r requirements.txt
pip install -e .

Using a Virtual Environment

git clone https://github.com/boson-ai/higgs-audio.git
cd higgs-audio
python3 -m venv higgs_audio_env
source higgs_audio_env/bin/activate
pip install -r requirements.txt
pip install -e .

Using Conda

git clone https://github.com/boson-ai/higgs-audio.git
cd higgs-audio
conda create -n higgs_audio_env python=3.10
conda activate higgs_audio_env
pip install -r requirements.txt
pip install -e .

Using uv

git clone https://github.com/boson-ai/higgs-audio.git
cd higgs-audio
uv venv --python 3.10
source .venv/bin/activate
uv pip install -r requirements.txt
uv pip install -e .

Usage Examples

Basic Python Code Example

from boson_multimodal.serve.serve_engine import HiggsAudioServeEngine, HiggsAudioResponse
from boson_multimodal.data_types import ChatMLSample, Message, AudioContent
import torch
import torchaudio
import time
import click

MODEL_PATH = "bosonai/higgs-audio-v2-generation-3B-base"
AUDIO_TOKENIZER_PATH = "bosonai/higgs-audio-v2-tokenizer"

system_prompt = (
"Generate audio following instruction.\n\n<|scene_desc_start|>\nAudio is recorded from a quiet room.\n<|scene_desc_end|>"
)

messages = [
    Message(
        role="system",
        content=system_prompt,
    ),
    Message(
        role="user",
        content="The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.",
    ),
]

device = "cuda" if torch.cuda.is_available() else "cpu"
serve_engine = HiggsAudioServeEngine(MODEL_PATH, AUDIO_TOKENIZER_PATH, device=device)

output: HiggsAudioResponse = serve_engine.generate(
    chat_ml_sample=ChatMLSample(messages=messages),
    max_new_tokens=1024,
    temperature=0.3,
    top_p=0.95,
    top_k=50,
    stop_strings=["<|end_of_text|>", "<|eot_id|>"],
)

torchaudio.save(f"output.wav", torch.from_numpy(output.audio)[None, :], output.sampling_rate)

Command Line Usage Examples

Generating Speech with Reference Audio

python3 examples/generation.py \
--transcript "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years." \
--ref_audio belinda \
--temperature 0.3 \
--out_path generation.wav

Generating Speech Without Reference Audio (Model Automatically Determines Voice)

python3 examples/generation.py \
--transcript "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years." \
--temperature 0.3 \
--out_path generation.wav

Generating Multi-Speaker Dialogue

python3 examples/generation.py \
--transcript examples/transcript/multi_speaker/en_argument.txt \
--seed 12345 \
--out_path generation.wav

Generating Multi-Speaker Dialogue with Specified Voices

python3 examples/generation.py \
--transcript examples/transcript/multi_speaker/en_argument.txt \
--ref_audio belinda,broom_salesman \
--ref_audio_in_system_message \
--chunk_method speaker \
--seed 12345 \
--out_path generation.wav

Performance Evaluation Results

Traditional TTS Benchmarks

SeedTTS-Eval and ESD Evaluation Results

Model	SeedTTS-Eval		ESD
	WER ↓	SIM ↑	WER ↓	SIM (emo2vec) ↑
Cosyvoice2	2.28	65.49	2.71	80.48
Qwen2.5-omni†	2.33	64.10	-	-
ElevenLabs Multilingual V2	1.43	50.00	1.66	65.87
Higgs Audio v1	2.18	66.27	1.49	82.84
Higgs Audio v2 (base)	2.44	67.70	1.78	86.13

EmergentTTS-Eval Benchmark

Model	Emotion (%) ↑	Question (%) ↑
Higgs Audio v2 (base)	75.71%	55.71%

Multi-Speaker Evaluation

The project also designed a dedicated multi-speaker evaluation benchmark, comprising three subsets:

two-speaker-conversation: 1000 synthetic dialogues involving two speakers
small talk (no ref): 250 synthetic dialogues characterized by short utterances and limited turns
small talk (ref): 250 similar synthetic dialogues containing even shorter utterances

Hardware Requirements

Important Note: For optimal performance, it is recommended to run generation examples on a GPU machine equipped with at least 24GB of VRAM.

Advanced Features

vLLM API Server

The project also provides an OpenAI-compatible API server based on the vLLM engine for high-throughput advanced usage. For details, please refer to the examples/vllm directory.

Custom Audio Tokenizer

The project introduces a new discretized audio tokenizer that operates at only 25 frames per second while maintaining or even improving audio quality compared to tokenizers with twice the bitrate. This model is the first unified system trained on 24 kHz data, covering speech, music, and sound events.

Project Significance

With the release of its open-source version, Higgs Audio V2 invites developers worldwide to participate in shaping the future of human-computer interaction. When you generate your first sample, you're not just hearing synthesized speech—you're experiencing the next evolutionary stage of voice technology.

Higgs Audio V2 represents a significant breakthrough in audio generation technology, providing developers and researchers with powerful tools to explore and innovate in audio AI applications.