MiniMax-H3

1. Project Overview

MiniMax-H3 is an open-weight, omni-modal generation model that turns text, image, video, and audio inputs into coherent video output with native, synchronized stereo audio — solving the problem of having to stitch together separate video and audio generation pipelines into one unified model.

2. Background & Positioning

MiniMax-H3 (also referred to as Hailuo AI 3.0 in MiniMax's consumer products) was built to close the gap between silent, vision-only video generators and the kind of audio-visual content creators actually need — a single clip with lip-synced dialogue, sound effects, and music baked in from the start rather than dubbed on afterward.

Its core mission is to let one model read a mixed multimodal context — free-form text plus reference images, video clips, and audio tracks — and produce a temporally and acoustically coherent result, instead of requiring separate models (and separate prompts) for visuals and audio.

Compared with similar open-weight video generators, MiniMax-H3's main differentiator is native joint audio-video generation at up to 2K resolution with 32 kHz stereo sound, plus first/last-frame and multi-reference conditioning modes that most comparable open models don't offer together in one checkpoint.

3. Feature Categories

🎬 Generation Modes — 3 core modes, covering the main ways to drive the model

• Text-to-video-with-audio (T2VA): generate a full audio-visual clip from a text prompt alone
• First/last-frame conditioning (FL2VA): anchor a clip to a specified start and/or end frame
• Multi-reference generation (Ref2VA): combine up to 9 images, 3 video clips, and 3 audio tracks as references
• Purpose: cover everything from pure ideation to precise, reference-constrained production

🧩 Model Architecture — 3 pipeline stages, forming the end-to-end generation path

• H3-Context-IR: preprocesses and interprets multimodal inputs into a structured intermediate representation
• H3-Base: the 33B-parameter Transformer that generates 768p video and audio
• H3-Regenerate-2K: in-context regeneration stage that upscales output to 2K
• Purpose: separate "understanding," "generation," and "refinement" into composable stages

🛠️ Deployment & Tooling — 4 supported inference paths, for different scale and integration needs

• SGLang and vLLM for high-throughput, multi-GPU server deployment
• diffusers for Python-native experimentation and pipelines
• ComfyUI for node-based, no-code workflows
• Purpose: let teams pick the integration path that matches their existing stack

📚 Skills & Prompting Resources — 9 bundled workflow skills, for prompt engineering and downstream tooling

• h3-prompt-writing: a portable prompting skill compatible with Claude, OpenAI, and other agent platforms
• Reference guides for base (base-en.txt) and reference-conditioned (ref-en.txt) prompting
• Purpose: shorten the learning curve for writing prompts that reliably produce the intended shot, motion, and audio

4. Key Highlights

• Native synchronized audio — video and 32 kHz stereo audio are generated jointly by one model, not composited afterward, so dialogue, music, and sound effects stay in sync with the visuals.
• Up to 2K output — the base model generates at 768p and the H3-Regenerate-2K stage upscales in-context, avoiding a disconnected, separately-trained super-resolution step.
• Rich multimodal referencing — Ref2VA mode accepts up to 9 images, 3 video clips, and 3 audio tracks simultaneously, enabling complex, composited scene construction from a single call.
• Flexible aspect ratios and duration — supports 21:9, 16:9, 4:3, 1:1, 3:4, and 9:16 outputs from 4–15 seconds at 24 FPS, covering both cinematic and short-form vertical formats.
• Multilingual support — 11 languages with stable output quality, plus partial support for others, making it usable for global content pipelines rather than English-only workflows.
• Multiple deployment tiers — from single-call cloud APIs to self-hosted multi-GPU inference via SGLang/vLLM, so teams can start with the hosted API and migrate to self-hosting as volume grows.

5. Use Cases by Role

• General developers: integrate video-with-audio generation into apps via the cloud API (platform.minimax.io) without managing GPU infrastructure.
• DevOps/SRE: deploy and scale the open-weight checkpoints on self-managed GPU clusters using SGLang or vLLM for high-throughput inference.
• Data/research scientists: study or extend the H3-Omni-Transformer architecture, modality-specific VAEs, and 3D multimodal rotary position embeddings for multimodal generation research.
• Project managers: evaluate MiniMax-H3 as a build-vs-buy option for products needing synchronized audio-video generation, using the hosted API for fast prototyping before committing to self-hosted infrastructure.

6. Getting Started

Find what you need — browse the bundled skills and prompting guides in the skills/ directory, starting with skills/h3-prompt-writing/references/base-en.txt for prompt-writing fundamentals.

Install / integrate — download the model weights from Hugging Face and serve them locally:

hf download MiniMaxAI/MiniMax-H3 --include "model_index.json" "FL2VA/*" "Ref2VA/*" --local-dir MiniMax-H3
sglang serve --model-path MiniMaxAI/MiniMax-H3 --num-gpus 4 --ulysses-degree 4

Or call the hosted API directly at platform.minimax.io (global) without any local setup.

Contribute — open issues or discussions on the GitHub repository, or reach the maintainers at [email protected] for collaboration inquiries.

7. Project Structure (optional)

MiniMax-H3/
├── FL2VA/              # First/last-frame-to-video checkpoint
├── Ref2VA/             # Multi-reference-to-video checkpoint
├── audio_scheduler/    # Audio generation scheduling logic
├── audio_vae/          # Audio variational autoencoder
├── processor/          # Input preprocessing for multimodal context
├── scheduler/          # Diffusion/generation scheduler
├── text_encoder/       # Text encoding module
├── tokenizer/          # Tokenizer for text inputs
├── transformer/         # H3-Omni-Transformer core model
├── vae/                # Visual variational autoencoder
├── skills/              # Bundled prompting and workflow skills
└── model_index.json     # Model component manifest

The two checkpoint directories (FL2VA/, Ref2VA/) hold task-specific weights, while the shared transformer/, vae/, and audio_vae/ components define the core omni-modal architecture.

8. Related Ecosystem

• Inference frameworks: SGLang and vLLM for server-grade deployment; diffusers for Python pipelines; ComfyUI for visual workflow building.
• Hosted platforms: platform.minimax.io (global API), hailuoai.video (consumer web app), hub.minimax.io (desktop app).
• Model hub: weights are distributed via Hugging Face under MiniMaxAI/MiniMax-H3, in both original and diffusers-compatible formats.
• Agent ecosystems: the h3-prompt-writing skill is designed to be portable across Claude, OpenAI, and other agent platforms.

9. License

MiniMax-H3 is released under the MiniMax H3 Community License Agreement.

• ✅ Use the model for research, personal projects, and most commercial applications permitted under the community license.
• ❌ Do not use the model to generate unlawful, pornographic, or rights-infringing content — this is explicitly restricted by the license's moderation guardrails.
• ℹ️ Review the full license text in the repository before commercial deployment, since community licenses often include usage-scale or attribution clauses not summarized here.

10. FAQ

Q: Can I generate video with audio from text alone, without any reference media?
A: Yes — the text-to-video-with-audio (T2VA) mode generates a full clip with synchronized stereo audio from a text prompt only.

Q: What's the difference between FL2VA and Ref2VA checkpoints?
A: FL2VA conditions generation on a specified first and/or last frame, while Ref2VA accepts a broader mix of up to 9 images, 3 video clips, and 3 audio tracks as references.

Q: Do I need a local GPU to use MiniMax-H3?
A: No — you can call the hosted API at platform.minimax.io without local infrastructure, or self-host the open weights via SGLang/vLLM if you need scale or customization.

Q: What resolutions and durations are supported?
A: Up to 2K resolution (768p by default, upscaled via H3-Regenerate-2K) and 4–15 second clips at 24 FPS, across six aspect ratios.

Q: Is the model limited to English prompts?
A: No — it has stable support for 11 languages, with partial support for additional languages.

11. Quick Links

• Repository: github.com/MiniMax-AI/MiniMax-H3
• Model weights: huggingface.co/MiniMaxAI/MiniMax-H3
• Hosted API: platform.minimax.io
• Consumer app: hailuoai.video
• Contact: [email protected]

12. Summary

MiniMax-H3 gives developers and researchers an open-weight path to generating video with native, synchronized stereo audio from mixed text, image, video, and audio inputs — a capability that typically requires stitching together separate models. Teams that need audio-visual content generation, from quick prototypes via the hosted API to large-scale self-hosted pipelines via SGLang or vLLM, are the primary audience, while researchers gain a fully inspectable omni-modal Transformer architecture to study or extend.