Skip to content
archdiff

models / compare / minimax-m3-vs-nemotron-3-ultra

MiniMax-M3 vs Nemotron 3 Ultra

How MiniMax-M3 (MiniMax, June 2026) and Nemotron 3 Ultra (NVIDIA, June 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 11 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ minimax-m3@2026-06+0~4−4

MiniMax-M3

MiniMax, June 2026, 427.0B total, 23.5B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in blocks 0–2 — AttentionFull grouped-query attention…AttentionMiniMax Sparse Attention (MSA) — AttentionMiniMax Sparse Attention (MS…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headBF16 weights plus an official MXFP8 checkpoint — Training & inferenceBF16 weights plus an official MXFP8 ch…Training & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ nemotron-3-ultra@2026-06+0~4−7

Nemotron 3 Ultra

NVIDIA, June 2026, 550B total, 55B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream, one sublayer per layer — Residual & normSingle residual stream, one sublayer p…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneMamba-2 selective state-space layers — AttentionMamba-2 selective state-spac…AttentionGlobal GQA attention without positional encoding — AttentionGlobal GQA attention without…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingLatentMoE routed experts (ReLU², 2,048-dim latent) — Mixture of expertsLatentMoE routed experts (ReLU², 2,048…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

Component by component

MiniMax-M3 and Nemotron 3 Ultra differ on 11 of 27 components.

Blueprint componentMiniMax-M3Nemotron 3 Ultra

Same as the blueprintA 32-layer CLIP-style ViT (width 1,280, 16 heads, GELU MLP of 5,120, LayerNorm; 631M parameters) embeds 14 × 14 patches with a Conv3d that spans 2 frames for video, using 3D (time, height, width) RoPE. A GELU MLP lifts each patch to 6,144 and a second fuses each 2 × 2 group into one token (234M projector parameters). MiniMax says M3 trained on mixed modalities from the first step and lists text, image and video inputs.

Not used

Single residual streamOne d = 6,144 residual stream with pre-norm sublayers; no hyper-connections.

Single residual stream, one sublayer per layer108 pre-norm layers on one d = 8,192 residual stream; each layer is one mixer or one MoE, not an attention+FFN pair.

Same as the blueprintPer-head Gemma-style RMSNorm (ε = 1e-6) on q and k before RoPE, with one 128-dim gain shared by all 64 query heads and one by the 4 key heads; the indexer's queries and key get their own. This replaces M2's layer-wide norm over the concatenated projections (qk_norm_type per_layer → per_head), so head-sharded tensor parallelism no longer needs a cross-rank reduction for it.

Not used

Not used

Mamba-2 selective state-space layers48 of the 60 mixer layers: SSD with 256 heads × 64 dims, state size 128, 8 B/C groups, causal conv of kernel 4.

Full grouped-query attention in blocks 0–2The first three blocks attend over the whole context: 64 query heads share 4 KV heads of dim 128; no indexer, gate or sink.

Global GQA attention without positional encoding12 layers of dense causal attention: 64 query heads share 2 KV heads (d_h = 128); no RoPE, QK-norm, gate or sink.

MiniMax Sparse Attention (MSA)Per GQA group, a 2-projection indexer picks the top 16 of 128-token KV blocks; exact softmax over ≤ 2,048 keys, in 57 of 60 layers.

Not used

Same as the blueprint128 experts of intermediate size 3,072, top-4 active, in blocks 3–59 (M2: 256 of width 1,536, top-8). Blocks 0–2 use a dense MLP of width 12,288 instead. Every FFN uses gpt-oss's clamped SwiGLU: gate clamped at 7, linear branch to ±7, output (x_lin + 1) · x_glu · σ(1.702 · x_glu). Routed experts hold 413.1B of the 427.0B parameters.

LatentMoE routed experts (ReLU², 2,048-dim latent)512 non-gated squared-ReLU experts of width 5,120 run in a 2,048-dim latent space; 22 active per token.

Not disclosed

Not used

BF16 weights plus an official MXFP8 checkpointBF16 release with an FP32 router; MiniMax-M3-MXFP8 stores the text stack's linears as E4M3 with one 8-bit scale per 32 weights.

Same as the blueprintPretrained on 20T tokens in NVFP4 (E2M1 with 16-element blocks, 2D weight scaling, Random Hadamard Transforms on wgrad inputs, stochastic rounding of gradients). The final 16 layers, Mamba output projections, latent projections, QKV/attention projections, MTP layers and embeddings stayed in higher precision. The train-loss gap to BF16 branches stayed below 0.4% on average.

Not disclosed

Same as the blueprintMulti-teacher on-policy distillation (MOPD) in two iterations. More than ten domain-specialized teachers (STEM, coding, SWE, search, terminal use, office work, chat, safety and others) give dense token-level reverse-KL signals on the student's own rollouts. The RLVR student acts as a self-teacher for uncovered domains.

Not disclosed

Same as the blueprintThe shared MTP head drafts recursively; the published serving recipes default to 5 draft tokens. On SPEED-Bench at draft length 7 with greedy decoding, the mean acceptance length is 4.39 tokens, and 4.58 after MTP Boosting.

More comparisons