Skip to content
archdiff

models / compare / minimax-m2-vs-nemotron-3-ultra

MiniMax-M2.7 vs Nemotron 3 Ultra

How MiniMax-M2.7 (MiniMax, March 2026) and Nemotron 3 Ultra (NVIDIA, June 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 9 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ minimax-m2@2026-03+0~4−7

MiniMax-M2.7

MiniMax, March 2026, 229.9B total, 9.8B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normLayer-wide QK RMSNorm — Residual & normLayer-wide QK RMSNormResidual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in all 62 layers — AttentionFull grouped-query attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 block-scaled checkpoint — Training & inferenceFP8 block-scaled checkpointTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ nemotron-3-ultra@2026-06+0~4−7

Nemotron 3 Ultra

NVIDIA, June 2026, 550B total, 55B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream, one sublayer per layer — Residual & normSingle residual stream, one sublayer p…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneMamba-2 selective state-space layers — AttentionMamba-2 selective state-spac…AttentionGlobal GQA attention without positional encoding — AttentionGlobal GQA attention without…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingLatentMoE routed experts (ReLU², 2,048-dim latent) — Mixture of expertsLatentMoE routed experts (ReLU², 2,048…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

Component by component

MiniMax-M2.7 and Nemotron 3 Ultra differ on 9 of 27 components.

Blueprint componentMiniMax-M2.7Nemotron 3 Ultra

Single residual streamOne d = 3,072 residual stream with pre-norm sublayers; no hyper-connections.

Single residual stream, one sublayer per layer108 pre-norm layers on one d = 8,192 residual stream; each layer is one mixer or one MoE, not an attention+FFN pair.

Layer-wide QK RMSNormOne RMSNorm over all query heads (6,144 dims) and one over all key heads (1,024 dims), applied before RoPE.

Not used

Not used

Mamba-2 selective state-space layers48 of the 60 mixer layers: SSD with 256 heads × 64 dims, state size 128, 8 B/C groups, causal conv of kernel 4.

Full grouped-query attention in all 62 layersCausal softmax over the whole context in every layer: 48 query heads share 8 KV heads of dim 128; no window, sink or gate.

Global GQA attention without positional encoding12 layers of dense causal attention: 64 query heads share 2 KV heads (d_h = 128); no RoPE, QK-norm, gate or sink.

Same as the blueprint256 SwiGLU experts of intermediate size 1,536, top-8 active, in all 62 layers (no dense layers, no shared expert). In MiniMax's 17.8B-A2B ablation, 128 experts with top-8 beat 32 with top-2 at equal active parameters (MATH 19.6 → 24.1, HumanEval 29.7 → 32.5).

LatentMoE routed experts (ReLU², 2,048-dim latent)512 non-gated squared-ReLU experts of width 5,120 run in a 2,048-dim latent space; 22 active per token.

Not used

Same as the blueprintOne shared expert per MoE layer at full width: a non-gated squared-ReLU MLP, 8,192 → 10,240 → 8,192, added to the routed output without a router weight. The released checkpoint stores it in FP8 (per-tensor), not NVFP4.

Not disclosed

Not used

FP8 block-scaled checkpointOpen weights ship as FP8 E4M3 with 128 × 128 block scales; router, its bias, embeddings and LM head stay in higher precision.

Same as the blueprintPretrained on 20T tokens in NVFP4 (E2M1 with 16-element blocks, 2D weight scaling, Random Hadamard Transforms on wgrad inputs, stochastic rounding of gradients). The final 16 layers, Mamba output projections, latent projections, QKV/attention projections, MTP layers and embeddings stayed in higher precision. The train-loss gap to BF16 branches stayed below 0.4% on average.

Not disclosed

Same as the blueprintMulti-teacher on-policy distillation (MOPD) in two iterations. More than ten domain-specialized teachers (STEM, coding, SWE, search, terminal use, office work, chat, safety and others) give dense token-level reverse-KL signals on the student's own rollouts. The RLVR student acts as a self-teacher for uncovered domains.

More comparisons