Skip to content
archdiff

models / compare / deepseek-v4-vs-nemotron-3-ultra

DeepSeek-V4-Pro vs Nemotron 3 Ultra

How DeepSeek-V4-Pro (DeepSeek, April 2026) and Nemotron 3 Ultra (NVIDIA, June 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 10 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ deepseek-v4@2026-04+0~1−3

DeepSeek-V4-Pro

DeepSeek, April 2026, 1.6T total, 49B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersHeavily Compressed Attention (HCA) — AttentionHeavily Compressed Attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ nemotron-3-ultra@2026-06+0~4−7

Nemotron 3 Ultra

NVIDIA, June 2026, 550B total, 55B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream, one sublayer per layer — Residual & normSingle residual stream, one sublayer p…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneMamba-2 selective state-space layers — AttentionMamba-2 selective state-spac…AttentionGlobal GQA attention without positional encoding — AttentionGlobal GQA attention without…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingLatentMoE routed experts (ReLU², 2,048-dim latent) — Mixture of expertsLatentMoE routed experts (ReLU², 2,048…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

Component by component

DeepSeek-V4-Pro and Nemotron 3 Ultra differ on 10 of 27 components.

Blueprint componentDeepSeek-V4-ProNemotron 3 Ultra

Same as the blueprintmHC with n = 4 streams of width 7,168 around every attention and MoE sublayer. The residual-mixing matrix is projected to doubly stochastic with 20 Sinkhorn–Knopp iterations; read-in uses σ(·) and write-back 2σ(·). DeepSeek reports a 6.7% wall-time overhead per overlapped 1F1B pipeline stage.

Single residual stream, one sublayer per layer108 pre-norm layers on one d = 8,192 residual stream; each layer is one mixer or one MoE, not an attention+FFN pair.

Same as the blueprintAn RMS normalization of each of the 128 query heads and an RMSNorm on the single 512-dim KV entry, applied just before core attention in both CSA and HCA. DeepSeek credits this for training with Muon without QK-Clip.

Not used

Not used

Mamba-2 selective state-space layers48 of the 60 mixer layers: SSD with 256 heads × 64 dims, state size 128, 8 B/C groups, causal conv of kernel 4.

Heavily Compressed Attention (HCA)Every 128 tokens pooled into one KV entry; dense shared-KV attention over all of them plus a 128-token window.

Global GQA attention without positional encoding12 layers of dense causal attention: 64 query heads share 2 KV heads (d_h = 128); no RoPE, QK-norm, gate or sink.

Same as the blueprintCompressed Sparse Attention in 30 of 61 layers (even layers 2–60), interleaved with HCA. Each KV entry pools 8 overlapping tokens at stride m = 4 with a per-channel softmax. A lightning indexer (64 heads × 128 dims, ReLU scores weighted per head) keeps the top 1,024 entries, which the 128 query heads attend to together with the SWA entries.

Not used

Same as the blueprintV4-Pro has no pure sliding-window layers (V4-Flash makes its first two layers pure SWA). Every CSA and HCA layer adds 128 uncompressed KV entries for the most recent tokens to the same softmax, which covers the tokens in the query's own not-yet-compressed block.

Not used

Same as the blueprintNo output gate. Each of the 128 heads has a learnable sink logit whose exponential is added to the softmax denominator in both CSA and HCA, so its attention weights can sum to less than 1.

Not used

Same as the blueprint384 SwiGLU experts of intermediate size 3,072, 6 active per token, in all 61 layers. The linear branch is clamped to [−10, 10] and the gate capped at 10. Expert weights went through MXFP4 quantization-aware training in post-training and ship in FP4.

LatentMoE routed experts (ReLU², 2,048-dim latent)512 non-gated squared-ReLU experts of width 5,120 run in a 2,048-dim latent space; 22 active per token.

Same as the blueprintMuon for most matrices: 10 hybrid Newton–Schulz steps (8 with coefficients (3.4445, −4.7750, 2.0315), then 2 with (2, −1.5, 0.5)), Nesterov momentum 0.95, weight decay 0.1, update RMS rescaled to 0.18. AdamW (β = 0.9 / 0.95) trains the embedding, prediction head, RMSNorm gains and mHC static biases and gates. No QK-Clip.

Not used

Not disclosed

Same as the blueprintThe shared MTP head drafts recursively; the published serving recipes default to 5 draft tokens. On SPEED-Bench at draft length 7 with greedy decoding, the mean acceptance length is 4.39 tokens, and 4.58 after MTP Boosting.

More comparisons