Skip to content
archdiff

models / compare / nemotron-3-ultra-vs-deepseek-v4-1-flash

Nemotron 3 Ultra vs DeepSeek-V4.1-Flash

How Nemotron 3 Ultra (NVIDIA, June 2026) and DeepSeek-V4.1-Flash (DeepSeek, September 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 12 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ nemotron-3-ultra@2026-06+0~4−7

Nemotron 3 Ultra

NVIDIA, June 2026, 550B total, 55B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream, one sublayer per layer — Residual & normSingle residual stream, one sublayer p…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneMamba-2 selective state-space layers — AttentionMamba-2 selective state-spac…AttentionGlobal GQA attention without positional encoding — AttentionGlobal GQA attention without…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingLatentMoE routed experts (ReLU², 2,048-dim latent) — Mixture of expertsLatentMoE routed experts (ReLU², 2,048…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ deepseek-v4-1-flash@2026-09+0~3−1

DeepSeek-V4.1-Flash

DeepSeek, September 2026, 552B + 196B Engram total, 16B (8B in prefill) active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersCED decoder attention (CSA2 over encoder-projected KV) — AttentionCED decoder attention (CSA2…AttentionCSA2 encoder layers (2× compression, Full + Reuse) — AttentionCSA2 encoder layers (2× comp…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsDSpark drafter (trained after pretraining) — Input & outputDSpark drafter (trained after pretrain…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

Component by component

Nemotron 3 Ultra and DeepSeek-V4.1-Flash differ on 12 of 27 components.

Blueprint componentNemotron 3 UltraDeepSeek-V4.1-Flash

Not used

Same as the blueprintDeepSeek-ViT, trained from scratch: 32 layers, width 1,024, 16 heads, 14-px patches embedded by a linear projection, 2D-RoPE, RMSNorm and SwiGLU. A 3×3 pixel-unshuffle cuts visual tokens 9× before a 2-layer MLP projector maps them to 5,120 dims. Images only, up to about 1344×1344 px.

Not used

Same as the blueprintTwo Engram modules (196B parameters, FP8) at layers 1 and 14: n-gram orders 2–4, 8 hash heads per order, 2,048 dims per order, about 16M rows per head table. A sigmoid gate on each mHC stream's normalized dot product with its own key adds a shared value. The short causal conv was dropped. Rows are prefetched from host memory over RDMA.

Single residual stream, one sublayer per layer108 pre-norm layers on one d = 8,192 residual stream; each layer is one mixer or one MoE, not an attention+FFN pair.

Same as the blueprintSingle-Pass mHC: n = 4 streams of width 5,120 and 20 Sinkhorn iterations as in V4, but each sublayer reads its input with the mixing coefficients predicted by the previous sublayer. Removing that dependency lets the Mega-mHC kernel read and write the residual once, halving activation traffic, with negligible quality loss reported.

Not used

Same as the blueprintRMSNorm on the single 512-dim KV entry, which the report uses to bound every FP4 KV channel to about √512 ≈ 22.6, and on the 1,280-dim query latent. Unlike V4's reference code, V4.1's applies no separate per-head RMS normalization to queries.

Mamba-2 selective state-space layers48 of the 60 mixer layers: SSD with 256 heads × 64 dims, state size 128, 8 B/C groups, causal conv of kernel 4.

Not used

Global GQA attention without positional encoding12 layers of dense causal attention: 64 query heads share 2 KV heads (d_h = 128); no RoPE, QK-norm, gate or sink.

CED decoder attention (CSA2 over encoder-projected KV)Layers 20–39 take uncompressed global KV from the encoder's last hidden state and re-select with a hierarchical indexer.

Not used

CSA2 encoder layers (2× compression, Full + Reuse)Layers 2–19: KV pooled 2→1 and computed in only 3 of 18 layers; the rest reuse that KV and its top-512 selection.

Not used

Same as the blueprintLayers 0–1 are pure sliding-window layers; every other layer adds the 128 most recent uncompressed tokens (SWA KV kept in FP8) to its sparse selection. SWA KV is never persisted: on a prefix-cache hit it is rebuilt by replaying only the last 128 tokens (SWA Bounded Replay), with negligible measured loss.

Not used

Same as the blueprintNo output gate. Each of the 64 heads has a learnable sink logit in the softmax denominator, per the reference inference code.

LatentMoE routed experts (ReLU², 2,048-dim latent)512 non-gated squared-ReLU experts of width 5,120 run in a 2,048-dim latent space; 22 active per token.

Same as the blueprint384 SwiGLU experts of intermediate size 2,304, 6 active per token, in all 40 layers, with V4's clamp (linear branch ±10, gate ≤ 10). The release stores them in FP4.

Same as the blueprintOne MTP block (an attention layer followed by a LatentMoE layer) whose weights are shared across steps and applied recursively. It ran 2 steps in pretraining (total loss weight 0.1, 0.05 per step). MTP Boosting then trains only this head with a forward-KL loss to the frozen backbone's logits over 7 steps, to match the inputs it sees when drafting.

DSpark drafter (trained after pretraining)Three 128-token-window blocks draft 5 tokens per pass from the backbone's last 3 layers; no MTP loss in pretraining.

Not used

Same as the blueprintMuon (momentum 0.95, weight decay 0.1, update RMS 0.18) for backbone linear weights, Engram projections and the vision projector, applied per head to query and key weights. Embeddings, the head and Engram tables use a momentum update with Sinkhorn row/column balancing (Engram learning rate 5×); AdamW covers norms and other non-matrix parameters.

More comparisons