Skip to content
archdiff

models / compare / deepseek-v4-1-flash-vs-deepseek-v4

DeepSeek-V4.1-Flash vs DeepSeek-V4-Pro

How DeepSeek-V4.1-Flash (DeepSeek, September 2026) and DeepSeek-V4-Pro (DeepSeek, April 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 6 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ deepseek-v4-1-flash@2026-09+0~3−1

DeepSeek-V4.1-Flash

DeepSeek, September 2026, 552B + 196B Engram total, 16B (8B in prefill) active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersCED decoder attention (CSA2 over encoder-projected KV) — AttentionCED decoder attention (CSA2…AttentionCSA2 encoder layers (2× compression, Full + Reuse) — AttentionCSA2 encoder layers (2× comp…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsDSpark drafter (trained after pretraining) — Input & outputDSpark drafter (trained after pretrain…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ deepseek-v4@2026-04+0~1−3

DeepSeek-V4-Pro

DeepSeek, April 2026, 1.6T total, 49B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersHeavily Compressed Attention (HCA) — AttentionHeavily Compressed Attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

DeepSeek-V4.1-Flash and DeepSeek-V4-Pro differ on 6 of 27 components.

Blueprint componentDeepSeek-V4.1-FlashDeepSeek-V4-Pro

Same as the blueprintDeepSeek-ViT, trained from scratch: 32 layers, width 1,024, 16 heads, 14-px patches embedded by a linear projection, 2D-RoPE, RMSNorm and SwiGLU. A 3×3 pixel-unshuffle cuts visual tokens 9× before a 2-layer MLP projector maps them to 5,120 dims. Images only, up to about 1344×1344 px.

Not used

Same as the blueprintTwo Engram modules (196B parameters, FP8) at layers 1 and 14: n-gram orders 2–4, 8 hash heads per order, 2,048 dims per order, about 16M rows per head table. A sigmoid gate on each mHC stream's normalized dot product with its own key adds a shared value. The short causal conv was dropped. Rows are prefetched from host memory over RDMA.

Not used

CED decoder attention (CSA2 over encoder-projected KV)Layers 20–39 take uncompressed global KV from the encoder's last hidden state and re-select with a hierarchical indexer.

Heavily Compressed Attention (HCA)Every 128 tokens pooled into one KV entry; dense shared-KV attention over all of them plus a 128-token window.

CSA2 encoder layers (2× compression, Full + Reuse)Layers 2–19: KV pooled 2→1 and computed in only 3 of 18 layers; the rest reuse that KV and its top-512 selection.

Same as the blueprintCompressed Sparse Attention in 30 of 61 layers (even layers 2–60), interleaved with HCA. Each KV entry pools 8 overlapping tokens at stride m = 4 with a per-channel softmax. A lightning indexer (64 heads × 128 dims, ReLU scores weighted per head) keeps the top 1,024 entries, which the 128 query heads attend to together with the SWA entries.

DSpark drafter (trained after pretraining)Three 128-token-window blocks draft 5 tokens per pass from the backbone's last 3 layers; no MTP loss in pretraining.

Same as the blueprintOne MTP module (depth 1), unchanged from DeepSeek-V3: an extra block that combines the normalized next-token embedding with the trunk's normalized state and shares the embedding and LM head. Its attention is sliding-window only. Loss weight 0.3, lowered to 0.1 when learning-rate decay starts.

Same as the blueprintServed with the DSpark drafter: 5 draft tokens per pass, with the verification length chosen per request from predicted acceptance and profiled throughput under current load. DSpark also accelerates RL and OPD rollouts.

Not disclosed

More comparisons