Skip to content
archdiff

models / compare / deepseek-v4-1-flash-vs-gemma-4

DeepSeek-V4.1-Flash vs Gemma 4 31B

How DeepSeek-V4.1-Flash (DeepSeek, September 2026) and Gemma 4 31B (Google DeepMind, April 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 18 of 29 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ deepseek-v4-1-flash@2026-09+0~3−1

DeepSeek-V4.1-Flash

DeepSeek, September 2026, 552B + 196B Engram total, 16B (8B in prefill) active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersCED decoder attention (CSA2 over encoder-projected KV) — AttentionCED decoder attention (CSA2…AttentionCSA2 encoder layers (2× compression, Full + Reuse) — AttentionCSA2 encoder layers (2× comp…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsDSpark drafter (trained after pretraining) — Input & outputDSpark drafter (trained after pretrain…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ gemma-4@2026-04+2~7−6

Gemma 4 31B

Google DeepMind, April 2026, 30.7B

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioSentencePiece BPE with byte fallback — Input & outputSentencePiece BPE with byte fallbackInput & outputVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream with per-layer scalars — Residual & normSingle residual stream with per-layer…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kRoPE by layer type: full RoPE local, p-RoPE global — AttentionRoPE by layer type: full RoPE local, p…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with keys reused as values — AttentionGlobal GQA with keys reused…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sinkPost-attention RMSNorm (sandwich norm) — Residual & normPost-attention RMSNorm (sandwich norm)Residual & norm⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normDense GeGLU feed-forward — Mixture of expertsDense GeGLU feed-forwardMixture of expertsPost-feed-forward RMSNorm (sandwich norm) — Residual & normPost-feed-forward RMSNorm (sandwich no…Residual & normRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormTied LM head with logit soft-capping — Input & outputTied LM head with logit soft-cappingInput & outputAutoregressive MTP drafter (4 layers, reads trunk KV) — Input & outputAutoregressive MTP drafter (4 layers,…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

DeepSeek-V4.1-Flash and Gemma 4 31B differ on 18 of 29 components.

Blueprint componentDeepSeek-V4.1-FlashGemma 4 31B

Same as the blueprintThe same 129,280-row vocabulary as DeepSeek-V4, with an image placeholder token (id 129,264). Engram hashes its n-grams over a separate 99,092-entry compressed vocabulary.

SentencePiece BPE with byte fallbackGemini's 262,144-entry SentencePiece vocabulary: character-level BPE, split digits, bytes only as a fallback.

Same as the blueprintTwo Engram modules (196B parameters, FP8) at layers 1 and 14: n-gram orders 2–4, 8 hash heads per order, 2,048 dims per order, about 16M rows per head table. A sigmoid gate on each mHC stream's normalized dot product with its own key adds a shared value. The short causal conv was dropped. Rows are prefetched from host memory over RDMA.

Not used

Same as the blueprintSingle-Pass mHC: n = 4 streams of width 5,120 and 20 Sinkhorn iterations as in V4, but each sublayer reads its input with the mixing coefficients predicted by the previous sublayer. Removing that dependency lets the Mega-mHC kernel read and write the residual once, halving activation traffic, with negligible quality loss reported.

Single residual stream with per-layer scalarsOne d = 5,376 stream, RMSNorm before and after each sublayer, and a stored scalar that rescales the stream after every block.

Same as the blueprintPartial RoPE on the last 64 of 512 dims, with the inverse rotation on each head's output as in V4. CSA2 layers use θ = 160,000 with YaRN (factor 16 over 65,536 tokens); the two pure-SWA layers use θ = 10,000 without YaRN. The FP4 KV cache is quantized after RoPE.

RoPE by layer type: full RoPE local, p-RoPE globalLocal layers rotate all 256 dims (θ = 10⁴); global layers keep only the 64 fastest of 256 frequency pairs (θ = 10⁶).

CED decoder attention (CSA2 over encoder-projected KV)Layers 20–39 take uncompressed global KV from the encoder's last hidden state and re-select with a hierarchical indexer.

Global GQA with keys reused as valuesEvery sixth layer: full causal attention, 32 query heads and 4 KV heads of dim 512, V taken from the key projection.

CSA2 encoder layers (2× compression, Full + Reuse)Layers 2–19: KV pooled 2→1 and computed in only 3 of 18 layers; the rest reuse that KV and its top-512 selection.

Not used

Same as the blueprintNo output gate. Each of the 64 heads has a learnable sink logit in the softmax denominator, per the reference inference code.

Not used

Same as the blueprint√softplus scores over 384 experts, top-6, with auxiliary-loss-free bias balancing kept separately for text and image tokens (each bias set updated from its own modality's load, speed 0.001) plus a 0.0001 sequence-level loss. Selected scores are normalized and scaled by 1.5. V4's hash-routed early layers are gone.

Dense GeGLU feed-forwardOne gated MLP per layer, 5,376 → 21,504 → 5,376 with tanh-approximated GELU; all 30.7B parameters are active.

Same as the blueprint384 SwiGLU experts of intermediate size 2,304, 6 active per token, in all 40 layers, with V4's clamp (linear branch ±10, gate ≤ 10). The release stores them in FP4.

Not used

Same as the blueprintOne always-active shared SwiGLU expert of intermediate size 2,304 in every MoE layer.

Not used

Same as the blueprintUntied 5,120 → 129,280 head, shared with the DSpark drafter. Like the embedding and the Engram tables it is trained with a Sinkhorn-balanced momentum update instead of AdamW.

Tied LM head with logit soft-cappingLogits come from the input embedding matrix (no separate head) and are squashed by 30·tanh(z/30).

DSpark drafter (trained after pretraining)Three 128-token-window blocks draft 5 tokens per pass from the backbone's last 3 layers; no MTP loss in pretraining.

Autoregressive MTP drafter (4 layers, reads trunk KV)A separate ~500M-parameter, 4-layer Transformer drafts tokens one at a time while attending to the 31B's own KV cache.

Same as the blueprintMuon (momentum 0.95, weight decay 0.1, update RMS 0.18) for backbone linear weights, Engram projections and the vision projector, applied per head to query and key weights. Embeddings, the head and Engram tables use a momentum update with Sinkhorn row/column balancing (Engram learning rate 5×); AdamW covers norms and other non-matrix parameters.

Not disclosed

Same as the blueprintNo dense warm-up: sparse attention is trained from scratch at 64K tokens, and the sequence length is extended to 1M at 34T of the 45T pretraining tokens. DeepSeek reports no training instability.

Not disclosed

Same as the blueprintSFT → RL → OPD with V4's algorithms unchanged; the new work is automated synthesis of agent tasks and environments. RL is asynchronous: a prompt's GRPO group is dispatched as soon as enough rollout slots free up, the off-policy ratio is bounded and overly stale tokens are masked from the loss. A reasoning-effort level from 1 to 100 in the system prompt is trained with an exponentially decaying length penalty.

Not disclosed

Same as the blueprintThe final stage is full-vocabulary on-policy distillation over all domains from more than 40 teachers, which may differ architecturally from the student, run with asynchronous generation.

Not disclosed

Not present

Post-attention RMSNorm (sandwich norm)The attention output is RMS-normalized with a learned gain before the residual add; there is no output gate or sink.

Not present

Post-feed-forward RMSNorm (sandwich norm)The MLP output is RMS-normalized with a learned gain before it is added to the residual stream.

More comparisons