Skip to content
archdiff

models / compare / qwen3-5-vs-deepseek-v4

Qwen3.5-397B-A17B vs DeepSeek-V4-Pro

How Qwen3.5-397B-A17B (Alibaba Qwen, February 2026) and DeepSeek-V4-Pro (DeepSeek, April 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 10 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ qwen3-5@2026-02+0~3−3

Qwen3.5-397B-A17B

Alibaba Qwen, February 2026, 397B total, 17B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGated grouped-query attention — AttentionGated grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ deepseek-v4@2026-04+0~1−3

DeepSeek-V4-Pro

DeepSeek, April 2026, 1.6T total, 49B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersHeavily Compressed Attention (HCA) — AttentionHeavily Compressed Attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Qwen3.5-397B-A17B and DeepSeek-V4-Pro differ on 10 of 27 components.

Blueprint componentQwen3.5-397B-A17BDeepSeek-V4-Pro

Same as the blueprintA 27-layer ViT (width 1,152, 16 heads, MLP 4,304) with 16-px patches; video frames are patched in pairs (temporal patch 2). A 2×2 spatial merge and an MLP project into d = 4,096, and the DeepStack multi-layer injection is disabled (empty index list). Trained by early fusion on trillions of multimodal tokens.

Not used

Single residual streamOne d = 4,096 pre-norm residual stream with zero-centred RMSNorm gains; no hyper-connections or depth attention.

Same as the blueprintmHC with n = 4 streams of width 7,168 around every attention and MoE sublayer. The residual-mixing matrix is projected to doubly stochastic with 20 Sinkhorn–Knopp iterations; read-in uses σ(·) and write-back 2σ(·). DeepSeek reports a 6.7% wall-time overhead per overlapped 1F1B pipeline stage.

Same as the blueprintGated DeltaNet in 45 of 60 layers, three before every attention layer. 16 query/key heads and 64 value heads of dim 128 (each q/k head serves 4 value heads); short causal conv (kernel 4) + SiLU on q, k, v; per-head scalar decay g = −eᴬ·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 64 × 128 × 128 state per sequence, held in FP32.

Not used

Gated grouped-query attentionEvery fourth layer: full causal GQA with 32 query heads sharing 2 KV heads of dim 256, QK-Norm and a sigmoid output gate.

Heavily Compressed Attention (HCA)Every 128 tokens pooled into one KV entry; dense shared-KV attention over all of them plus a 128-token window.

Not used

Same as the blueprintCompressed Sparse Attention in 30 of 61 layers (even layers 2–60), interleaved with HCA. Each KV entry pools 8 overlapping tokens at stride m = 4 with a per-channel softmax. A lightning indexer (64 heads × 128 dims, ReLU scores weighted per head) keeps the top 1,024 entries, which the 128 query heads attend to together with the SWA entries.

Not used

Same as the blueprintV4-Pro has no pure sliding-window layers (V4-Flash makes its first two layers pure SWA). Every CSA and HCA layer adds 128 uncompressed KV entries for the most recent tokens to the same softmax, which covers the tokens in the query's own not-yet-compressed block.

Softmax top-k routerA linear router scores 512 experts; softmax, keep the top 10, renormalize. No selection bias.

Same as the blueprint√softplus affinities over 384 experts with a per-expert bias used only to pick the top 6 (auxiliary-loss-free balancing, bias speed 0.001) plus a 0.0001 sequence-wise balance loss. The selected scores are normalized and scaled by 2.5. V3's node limit is removed, and the first 3 MoE layers pick experts by a fixed token-id hash. Anticipatory Routing (indices from Δt-step-old parameters) was switched on only around loss spikes.

Not disclosed

Same as the blueprintMuon for most matrices: 10 hybrid Newton–Schulz steps (8 with coefficients (3.4445, −4.7750, 2.0315), then 2 with (2, −1.5, 0.5)), Nesterov momentum 0.95, weight decay 0.1, update RMS rescaled to 0.18. AdamW (β = 0.9 / 0.95) trains the embedding, prediction head, RMSNorm gains and mHC static biases and gates. No QK-Clip.

Not disclosed

Same as the blueprintMore than ten specialist teachers are merged into one student by on-policy distillation with a weighted, full-vocabulary reverse-KL loss. Teacher logits are rebuilt on the fly from cached last-layer hidden states instead of being stored.

Same as the blueprintMTP self-speculation is the recommended serving mode: SGLang NEXTN with 3 steps and 4 draft tokens, or vLLM qwen3_next_mtp with 2 speculative tokens. Qwen's RL rollout engine also uses speculative decoding.

Not disclosed

More comparisons