Skip to content
archdiff

models / compare / glm-5-vs-deepseek-v4

GLM-5 vs DeepSeek-V4-Pro

How GLM-5 (Z.ai (Zhipu AI), February 2026) and DeepSeek-V4-Pro (DeepSeek, April 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 7 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ glm-5@2026-02+0~3−5

GLM-5

Z.ai (Zhipu AI), February 2026, 744B total, 40B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normRMSNorm on MLA latents (no per-head QK-Norm) — Residual & normRMSNorm on MLA latents (no per-head QK…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headINT4 QAT in SFT, FP8 rollouts — Training & inferenceINT4 QAT in SFT, FP8 rolloutsTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ deepseek-v4@2026-04+0~1−3

DeepSeek-V4-Pro

DeepSeek, April 2026, 1.6T total, 49B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersHeavily Compressed Attention (HCA) — AttentionHeavily Compressed Attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

GLM-5 and DeepSeek-V4-Pro differ on 7 of 27 components.

Blueprint componentGLM-5DeepSeek-V4-Pro

Single residual streamOne d = 6,144 residual stream with pre-norm sublayers; no hyper-connections in GLM-5 through GLM-5.3.

Same as the blueprintmHC with n = 4 streams of width 7,168 around every attention and MoE sublayer. The residual-mixing matrix is projected to doubly stochastic with 20 Sinkhorn–Knopp iterations; read-in uses σ(·) and write-back 2σ(·). DeepSeek reports a 6.7% wall-time overhead per overlapped 1F1B pipeline stage.

RMSNorm on MLA latents (no per-head QK-Norm)Queries and keys are normalized as compressed latents (2,048 and 512 dims) before up-projection, not per head.

Same as the blueprintAn RMS normalization of each of the 128 query heads and an RMSNorm on the single 512-dim KV entry, applied just before core attention in both CSA and HCA. DeepSeek credits this for training with Muon without QK-Clip.

Same as the blueprintMLA in all 78 layers and the MTP layer, with 64 heads: 2,048-dim query latent, 512-dim KV latent plus a 64-dim rotary key (576 cached values per token per layer), q/k head dim 256 (192 + 64 rotary), v head dim 256. This is the MLA-256 variant: head dim raised from 192 to 256 with one-third fewer heads, keeping training FLOPs and parameters roughly constant while lowering decode compute.

Heavily Compressed Attention (HCA)Every 128 tokens pooled into one KV entry; dense shared-KV attention over all of them plus a 128-token window.

Not used

Same as the blueprintV4-Pro has no pure sliding-window layers (V4-Flash makes its first two layers pure SWA). Every CSA and HCA layer adds 128 uncompressed KV entries for the most recent tokens to the same softmax, which covers the tokens in the query's own not-yet-compressed block.

Not used

Same as the blueprintNo output gate. Each of the 128 heads has a learnable sink logit whose exponential is added to the softmax denominator in both CSA and HCA, so its attention weights can sum to less than 1.

INT4 QAT in SFT, FP8 rolloutsINT4 quantization-aware training during SFT with a bitwise-identical kernel; FP8 inference for RL rollouts and release.

Same as the blueprintFP8 mixed-precision training, with most released weights in FP8. In post-training, MXFP4 quantization-aware training covers the routed-expert weights and the CSA indexer's QK path, which is cached and multiplied in FP4; index scores drop to BF16 for a 2× faster top-k at 99.7% recall. FP4 expert weights dequantize losslessly to FP8, so the FP8 training stack is reused.

Same as the blueprintThe shared-weight MTP layer drafts tokens for self-speculative decoding in serving and in RL rollouts, where small-batch decoding makes it most effective. GLM-5.2's MTP improvements raise acceptance length by up to 20%.

Not disclosed

More comparisons