Skip to content
archdiff

models / compare / glm-5-vs-kimi-k3-1-pro

GLM-5 vs Kimi K3.1 Pro

How GLM-5 (Z.ai (Zhipu AI), February 2026) and Kimi K3.1 Pro (Moonshot AI, October 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 10 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ glm-5@2026-02+0~3−5

GLM-5

Z.ai (Zhipu AI), February 2026, 744B total, 40B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normRMSNorm on MLA latents (no per-head QK-Norm) — Residual & normRMSNorm on MLA latents (no per-head QK…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headINT4 QAT in SFT, FP8 rollouts — Training & inferenceINT4 QAT in SFT, FP8 rolloutsTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ kimi-k3-1-pro@2026-10+0~5−2

Kimi K3.1 Pro

Moonshot AI, October 2026, 2.8T total, 104B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Block Attention Residuals (AttnRes) — Residual & normBlock Attention Residuals (AttnRes)Residual & normRMSNorm — pre-normRMSNormpre-normQK-Clip and normalized latents; LayerNorm on the indexer key — Residual & normQK-Clip and normalized latents; LayerN…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with Quantile Balancing — Mixture of expertsSigmoid router with Quantile BalancingMixture of expertsStable LatentMoE routed experts (SiTU-GLU) — Mixture of expertsStable LatentMoE routed experts (SiTU-…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsNo MTP layer in the open checkpoint — Input & outputNo MTP layer in the open checkpointInput & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

GLM-5 and Kimi K3.1 Pro differ on 10 of 27 components.

Blueprint componentGLM-5Kimi K3.1 Pro

Not used

Same as the blueprintMoonViT-V2, as in K3: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch jointly with the language model. Images and video share weights; a 2×2 pixel shuffle cuts tokens 4× before an MLP projector to 7,168 dims. The model card lists text and image input; Moonshot's API docs also list video.

Single residual streamOne d = 6,144 residual stream with pre-norm sublayers; no hyper-connections in GLM-5 through GLM-5.3.

Block Attention Residuals (AttnRes)Each sublayer reads a softmax-weighted mix of the embedding and earlier 12-layer block summaries, as in K3.

RMSNorm on MLA latents (no per-head QK-Norm)Queries and keys are normalized as compressed latents (2,048 and 512 dims) before up-projection, not per head.

QK-Clip and normalized latents; LayerNorm on the indexer keyNo per-head QK-Norm: K2-style weight clipping for MLA, L2-normalized q/k in KDA, a LayerNorm on the DSA indexer key.

Not used

Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1), unchanged from K3: 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + Swish on q, k, v; L2-normalized q, k. The log-decay is bounded as g = −5·σ(eᴬ·z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

Not used

Same as the blueprintBoth mixers keep K3's input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In gated DSA it multiplies the sparse MLA output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. No sink logits.

Same as the blueprintSigmoid affinities over 256 experts with a per-expert correction bias that affects only the top-8 selection (noaux_tc, no group-limited routing). The selected scores are renormalized to sum to 1 and multiplied by a routed scaling factor of 2.5. GLM-5.2's config pins the router to FP32.

Sigmoid router with Quantile BalancingSigmoid scores pick 16 of 896 experts; each expert's selection bias is set every step from a quantile of routing margins.

Same as the blueprint256 routed SwiGLU experts of intermediate size 2,048, top-8 active, in 75 MoE layers. The first 3 layers use a dense SwiGLU FFN of width 12,288 instead. The MTP layer is also an MoE layer.

Stable LatentMoE routed experts (SiTU-GLU)896 SiTU-GLU experts run in a 3,584-dim latent (half of d); 16 fire per token; an RMSNorm precedes the up-projection.

Same as the blueprintOne MTP layer (embedding and hidden-state norms, a projection, then an MLA + DSA + MoE block) whose weights are shared across 3 prediction steps in training. Acceptance length is 2.76 vs. DeepSeek-V3.2's 2.55 at 4 speculative steps. GLM-5.2 reuses the first step's top-k indices and KV in later steps and adds rejection sampling and an end-to-end TV loss: 4.56 → 5.47 accepted tokens at 7 steps in a GLM-5.1-based ablation.

No MTP layer in the open checkpointThe checkpoint ships without MTP weights; Moonshot has not described a K3.1 drafter, so serving engines bring their own.

INT4 QAT in SFT, FP8 rolloutsINT4 quantization-aware training during SFT with a bitwise-identical kernel; FP8 inference for RL rollouts and release.

Same as the blueprintRouted-expert weights in MXFP4 (group 32, 8-bit shared scale) with MXFP8 activations under quantization-aware training, as in K3. Everything else is BF16, including the new indexer, whose 178.5M parameters are the only change in the per-dtype parameter counts from K3. vLLM and SGLang cache MLA latents and indexer keys in FP8 by default.

Same as the blueprintThe last post-training stage is on-policy cross-stage distillation: final checkpoints of earlier SFT and RL stages are teachers, and the advantage is replaced by the stop-gradient log-ratio log π_teacher − log π_train, with group size 1 and batch 1,024. GLM-5.2 merged more than ten expert models this way in about two days.

Not disclosed

More comparisons