Skip to content
archdiff

models / compare / glm-5-vs-kimi-k3

GLM-5 vs Kimi K3

How GLM-5 (Z.ai (Zhipu AI), February 2026) and Kimi K3 (Moonshot AI, July 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 10 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ glm-5@2026-02+0~3−5

GLM-5

Z.ai (Zhipu AI), February 2026, 744B total, 40B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normRMSNorm on MLA latents (no per-head QK-Norm) — Residual & normRMSNorm on MLA latents (no per-head QK…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headINT4 QAT in SFT, FP8 rollouts — Training & inferenceINT4 QAT in SFT, FP8 rolloutsTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ kimi-k3@2026-07+0~5−3

Kimi K3

Moonshot AI, July 2026, 2.8T total, 104B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Block Attention Residuals (AttnRes) — Residual & normBlock Attention Residuals (AttnRes)Residual & normRMSNorm — pre-normRMSNormpre-normQK-Clip and normalized latents (no per-head QK-Norm) — Residual & normQK-Clip and normalized latents (no per…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with Quantile Balancing — Mixture of expertsSigmoid router with Quantile BalancingMixture of expertsStable LatentMoE routed experts (SiTU-GLU) — Mixture of expertsStable LatentMoE routed experts (SiTU-…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP layer turned EAGLE-3 drafter (not in open weights) — Input & outputMTP layer turned EAGLE-3 drafter (not…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

Component by component

GLM-5 and Kimi K3 differ on 10 of 27 components.

Blueprint componentGLM-5Kimi K3

Not used

Same as the blueprintMoonViT-V2: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch with the language-model loss instead of a SigLIP initialization. Images and video share weights (factorized spatial/temporal attention, temporal pooling); a 2×2 pixel shuffle cuts tokens 4× before an MLP projector, allowing inputs up to 3,584 × 3,584 px.

Single residual streamOne d = 6,144 residual stream with pre-norm sublayers; no hyper-connections in GLM-5 through GLM-5.3.

Block Attention Residuals (AttnRes)Each sublayer reads a softmax-weighted mix of the embedding and earlier 12-layer block summaries, not one running sum.

RMSNorm on MLA latents (no per-head QK-Norm)Queries and keys are normalized as compressed latents (2,048 and 512 dims) before up-projection, not per head.

QK-Clip and normalized latents (no per-head QK-Norm)No per-head RMSNorm on q and k: MLA logits are held by K2's weight clipping, KDA's q and k are L2-normalized.

Not used

Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1): 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + SiLU on q, k, v; L2-normalized q, k. K3 bounds the per-channel log-decay as g = −5·σ(eᴬ·z) instead of Kimi Linear's −eᴬ·softplus(z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

Same as the blueprintDSA in every layer, not alternated with MLA: a lightning indexer (32 heads × 128 dims, ReLU scores with learned per-head weights) selects the top 2,048 tokens per query and MLA attends only to those. It was added after mid-training: 1,000 warm-up steps on 14 × 202,752-token sequences, then 20B tokens of sparse adaptation. The indexer is frozen in RL and uses a deterministic torch.topk. GLM-5.2 runs it in 21 of 78 layers.

Not used

Not used

Same as the blueprintBoth mixers end in an input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In MLA it multiplies the attention output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. K3 replaced Kimi Linear's low-rank KDA gate with this full-rank one. No sink logits.

Same as the blueprintSigmoid affinities over 256 experts with a per-expert correction bias that affects only the top-8 selection (noaux_tc, no group-limited routing). The selected scores are renormalized to sum to 1 and multiplied by a routed scaling factor of 2.5. GLM-5.2's config pins the router to FP32.

Sigmoid router with Quantile BalancingSigmoid scores pick 16 of 896 experts; each expert's selection bias is reset every step from a quantile of routing margins.

Same as the blueprint256 routed SwiGLU experts of intermediate size 2,048, top-8 active, in 75 MoE layers. The first 3 layers use a dense SwiGLU FFN of width 12,288 instead. The MTP layer is also an MoE layer.

Stable LatentMoE routed experts (SiTU-GLU)896 SiTU-GLU experts run in a 3,584-dim latent (half of d); 16 fire per token; an RMSNorm precedes the up-projection.

Same as the blueprintOne MTP layer (embedding and hidden-state norms, a projection, then an MLA + DSA + MoE block) whose weights are shared across 3 prediction steps in training. Acceptance length is 2.76 vs. DeepSeek-V3.2's 2.55 at 4 speculative steps. GLM-5.2 reuses the first step's top-k indices and KV in later steps and adds rejection sampling and an end-to-end TV loss: 4.56 → 5.47 accepted tokens at 7 steps in a GLM-5.1-based ablation.

MTP layer turned EAGLE-3 drafter (not in open weights)One pretrained MTP layer is fine-tuned into an EAGLE-3-style draft model that reads features from three depths.

INT4 QAT in SFT, FP8 rolloutsINT4 quantization-aware training during SFT with a bitwise-identical kernel; FP8 inference for RL rollouts and release.

Same as the blueprintQuantization-aware training through all of post-training (SFT and RL): routed-expert weights in MXFP4 (one shared 8-bit scale per 32 values) with MXFP8 activations. Attention, latent MoE projections, shared experts, router, the dense layer and the LM head stay in BF16. Rollouts and training share the scheme, so RL sees no train–inference precision mismatch.

More comparisons