GLM-5 vs Gemma 4 31B

=Same as the blueprintByte-level BPE with 154,820 regular tokens and 36 special tokens (chat roles, <think>, tool-call and argument tags), padded to a 154,880-entry embedding (GLM-4.5: 151,552). The pre-tokenizer splits digit runs into groups of 1–3.

=Same as the blueprintA ~550M-parameter ViT (27 layers, width 1,152, MLP 4,304, 16 heads, 16-px patches) with 2D RoPE and 2D absolute position embeddings, kept frozen during pretraining. Its outputs are pooled 3 × 3, so each soft token covers 48 × 48 px, under a per-image budget of 70, 140, 280 (default), 560 or 1,120 tokens. Video enters as frames; the 31B has no audio encoder.

=Same as the blueprintPer-head RMSNorm with a learned gain on queries and keys (256-dim heads in local layers, 512-dim in global ones), so the logits use scale 1 instead of 1/√d. Values get a gain-free RMSNorm. There is no attention-logit soft-cap.

=Same as the blueprintDecoupled partial RoPE as in DeepSeek MLA: 64 of each head's 256 q/k dims rotate, using one 64-dim rotary key shared by all heads, with interleaved pairs and base θ = 1,000,000. The DSA indexer rotates 64 of its 128 dims. GLM-5.2 raises θ to 8,000,000 for its 1M window; neither config uses YaRN.

=Same as the blueprintMLA in all 78 layers and the MTP layer, with 64 heads: 2,048-dim query latent, 512-dim KV latent plus a 64-dim rotary key (576 cached values per token per layer), q/k head dim 256 (192 + 64 rotary), v head dim 256. This is the MLA-256 variant: head dim raised from 192 to 256 with one-third fewer heads, keeping training FLOPs and parameters roughly constant while lowering decode compute.

=Same as the blueprintDSA in every layer, not alternated with MLA: a lightning indexer (32 heads × 128 dims, ReLU scores with learned per-head weights) selects the top 2,048 tokens per query and MLA attends only to those. It was added after mid-training: 1,000 warm-up steps on 14 × 202,752-token sequences, then 20B tokens of sparse adaptation. The indexer is frozen in RL and uses a deterministic torch.topk. GLM-5.2 runs it in 21 of 78 layers.

=Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

=Same as the blueprintSigmoid affinities over 256 experts with a per-expert correction bias that affects only the top-8 selection (noaux_tc, no group-limited routing). The selected scores are renormalized to sum to 1 and multiplied by a routed scaling factor of 2.5. GLM-5.2's config pins the router to FP32.

=Same as the blueprint256 routed SwiGLU experts of intermediate size 2,048, top-8 active, in 75 MoE layers. The first 3 layers use a dense SwiGLU FFN of width 12,288 instead. The MTP layer is also an MoE layer.

=Same as the blueprintOne always-active shared SwiGLU expert (intermediate size 2,048) in every MoE layer.

=Same as the blueprintUntied 6,144 × 154,880 output projection. The MTP layer's output head is co-located with it on the last pipeline stage so the two share parameters.

=Same as the blueprintOne MTP layer (embedding and hidden-state norms, a projection, then an MLA + DSA + MoE block) whose weights are shared across 3 prediction steps in training. Acceptance length is 2.76 vs. DeepSeek-V3.2's 2.55 at 4 speculative steps. GLM-5.2 reuses the first step's top-k indices and KV in later steps and adds rejection sampling and an end-to-end TV loss: 4.56 → 5.47 accepted tokens at 7 steps in a GLM-5.1-based ablation.

=Same as the blueprintMuon, following GLM-4.5, with Muon Split: the MLA up-projections W_UQ, W_UK and W_UV are split per head and each block is orthogonalized separately, which brought MLA level with GQA-8 in Z.ai's ablation. The distributed implementation all-gathers only each rank's own shards. LR peaks at 2e-4, decays to 4e-5 in pretraining, then 1e-5 in mid-training.

?Not disclosed

=Same as the blueprintWeights are released in BF16 and the pretraining number format is not stated. Google also releases quantization-aware-trained checkpoints: block-wise 4-bit Q4_0 (19.2 GB for the 31B vs. 64 GB in BF16 in the report's table) and, for E2B and E4B, mobile quantization with mixed int2/int4 weights and int8 activations.

=Same as the blueprint28.5T tokens in total. Mid-training extends context in three stages: 32K (1T tokens), 128K (500B) and 200K (50B), and DSA is then trained at 202,752 tokens. GLM-5.2 reaches 1,048,576 tokens with a larger RoPE base and expanded 1M-context training on coding-agent data, using IndexShare from its 128K mid-training stage onward.

?Not disclosed

=Same as the blueprintReasoning RL is GRPO with IcePop masking of train/inference mismatch (β = 2), no KL term, ε_low = 0.2, ε_high = 0.28, group size 32. Agentic RL runs fully asynchronously on slime with 1k+ concurrent rollouts, a token-in/token-out gateway and double-sided token-level importance sampling. GLM-5.2 moves long-horizon RL to critic-based PPO on individual rollouts.

?Not disclosed

=Same as the blueprintThe last post-training stage is on-policy cross-stage distillation: final checkpoints of earlier SFT and RL stages are teachers, and the advantage is replaced by the stop-gradient log-ratio log π_teacher − log π_train, with group size 1 and batch 1,024. GLM-5.2 merged more than ten expert models this way in about two days.

?Not disclosed