GLM-5 vs Qwen3.5-397B-A17B

=Same as the blueprintA 27-layer ViT (width 1,152, 16 heads, MLP 4,304) with 16-px patches; video frames are patched in pairs (temporal patch 2). A 2×2 spatial merge and an MLP project into d = 4,096, and the DeepStack multi-layer injection is disabled (empty index list). Trained by early fusion on trillions of multimodal tokens.

=Same as the blueprintPer-head RMSNorm (zero-centred gain) over the 256-dim heads on q and k in the 15 gated-attention layers, applied before RoPE. The Gated DeltaNet layers instead L2-normalize q and k inside the kernel.

=Same as the blueprintGated DeltaNet in 45 of 60 layers, three before every attention layer. 16 query/key heads and 64 value heads of dim 128 (each q/k head serves 4 value heads); short causal conv (kernel 4) + SiLU on q, k, v; per-head scalar decay g = −eᴬ·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 64 × 128 × 128 state per sequence, held in FP32.

=Same as the blueprintMLA in all 78 layers and the MTP layer, with 64 heads: 2,048-dim query latent, 512-dim KV latent plus a 64-dim rotary key (576 cached values per token per layer), q/k head dim 256 (192 + 64 rotary), v head dim 256. This is the MLA-256 variant: head dim raised from 192 to 256 with one-third fewer heads, keeping training FLOPs and parameters roughly constant while lowering decode compute.

=Same as the blueprintDSA in every layer, not alternated with MLA: a lightning indexer (32 heads × 128 dims, ReLU scores with learned per-head weights) selects the top 2,048 tokens per query and MLA attends only to those. It was added after mid-training: 1,000 warm-up steps on 14 × 202,752-token sequences, then 20B tokens of sparse adaptation. The indexer is frozen in RL and uses a deterministic torch.topk. GLM-5.2 runs it in 21 of 78 layers.

=Same as the blueprintIn the attention layers q_proj emits 2 × 32 × 256 values per token; the second half is an elementwise sigmoid gate, computed from the layer input, applied to the attention output before W_o. Gated DeltaNet layers gate their output with SiLU(z) after a per-head RMSNorm. No sink logits.

=Same as the blueprintSigmoid affinities over 256 experts with a per-expert correction bias that affects only the top-8 selection (noaux_tc, no group-limited routing). The selected scores are renormalized to sum to 1 and multiplied by a routed scaling factor of 2.5. GLM-5.2's config pins the router to FP32.

=Same as the blueprintMuon, following GLM-4.5, with Muon Split: the MLA up-projections W_UQ, W_UK and W_UV are split per head and each block is orthogonalized separately, which brought MLA level with GQA-8 in Z.ai's ablation. The distributed implementation all-gathers only each rank's own shards. LR peaks at 2e-4, decays to 4e-5 in pretraining, then 1e-5 in mid-training.

?Not disclosed

=Same as the blueprintPretraining ran a native FP8 pipeline (activations, MoE routing and GEMMs in FP8, with runtime monitoring keeping sensitive layers in BF16), which Qwen credits with ~50% less activation memory and >10% speedup; RL also runs FP8 end to end. The official FP8 checkpoint (128 × 128 weight blocks) keeps embeddings, LM head, routers, shared-expert gates, the DeltaNet conv and decay/β projections and the vision encoder in BF16.

=Same as the blueprintThe last post-training stage is on-policy cross-stage distillation: final checkpoints of earlier SFT and RL stages are teachers, and the advantage is replaced by the stop-gradient log-ratio log π_teacher − log π_train, with group size 1 and batch 1,024. GLM-5.2 merged more than ten expert models this way in about two days.

?Not disclosed