GLM-5 vs Qwen3.8-2.4T-A95B

=Same as the blueprintPer-head RMSNorm (zero-centred gain) over the 256-dim heads on q and k in the 23 gated-attention layers, applied before RoPE. The Gated DeltaNet layers instead L2-normalize q and k inside the kernel.

=Same as the blueprintGated DeltaNet in 69 of 92 layers, three before every attention layer. 16 query/key heads and 128 value heads of dim 128, so each q/k head serves 8 value heads (4 in Qwen3.5). A depthwise causal conv (kernel 4) + SiLU runs over q, k and v; each value head has a scalar decay g = −exp(A_log)·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 128 × 128 × 128 state per sequence, which the config holds in FP32.

=Same as the blueprintMLA in all 78 layers and the MTP layer, with 64 heads: 2,048-dim query latent, 512-dim KV latent plus a 64-dim rotary key (576 cached values per token per layer), q/k head dim 256 (192 + 64 rotary), v head dim 256. This is the MLA-256 variant: head dim raised from 192 to 256 with one-third fewer heads, keeping training FLOPs and parameters roughly constant while lowering decode compute.

=Same as the blueprintDSA in every layer, not alternated with MLA: a lightning indexer (32 heads × 128 dims, ReLU scores with learned per-head weights) selects the top 2,048 tokens per query and MLA attends only to those. It was added after mid-training: 1,000 warm-up steps on 14 × 202,752-token sequences, then 20B tokens of sparse adaptation. The indexer is frozen in RL and uses a deterministic torch.topk. GLM-5.2 runs it in 21 of 78 layers.

=Same as the blueprintIn the attention layers q_proj emits 2 × 64 × 256 values per token; each head's second 256 values are an elementwise sigmoid gate on its attention output before W_o. The Gated DeltaNet layers gate their output with SiLU(z) after a per-head RMSNorm (output_gate_type: swish). No sink logits.

=Same as the blueprintSigmoid affinities over 256 experts with a per-expert correction bias that affects only the top-8 selection (noaux_tc, no group-limited routing). The selected scores are renormalized to sum to 1 and multiplied by a routed scaling factor of 2.5. GLM-5.2's config pins the router to FP32.

=Same as the blueprintMuon, following GLM-4.5, with Muon Split: the MLA up-projections W_UQ, W_UK and W_UV are split per head and each block is orthogonalized separately, which brought MLA level with GQA-8 in Z.ai's ablation. The distributed implementation all-gathers only each rank's own shards. LR peaks at 2e-4, decays to 4e-5 in pretraining, then 1e-5 in mid-training.

?Not disclosed

=Same as the blueprintQwen has not published the training precision. The weights ship in BF16 (4.89 TB). The official FP8 checkpoint uses 128 × 128 weight blocks with dynamic activation scales, but converts only the routed experts: embeddings, LM head, routers, shared experts and their gates, every attention and Gated DeltaNet projection, and the MTP input projection stay BF16. Qwen3.5's FP8 release also quantized the attention, DeltaNet and shared-expert projections.

=Same as the blueprintThe last post-training stage is on-policy cross-stage distillation: final checkpoints of earlier SFT and RL stages are teachers, and the advantage is replaced by the stop-gradient log-ratio log π_teacher − log π_train, with group size 1 and batch 1,024. GLM-5.2 merged more than ten expert models this way in about two days.

?Not disclosed