GLM-5 vs gpt-oss-120b

=Same as the blueprintDecoupled partial RoPE as in DeepSeek MLA: 64 of each head's 256 q/k dims rotate, using one 64-dim rotary key shared by all heads, with interleaved pairs and base θ = 1,000,000. The DSA indexer rotates 64 of its 128 dims. GLM-5.2 raises θ to 8,000,000 for its 1M window; neither config uses YaRN.

=Same as the blueprintMLA in all 78 layers and the MTP layer, with 64 heads: 2,048-dim query latent, 512-dim KV latent plus a 64-dim rotary key (576 cached values per token per layer), q/k head dim 256 (192 + 64 rotary), v head dim 256. This is the MLA-256 variant: head dim raised from 192 to 256 with one-third fewer heads, keeping training FLOPs and parameters roughly constant while lowering decode compute.

=Same as the blueprintDSA in every layer, not alternated with MLA: a lightning indexer (32 heads × 128 dims, ReLU scores with learned per-head weights) selects the top 2,048 tokens per query and MLA attends only to those. It was added after mid-training: 1,000 warm-up steps on 14 × 202,752-token sequences, then 20B tokens of sparse adaptation. The indexer is frozen in RL and uses a deterministic torch.topk. GLM-5.2 runs it in 21 of 78 layers.

=Same as the blueprintEvery other block (18 of 36) attends only to a banded window of the last 128 tokens, alternating with the dense blocks; same GQA head layout.

=Same as the blueprintNo output gate. Each head instead has a learned sink logit added to the softmax denominator, so it can assign attention mass to nothing.

=Same as the blueprintSigmoid affinities over 256 experts with a per-expert correction bias that affects only the top-8 selection (noaux_tc, no group-limited routing). The selected scores are renormalized to sum to 1 and multiplied by a routed scaling factor of 2.5. GLM-5.2's config pins the router to FP32.

=Same as the blueprintOne always-active shared SwiGLU expert (intermediate size 2,048) in every MoE layer.

=Same as the blueprintOne MTP layer (embedding and hidden-state norms, a projection, then an MLA + DSA + MoE block) whose weights are shared across 3 prediction steps in training. Acceptance length is 2.76 vs. DeepSeek-V3.2's 2.55 at 4 speculative steps. GLM-5.2 reuses the first step's top-k indices and KV in later steps and adds rejection sampling and an end-to-end TV loss: 4.56 → 5.47 accepted tokens at 7 steps in a GLM-5.1-based ablation.

=Same as the blueprintMuon, following GLM-4.5, with Muon Split: the MLA up-projections W_UQ, W_UK and W_UV are split per head and each block is orthogonalized separately, which brought MLA level with GQA-8 in Z.ai's ablation. The distributed implementation all-gathers only each rank's own shards. LR peaks at 2e-4, decays to 4e-5 in pretraining, then 1e-5 in mid-training.

?Not disclosed

=Same as the blueprintMoE weights were post-trained and are released in MXFP4 (block-scaled 4-bit, about 4.25 bits per parameter), which is what lets the 120b run on one 80 GB GPU. Attention, router, embeddings and head are not quantized.

=Same as the blueprintThe last post-training stage is on-policy cross-stage distillation: final checkpoints of earlier SFT and RL stages are teachers, and the advantage is replaced by the stop-gradient log-ratio log π_teacher − log π_train, with group size 1 and batch 1,024. GLM-5.2 merged more than ten expert models this way in about two days.

?Not disclosed

=Same as the blueprintThe shared-weight MTP layer drafts tokens for self-speculative decoding in serving and in RL rollouts, where small-batch decoding makes it most effective. GLM-5.2's MTP improvements raise acceptance length by up to 20%.

=Same as the blueprintMLA caches 576 values per token per layer (512-dim latent + 64-dim rotary key), plus a 128-dim indexer key in layers that run an indexer. RL rollouts use DP-aware routing to maximize prefix-cache reuse. For 1M context, GLM-5.2 adds finer-grained KV memory management in serving; Z.ai notes its architecture cuts per-token FLOPs but not KV size proportionally.