Kimi K3 vs GLM-5

=Same as the blueprintMoonViT-V2: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch with the language-model loss instead of a SigLIP initialization. Images and video share weights (factorized spatial/temporal attention, temporal pooling); a 2×2 pixel shuffle cuts tokens 4× before an MLP projector, allowing inputs up to 3,584 × 3,584 px.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1): 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + SiLU on q, k, v; L2-normalized q, k. K3 bounds the per-channel log-decay as g = −5·σ(eᴬ·z) instead of Kimi Linear's −eᴬ·softplus(z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintDSA in every layer, not alternated with MLA: a lightning indexer (32 heads × 128 dims, ReLU scores with learned per-head weights) selects the top 2,048 tokens per query and MLA attends only to those. It was added after mid-training: 1,000 warm-up steps on 14 × 202,752-token sequences, then 20B tokens of sparse adaptation. The indexer is frozen in RL and uses a deterministic torch.topk. GLM-5.2 runs it in 21 of 78 layers.

=Same as the blueprintBoth mixers end in an input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In MLA it multiplies the attention output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. K3 replaced Kimi Linear's low-rank KDA gate with this full-rank one. No sink logits.

=Same as the blueprintSigmoid affinities over 256 experts with a per-expert correction bias that affects only the top-8 selection (noaux_tc, no group-limited routing). The selected scores are renormalized to sum to 1 and multiplied by a routed scaling factor of 2.5. GLM-5.2's config pins the router to FP32.

=Same as the blueprint256 routed SwiGLU experts of intermediate size 2,048, top-8 active, in 75 MoE layers. The first 3 layers use a dense SwiGLU FFN of width 12,288 instead. The MTP layer is also an MoE layer.

=Same as the blueprintOne MTP layer (embedding and hidden-state norms, a projection, then an MLA + DSA + MoE block) whose weights are shared across 3 prediction steps in training. Acceptance length is 2.76 vs. DeepSeek-V3.2's 2.55 at 4 speculative steps. GLM-5.2 reuses the first step's top-k indices and KV in later steps and adds rejection sampling and an end-to-end TV loss: 4.56 → 5.47 accepted tokens at 7 steps in a GLM-5.1-based ablation.

=Same as the blueprintQuantization-aware training through all of post-training (SFT and RL): routed-expert weights in MXFP4 (one shared 8-bit scale per 32 values) with MXFP8 activations. Attention, latent MoE projections, shared experts, router, the dense layer and the LM head stay in BF16. Rollouts and training share the scheme, so RL sees no train–inference precision mismatch.