Gemma 4 31B vs Kimi K3

=Same as the blueprinttiktoken byte-level BPE with 163,584 merge ranks plus 256 reserved special tokens, 163,840 ids in total (the same 160K size as Kimi K2). The pre-tokenizer keeps runs of Han characters together and splits digits into groups of 1–3.

=Same as the blueprintPer-head RMSNorm with a learned gain on queries and keys (256-dim heads in local layers, 512-dim in global ones), so the logits use scale 1 instead of 1/√d. Values get a gain-free RMSNorm. There is no attention-logit soft-cap.

=Same as the blueprintNoPE throughout. The 24 MLA layers apply no rotary or other positional encoding (the 64-dim decoupled key slot of MLA is kept but never rotated), and KDA's decay and short convolutions carry order. Context therefore extends to 1M tokens without RoPE rescaling or YaRN.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1): 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + SiLU on q, k, v; L2-normalized q, k. K3 bounds the per-channel log-decay as g = −5·σ(eᴬ·z) instead of Kimi Linear's −eᴬ·softplus(z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in 24 layers: every fourth layer (4, 8, …, 92) plus layer 93. 96 heads; queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer); q/k head dim 192, v head dim 128. The attention output is kept in FP32 during training to avoid flash attention's biased rounding.

=Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

=Same as the blueprintBoth mixers end in an input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In MLA it multiplies the attention output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. K3 replaced Kimi Linear's low-rank KDA gate with this full-rank one. No sink logits.

=Same as the blueprintTwo full-width shared experts per MoE layer, stored fused as one SiTU-GLU MLP with intermediate size 6,144 (2 × 3,072) at d = 7,168. They read the un-projected input, are ungated and stay in BF16.

=Same as the blueprintUntied 7,168 → 163,840 output projection, excluded from MXFP4 quantization.

?Not disclosed

=Same as the blueprintPer-Head Muon: momentum of the Q, K and V projections is split along the head dimension and each head's block is orthogonalized separately, equalizing update scale across heads; other matrices use standard Muon. Pretraining also keeps Kimi K2's weight-clipping mechanism, a cosine LR schedule with 1% linear warmup and weight decay 0.1.

?Not disclosed

=Same as the blueprintFour-stage curriculum: 8K → 64K tokens during pretraining, then 256K → 1M during the cooldown, with upsampled long documents and videos plus synthetic tasks that need evidence spread over the full window. No positional encoding means no RoPE rescaling; KDA Context Parallelism exchanges only fixed-size state fragments between ranks.

?Not disclosed

=Same as the blueprintRL in three domains (general tasks, general agents, coding agents), each trained at three effort levels (low, high, max), giving nine expert policies. Partial rollouts pause generation once a fraction λ of trajectories finish, with a per-token regularizer tolerating the staleness. Effort is trained by budget control (reward −1 above τ·b₀(x)); non-verifiable tasks use an agentic generative reward model with binary comparisons.

?Not disclosed

=Same as the blueprintMulti-teacher on-policy distillation merges the nine domain × effort experts: for a sampled domain and effort level, the matching teacher scores each student token with the clipped log-ratio clip(sg(log π_teacher − log π_θ), −R_max, R_max), used as a dense reward inside the RL framework. Top-k distillation objectives showed no advantage.