Kimi K3.1 Pro vs Gemma 4 31B

=Same as the blueprintUnchanged from K3 (the tiktoken.model file is byte-identical): byte-level BPE with 163,584 merge ranks plus 256 reserved special tokens, 163,840 ids in total. The pre-tokenizer keeps runs of Han characters together and splits digits into groups of 1–3.

=Same as the blueprintPer-head RMSNorm with a learned gain on queries and keys (256-dim heads in local layers, 512-dim in global ones), so the logits use scale 1 instead of 1/√d. Values get a gain-free RMSNorm. There is no attention-logit soft-cap.

=Same as the blueprintNoPE throughout. MLA applies no rotary encoding (the 64-dim decoupled key slot is kept but never rotated), and the new indexer sets index_rope_dim = 0, whereas DeepSeek-V3.2's indexer rotates 64 of its 128 dims. KDA's decay and short convolutions carry order, so the 1M window needs no RoPE rescaling or YaRN.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1), unchanged from K3: 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + Swish on q, k, v; L2-normalized q, k. The log-decay is bounded as g = −5·σ(eᴬ·z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in the same 24 layers as K3 (4, 8, …, 92 plus 93), every projection the same shape: 96 heads, queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer), q/k head dim 192, v head dim 128. The difference is the key set: each query now attends only to the 2,048 tokens its layer's DSA indexer selects, not the full prefix.

=Same as the blueprintNew in K3.1: a DSA lightning indexer in each of the 24 MLA layers, each with its own weights (vLLM: no index sharing). 32 heads × 128 dims; queries from the 1,536-dim query latent (1,536 → 4,096), one shared 128-dim key from the hidden state with a LayerNorm, per-head weights from a 7,168 → 32 projection. Scores I = Σₕ wₕ·ReLU(qₕ·k) pick the top 2,048 tokens per query. 7.44M BF16 parameters per layer. DeepSeek-V3.2's indexer has 64 heads. How it was trained is unpublished.

=Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

=Same as the blueprintBoth mixers keep K3's input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In gated DSA it multiplies the sparse MLA output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. No sink logits.

=Same as the blueprintTwo full-width shared experts per MoE layer, stored fused as one SiTU-GLU MLP with intermediate size 6,144 (2 × 3,072) at d = 7,168. They read the un-projected input, are ungated and stay in BF16.

=Same as the blueprintUntied 7,168 → 163,840 output projection, kept in BF16 (excluded from MXFP4 quantization).

=Same as the blueprintInherited from K3's pretraining: Per-Head Muon, which orthogonalizes each head's block of the Q, K and V momentum separately, plus Kimi K2's weight clipping, a cosine LR schedule with 1% linear warmup and weight decay 0.1. Moonshot has not said which optimizer trained the new indexer or K3.1's own training stages.

?Not disclosed

=Same as the blueprintThe 1,048,576-token window comes from K3's four-stage curriculum (8K → 64K in pretraining, 256K → 1M in cooldown), with no positional encoding to rescale. DSA now caps each MLA query at 2,048 attended tokens, while the indexer still scores the whole prefix. The context lengths the indexer was trained at are not published.

?Not disclosed

=Same as the blueprintPost-trained from K3 with large-scale joint RL across hundreds of downstream domains and substantially more post-training compute than K3. For the open weights, cyber-related data was removed from the mid-training corpus and the RL environment mix, which cut xdev-bench-hard pass@3 from 90.5% to 81.0%. Reasoning effort is low, high or max (default max).

?Not disclosed