Kimi K3.1 Pro vs MiniMax-M3

=Same as the blueprintPer-head Gemma-style RMSNorm (ε = 1e-6) on q and k before RoPE, with one 128-dim gain shared by all 64 query heads and one by the 4 key heads; the indexer's queries and key get their own. This replaces M2's layer-wide norm over the concatenated projections (qk_norm_type per_layer → per_head), so head-sharded tensor parallelism no longer needs a cross-rank reduction for it.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1), unchanged from K3: 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + Swish on q, k, v; L2-normalized q, k. The log-decay is bounded as g = −5·σ(eᴬ·z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in the same 24 layers as K3 (4, 8, …, 92 plus 93), every projection the same shape: 96 heads, queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer), q/k head dim 192, v head dim 128. The difference is the key set: each query now attends only to the 2,048 tokens its layer's DSA indexer selects, not the full prefix.

=Same as the blueprintNew in K3.1: a DSA lightning indexer in each of the 24 MLA layers, each with its own weights (vLLM: no index sharing). 32 heads × 128 dims; queries from the 1,536-dim query latent (1,536 → 4,096), one shared 128-dim key from the hidden state with a LayerNorm, per-head weights from a 7,168 → 32 projection. Scores I = Σₕ wₕ·ReLU(qₕ·k) pick the top 2,048 tokens per query. 7.44M BF16 parameters per layer. DeepSeek-V3.2's indexer has 64 heads. How it was trained is unpublished.

=Same as the blueprintBoth mixers keep K3's input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In gated DSA it multiplies the sparse MLA output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. No sink logits.

=Same as the blueprintAn FP32 linear router scores 128 experts with sigmoids. A per-expert bias (e_score_correction_bias, FP32) is added only to choose the top 4, the 4 unbiased scores are renormalized to sum to 1, and the routed output is multiplied by 2.0 (routed_scaling_factor) before the shared expert is added. MiniMax has not described how M3's biases are updated or whether an auxiliary balance loss is used.

=Same as the blueprint128 experts of intermediate size 3,072, top-4 active, in blocks 3–59 (M2: 256 of width 1,536, top-8). Blocks 0–2 use a dense MLP of width 12,288 instead. Every FFN uses gpt-oss's clamped SwiGLU: gate clamped at 7, linear branch to ±7, output (x_lin + 1) · x_glu · σ(1.702 · x_glu). Routed experts hold 413.1B of the 427.0B parameters.

=Same as the blueprintNot in the released weights: neither the BF16 nor the MXFP8 checkpoint contains MTP tensors, and the Transformers port discards mtp.* keys on load. The configs disagree on the module count (num_mtp_modules = 7 with num_nextn_predict_layers = 1 in the BF16 config, 1 in the MXFP8 config), and MiniMax has not described M3's MTP. M2 used 3 modules.

=Same as the blueprintInherited from K3's pretraining: Per-Head Muon, which orthogonalizes each head's block of the Q, K and V momentum separately, plus Kimi K2's weight clipping, a cosine LR schedule with 1% linear warmup and weight decay 0.1. Moonshot has not said which optimizer trained the new indexer or K3.1's own training stages.

?Not disclosed

=Same as the blueprintRouted-expert weights in MXFP4 (group 32, 8-bit shared scale) with MXFP8 activations under quantization-aware training, as in K3. Everything else is BF16, including the new indexer, whose 178.5M parameters are the only change in the per-dtype parameter counts from K3. vLLM and SGLang cache MLA latents and indexer keys in FP8 by default.

=Same as the blueprintMoonshot has not described speculative decoding for K3.1. The vLLM recipe drafts with Inferact's 5-layer DSpark preview model and SGLang with RadixArk's DFlash2 preview (block 8). Neither drafter comes from Moonshot, and Moonshot has not explained the 2–3× speed of the hosted Highspeed variant.

?Not disclosed