Kimi K3.1 Pro vs MiniMax-M2.7

=Same as the blueprintMoonViT-V2, as in K3: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch jointly with the language model. Images and video share weights; a 2×2 pixel shuffle cuts tokens 4× before an MLP projector to 7,168 dims. The model card lists text and image input; Moonshot's API docs also list video.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1), unchanged from K3: 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + Swish on q, k, v; L2-normalized q, k. The log-decay is bounded as g = −5·σ(eᴬ·z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in the same 24 layers as K3 (4, 8, …, 92 plus 93), every projection the same shape: 96 heads, queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer), q/k head dim 192, v head dim 128. The difference is the key set: each query now attends only to the 2,048 tokens its layer's DSA indexer selects, not the full prefix.

=Same as the blueprintNew in K3.1: a DSA lightning indexer in each of the 24 MLA layers, each with its own weights (vLLM: no index sharing). 32 heads × 128 dims; queries from the 1,536-dim query latent (1,536 → 4,096), one shared 128-dim key from the hidden state with a LayerNorm, per-head weights from a 7,168 → 32 projection. Scores I = Σₕ wₕ·ReLU(qₕ·k) pick the top 2,048 tokens per query. 7.44M BF16 parameters per layer. DeepSeek-V3.2's indexer has 64 heads. How it was trained is unpublished.

=Same as the blueprintBoth mixers keep K3's input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In gated DSA it multiplies the sparse MLA output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. No sink logits.

=Same as the blueprintSigmoid scores computed in FP32 over 256 experts; a per-expert bias (e_score_correction_bias) is added only to choose the top 8, and the 8 unbiased scores are renormalized to sum to 1. The report calls the biases learnable and says they let the auxiliary balance loss be greatly reduced, not removed (the M2 config lists router_aux_loss_coef = 0.001).

=Same as the blueprint256 SwiGLU experts of intermediate size 1,536, top-8 active, in all 62 layers (no dense layers, no shared expert). In MiniMax's 17.8B-A2B ablation, 128 experts with top-8 beat 32 with top-2 at equal active parameters (MATH 19.6 → 24.1, HumanEval 29.7 → 32.5).

=Same as the blueprintTwo full-width shared experts per MoE layer, stored fused as one SiTU-GLU MLP with intermediate size 6,144 (2 × 3,072) at d = 7,168. They read the un-projected input, are ungated and stay in BF16.

=Same as the blueprintDeepSeek-V3-style sequential MTP, each module one Transformer layer. Pretraining used a single module with loss weight 0.3, annealed to 0.1; in the decay phase it was expanded to 3 modules initialized by copying main-model weights, trained alone first and then jointly. The released M2 and M2.7 checkpoints contain no MTP weights.

=Same as the blueprintInherited from K3's pretraining: Per-Head Muon, which orthogonalizes each head's block of the Q, K and V momentum separately, plus Kimi K2's weight clipping, a cosine LR schedule with 1% linear warmup and weight decay 0.1. Moonshot has not said which optimizer trained the new indexer or K3.1's own training stages.

?Not disclosed

=Same as the blueprintRouted-expert weights in MXFP4 (group 32, 8-bit shared scale) with MXFP8 activations under quantization-aware training, as in K3. Everything else is BF16, including the new indexer, whose 178.5M parameters are the only change in the per-dtype parameter counts from K3. vLLM and SGLang cache MLA latents and indexer keys in FP8 by default.