MiniMax-M2.7 vs Kimi K3

=Same as the blueprintMoonViT-V2: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch with the language-model loss instead of a SigLIP initialization. Images and video share weights (factorized spatial/temporal attention, temporal pooling); a 2×2 pixel shuffle cuts tokens 4× before an MLP projector, allowing inputs up to 3,584 × 3,584 px.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1): 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + SiLU on q, k, v; L2-normalized q, k. K3 bounds the per-channel log-decay as g = −5·σ(eᴬ·z) instead of Kimi Linear's −eᴬ·softplus(z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in 24 layers: every fourth layer (4, 8, …, 92) plus layer 93. 96 heads; queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer); q/k head dim 192, v head dim 128. The attention output is kept in FP32 during training to avoid flash attention's biased rounding.

=Same as the blueprintBoth mixers end in an input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In MLA it multiplies the attention output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. K3 replaced Kimi Linear's low-rank KDA gate with this full-rank one. No sink logits.

=Same as the blueprintSigmoid scores computed in FP32 over 256 experts; a per-expert bias (e_score_correction_bias) is added only to choose the top 8, and the 8 unbiased scores are renormalized to sum to 1. The report calls the biases learnable and says they let the auxiliary balance loss be greatly reduced, not removed (the M2 config lists router_aux_loss_coef = 0.001).

=Same as the blueprint256 SwiGLU experts of intermediate size 1,536, top-8 active, in all 62 layers (no dense layers, no shared expert). In MiniMax's 17.8B-A2B ablation, 128 experts with top-8 beat 32 with top-2 at equal active parameters (MATH 19.6 → 24.1, HumanEval 29.7 → 32.5).

=Same as the blueprintTwo full-width shared experts per MoE layer, stored fused as one SiTU-GLU MLP with intermediate size 6,144 (2 × 3,072) at d = 7,168. They read the un-projected input, are ungated and stay in BF16.

=Same as the blueprintDeepSeek-V3-style sequential MTP, each module one Transformer layer. Pretraining used a single module with loss weight 0.3, annealed to 0.1; in the decay phase it was expanded to 3 modules initialized by copying main-model weights, trained alone first and then jointly. The released M2 and M2.7 checkpoints contain no MTP weights.

?Not disclosed

=Same as the blueprintPer-Head Muon: momentum of the Q, K and V projections is split along the head dimension and each head's block is orthogonalized separately, equalizing update scale across heads; other matrices use standard Muon. Pretraining also keeps Kimi K2's weight-clipping mechanism, a cosine LR schedule with 1% linear warmup and weight decay 0.1.

=Same as the blueprintQuantization-aware training through all of post-training (SFT and RL): routed-expert weights in MXFP4 (one shared 8-bit scale per 32 values) with MXFP8 activations. Attention, latent MoE projections, shared experts, router, the dense layer and the LM head stay in BF16. Rollouts and training share the scheme, so RL sees no train–inference precision mismatch.

?Not disclosed

=Same as the blueprintMulti-teacher on-policy distillation merges the nine domain × effort experts: for a sampled domain and effort level, the matching teacher scores each student token with the clipped log-ratio clip(sg(log π_teacher − log π_θ), −R_max, R_max), used as a dense reward inside the RL framework. Top-k distillation objectives showed no advantage.