MiniMax-M2.7 vs Gemma 4 31B

=Same as the blueprintByte-level BPE with 200,000 regular tokens and 54 special tokens from id 200,000 (fill-in-the-middle, repository and commit markers, <think>, tool-call tags), padded to a 200,064-entry embedding. The pre-tokenizer splits digit runs into groups of 1–3.

=Same as the blueprintA ~550M-parameter ViT (27 layers, width 1,152, MLP 4,304, 16 heads, 16-px patches) with 2D RoPE and 2D absolute position embeddings, kept frozen during pretraining. Its outputs are pooled 3 × 3, so each soft token covers 48 × 48 px, under a per-image budget of 70, 140, 280 (default), 560 or 1,120 tokens. Video enters as frames; the 31B has no audio encoder.

=Same as the blueprintPer-head RMSNorm with a learned gain on queries and keys (256-dim heads in local layers, 512-dim in global ones), so the logits use scale 1 instead of 1/√d. Values get a gain-free RMSNorm. There is no attention-logit soft-cap.

=Same as the blueprintPartial RoPE in all 62 layers: the first 64 of each head's 128 dims are rotated (rotate-half layout), the other 64 carry no position, with base θ = 5,000,000. No YaRN or other RoPE scaling is configured.

=Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

=Same as the blueprintSigmoid scores computed in FP32 over 256 experts; a per-expert bias (e_score_correction_bias) is added only to choose the top 8, and the 8 unbiased scores are renormalized to sum to 1. The report calls the biases learnable and says they let the auxiliary balance loss be greatly reduced, not removed (the M2 config lists router_aux_loss_coef = 0.001).

=Same as the blueprint256 SwiGLU experts of intermediate size 1,536, top-8 active, in all 62 layers (no dense layers, no shared expert). In MiniMax's 17.8B-A2B ablation, 128 experts with top-8 beat 32 with top-2 at equal active parameters (MATH 19.6 → 24.1, HumanEval 29.7 → 32.5).

=Same as the blueprintUntied 3,072 × 200,064 output projection, left unquantized in the FP8 checkpoint.

=Same as the blueprintDeepSeek-V3-style sequential MTP, each module one Transformer layer. Pretraining used a single module with loss weight 0.3, annealed to 0.1; in the decay phase it was expanded to 3 modules initialized by copying main-model weights, trained alone first and then jointly. The released M2 and M2.7 checkpoints contain no MTP weights.

=Same as the blueprintWeights are released in BF16 and the pretraining number format is not stated. Google also releases quantization-aware-trained checkpoints: block-wise 4-bit Q4_0 (19.2 GB for the 31B vs. 64 GB in BF16 in the report's table) and, for E2B and E4B, mobile quantization with mixed int2/int4 weights and int8 activations.

=Same as the blueprint29.2T pretraining tokens: 19.9T in the constant phase, then a 9.3T-token decay phase that extends context from 8K through 32K to 192K using concatenated code, long PDFs and topic-packed documents. M2 to M2.5 configs allow 196,608 positions and M2.7 allows 204,800, with no RoPE scaling.

?Not disclosed

=Same as the blueprintCISPO (clipped, stop-gradient importance weights) with reward-to-go advantages, process rewards and a completion-time reward, mixed across reasoning, coding, agent and general domains. Runs on Forge: decoupled agent, gateway and trainer, windowed-FIFO scheduling (window 0.3N), and prefix-tree merging of shared prefixes for up to 40× training speedup.

?Not disclosed