Skip to content
archdiff

models / compare / gemma-4-vs-minimax-m2

Gemma 4 31B vs MiniMax-M2.7

How Gemma 4 31B (Google DeepMind, April 2026) and MiniMax-M2.7 (MiniMax, March 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 16 of 29 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ gemma-4@2026-04+2~7−6

Gemma 4 31B

Google DeepMind, April 2026, 30.7B

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioSentencePiece BPE with byte fallback — Input & outputSentencePiece BPE with byte fallbackInput & outputVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream with per-layer scalars — Residual & normSingle residual stream with per-layer…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kRoPE by layer type: full RoPE local, p-RoPE global — AttentionRoPE by layer type: full RoPE local, p…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with keys reused as values — AttentionGlobal GQA with keys reused…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sinkPost-attention RMSNorm (sandwich norm) — Residual & normPost-attention RMSNorm (sandwich norm)Residual & norm⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normDense GeGLU feed-forward — Mixture of expertsDense GeGLU feed-forwardMixture of expertsPost-feed-forward RMSNorm (sandwich norm) — Residual & normPost-feed-forward RMSNorm (sandwich no…Residual & normRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormTied LM head with logit soft-capping — Input & outputTied LM head with logit soft-cappingInput & outputAutoregressive MTP drafter (4 layers, reads trunk KV) — Input & outputAutoregressive MTP drafter (4 layers,…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ minimax-m2@2026-03+0~4−7

MiniMax-M2.7

MiniMax, March 2026, 229.9B total, 9.8B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normLayer-wide QK RMSNorm — Residual & normLayer-wide QK RMSNormResidual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in all 62 layers — AttentionFull grouped-query attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 block-scaled checkpoint — Training & inferenceFP8 block-scaled checkpointTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Gemma 4 31B and MiniMax-M2.7 differ on 16 of 29 components.

Blueprint componentGemma 4 31BMiniMax-M2.7

SentencePiece BPE with byte fallbackGemini's 262,144-entry SentencePiece vocabulary: character-level BPE, split digits, bytes only as a fallback.

Same as the blueprintByte-level BPE with 200,000 regular tokens and 54 special tokens from id 200,000 (fill-in-the-middle, repository and commit markers, <think>, tool-call tags), padded to a 200,064-entry embedding. The pre-tokenizer splits digit runs into groups of 1–3.

Same as the blueprintA ~550M-parameter ViT (27 layers, width 1,152, MLP 4,304, 16 heads, 16-px patches) with 2D RoPE and 2D absolute position embeddings, kept frozen during pretraining. Its outputs are pooled 3 × 3, so each soft token covers 48 × 48 px, under a per-image budget of 70, 140, 280 (default), 560 or 1,120 tokens. Video enters as frames; the 31B has no audio encoder.

Not used

Single residual stream with per-layer scalarsOne d = 5,376 stream, RMSNorm before and after each sublayer, and a stored scalar that rescales the stream after every block.

Single residual streamOne d = 3,072 residual stream with pre-norm sublayers; no hyper-connections.

Same as the blueprintPer-head RMSNorm with a learned gain on queries and keys (256-dim heads in local layers, 512-dim in global ones), so the logits use scale 1 instead of 1/√d. Values get a gain-free RMSNorm. There is no attention-logit soft-cap.

Layer-wide QK RMSNormOne RMSNorm over all query heads (6,144 dims) and one over all key heads (1,024 dims), applied before RoPE.

RoPE by layer type: full RoPE local, p-RoPE globalLocal layers rotate all 256 dims (θ = 10⁴); global layers keep only the 64 fastest of 256 frequency pairs (θ = 10⁶).

Same as the blueprintPartial RoPE in all 62 layers: the first 64 of each head's 128 dims are rotated (rotate-half layout), the other 64 carry no position, with base θ = 5,000,000. No YaRN or other RoPE scaling is configured.

Global GQA with keys reused as valuesEvery sixth layer: full causal attention, 32 query heads and 4 KV heads of dim 512, V taken from the key projection.

Full grouped-query attention in all 62 layersCausal softmax over the whole context in every layer: 48 query heads share 8 KV heads of dim 128; no window, sink or gate.

Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

Not used

Post-attention RMSNorm (sandwich norm)The attention output is RMS-normalized with a learned gain before the residual add; there is no output gate or sink.

Not present

Dense GeGLU feed-forwardOne gated MLP per layer, 5,376 → 21,504 → 5,376 with tanh-approximated GELU; all 30.7B parameters are active.

Same as the blueprintSigmoid scores computed in FP32 over 256 experts; a per-expert bias (e_score_correction_bias) is added only to choose the top 8, and the 8 unbiased scores are renormalized to sum to 1. The report calls the biases learnable and says they let the auxiliary balance loss be greatly reduced, not removed (the M2 config lists router_aux_loss_coef = 0.001).

Post-feed-forward RMSNorm (sandwich norm)The MLP output is RMS-normalized with a learned gain before it is added to the residual stream.

Not present

Not used

Same as the blueprint256 SwiGLU experts of intermediate size 1,536, top-8 active, in all 62 layers (no dense layers, no shared expert). In MiniMax's 17.8B-A2B ablation, 128 experts with top-8 beat 32 with top-2 at equal active parameters (MATH 19.6 → 24.1, HumanEval 29.7 → 32.5).

Tied LM head with logit soft-cappingLogits come from the input embedding matrix (no separate head) and are squashed by 30·tanh(z/30).

Same as the blueprintUntied 3,072 × 200,064 output projection, left unquantized in the FP8 checkpoint.

Autoregressive MTP drafter (4 layers, reads trunk KV)A separate ~500M-parameter, 4-layer Transformer drafts tokens one at a time while attending to the 31B's own KV cache.

Same as the blueprintDeepSeek-V3-style sequential MTP, each module one Transformer layer. Pretraining used a single module with loss weight 0.3, annealed to 0.1; in the decay phase it was expanded to 3 modules initialized by copying main-model weights, trained alone first and then jointly. The released M2 and M2.7 checkpoints contain no MTP weights.

Same as the blueprintWeights are released in BF16 and the pretraining number format is not stated. Google also releases quantization-aware-trained checkpoints: block-wise 4-bit Q4_0 (19.2 GB for the 31B vs. 64 GB in BF16 in the report's table) and, for E2B and E4B, mobile quantization with mixed int2/int4 weights and int8 activations.

FP8 block-scaled checkpointOpen weights ship as FP8 E4M3 with 128 × 128 block scales; router, its bias, embeddings and LM head stay in higher precision.

Not disclosed

Same as the blueprint29.2T pretraining tokens: 19.9T in the constant phase, then a 9.3T-token decay phase that extends context from 8K through 32K to 192K using concatenated code, long PDFs and topic-packed documents. M2 to M2.5 configs allow 196,608 positions and M2.7 allows 204,800, with no RoPE scaling.

Not disclosed

Same as the blueprintCISPO (clipped, stop-gradient importance weights) with reward-to-go advantages, process rewards and a completion-time reward, mixed across reasoning, coding, agent and general domains. Runs on Forge: decoupled agent, gateway and trainer, windowed-FIFO scheduling (window 0.3N), and prefix-tree merging of shared prefixes for up to 40× training speedup.

More comparisons