Skip to content
archdiff

models / compare / minimax-m3-vs-gemma-4

MiniMax-M3 vs Gemma 4 31B

How MiniMax-M3 (MiniMax, June 2026) and Gemma 4 31B (Google DeepMind, April 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 17 of 29 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ minimax-m3@2026-06+0~4−4

MiniMax-M3

MiniMax, June 2026, 427.0B total, 23.5B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in blocks 0–2 — AttentionFull grouped-query attention…AttentionMiniMax Sparse Attention (MSA) — AttentionMiniMax Sparse Attention (MS…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headBF16 weights plus an official MXFP8 checkpoint — Training & inferenceBF16 weights plus an official MXFP8 ch…Training & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ gemma-4@2026-04+2~7−6

Gemma 4 31B

Google DeepMind, April 2026, 30.7B

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioSentencePiece BPE with byte fallback — Input & outputSentencePiece BPE with byte fallbackInput & outputVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream with per-layer scalars — Residual & normSingle residual stream with per-layer…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kRoPE by layer type: full RoPE local, p-RoPE global — AttentionRoPE by layer type: full RoPE local, p…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with keys reused as values — AttentionGlobal GQA with keys reused…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sinkPost-attention RMSNorm (sandwich norm) — Residual & normPost-attention RMSNorm (sandwich norm)Residual & norm⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normDense GeGLU feed-forward — Mixture of expertsDense GeGLU feed-forwardMixture of expertsPost-feed-forward RMSNorm (sandwich norm) — Residual & normPost-feed-forward RMSNorm (sandwich no…Residual & normRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormTied LM head with logit soft-capping — Input & outputTied LM head with logit soft-cappingInput & outputAutoregressive MTP drafter (4 layers, reads trunk KV) — Input & outputAutoregressive MTP drafter (4 layers,…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

MiniMax-M3 and Gemma 4 31B differ on 17 of 29 components.

Blueprint componentMiniMax-M3Gemma 4 31B

Same as the blueprintByte-level BPE with the same 200,000 regular tokens, merges and pre-tokenizer as M2.7 (digit runs split into groups of 1–3), padded to a 200,064-entry embedding. Special tokens grow from 54 to 61 (ids 200,000–200,060), adding three frame markers, <mm:think> tags and two other markers, and renaming the tool-call tags from <minimax:tool_call> to <tool_call>. The image and video placeholder tokens were already in M2's set.

SentencePiece BPE with byte fallbackGemini's 262,144-entry SentencePiece vocabulary: character-level BPE, split digits, bytes only as a fallback.

Single residual streamOne d = 6,144 residual stream with pre-norm sublayers; no hyper-connections.

Single residual stream with per-layer scalarsOne d = 5,376 stream, RMSNorm before and after each sublayer, and a stored scalar that rescales the stream after every block.

Same as the blueprintPartial RoPE in all 60 layers, unchanged from M2: the first 64 of each head's 128 dims rotate (rotate-half layout) and the other 64 carry no position, with base θ = 5,000,000. The indexer's queries and key are rotated the same way. The config allows 1,048,576 positions with no YaRN or other scaling.

RoPE by layer type: full RoPE local, p-RoPE globalLocal layers rotate all 256 dims (θ = 10⁴); global layers keep only the 64 fastest of 256 frequency pairs (θ = 10⁶).

Full grouped-query attention in blocks 0–2The first three blocks attend over the whole context: 64 query heads share 4 KV heads of dim 128; no indexer, gate or sink.

Global GQA with keys reused as valuesEvery sixth layer: full causal attention, 32 query heads and 4 KV heads of dim 512, V taken from the key projection.

MiniMax Sparse Attention (MSA)Per GQA group, a 2-projection indexer picks the top 16 of 128-token KV blocks; exact softmax over ≤ 2,048 keys, in 57 of 60 layers.

Not used

Not used

Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

Same as the blueprintAn FP32 linear router scores 128 experts with sigmoids. A per-expert bias (e_score_correction_bias, FP32) is added only to choose the top 4, the 4 unbiased scores are renormalized to sum to 1, and the routed output is multiplied by 2.0 (routed_scaling_factor) before the shared expert is added. MiniMax has not described how M3's biases are updated or whether an auxiliary balance loss is used.

Dense GeGLU feed-forwardOne gated MLP per layer, 5,376 → 21,504 → 5,376 with tanh-approximated GELU; all 30.7B parameters are active.

Same as the blueprint128 experts of intermediate size 3,072, top-4 active, in blocks 3–59 (M2: 256 of width 1,536, top-8). Blocks 0–2 use a dense MLP of width 12,288 instead. Every FFN uses gpt-oss's clamped SwiGLU: gate clamped at 7, linear branch to ±7, output (x_lin + 1) · x_glu · σ(1.702 · x_glu). Routed experts hold 413.1B of the 427.0B parameters.

Not used

Same as the blueprintOne always-active shared expert per MoE layer: a clamped-SwiGLU MLP of width 3,072, the size of one routed expert, added ungated to the scaled routed output. M2 had no shared expert.

Not used

Same as the blueprintUntied 6,144 × 200,064 output projection, left in BF16 in the MXFP8 checkpoint.

Tied LM head with logit soft-cappingLogits come from the input embedding matrix (no separate head) and are squashed by 30·tanh(z/30).

Same as the blueprintNot in the released weights: neither the BF16 nor the MXFP8 checkpoint contains MTP tensors, and the Transformers port discards mtp.* keys on load. The configs disagree on the module count (num_mtp_modules = 7 with num_nextn_predict_layers = 1 in the BF16 config, 1 in the MXFP8 config), and MiniMax has not described M3's MTP. M2 used 3 modules.

Autoregressive MTP drafter (4 layers, reads trunk KV)A separate ~500M-parameter, 4-layer Transformer drafts tokens one at a time while attending to the 31B's own KV cache.

BF16 weights plus an official MXFP8 checkpointBF16 release with an FP32 router; MiniMax-M3-MXFP8 stores the text stack's linears as E4M3 with one 8-bit scale per 32 weights.

Same as the blueprintWeights are released in BF16 and the pretraining number format is not stated. Google also releases quantization-aware-trained checkpoints: block-wise 4-bit Q4_0 (19.2 GB for the 31B vs. 64 GB in BF16 in the report's table) and, for E2B and E4B, mobile quantization with mixed int2/int4 weights and int8 activations.

Same as the blueprintThe config allows 1,048,576 positions with partial RoPE at θ = 5,000,000 and no scaling. MiniMax's API guarantees at least 512K and bills inputs above 512K at a higher rate. M3's context-extension schedule and pretraining token count are not published; in the MSA paper's 109B experiment, about 140B tokens of long-context training after sparse conversion kept RULER-128K level with full attention.

Not disclosed

Same as the blueprintNo overall recipe is published. MiniMax's MaxProof paper covers M3's proof track: a Proof Expert trained with CISPO and group-normalized advantages on a frozen multi-judge verifier's 0–7 score, skipping groups with too little reward spread; a Verifier Expert distilled from that verifier; and a Fixer Expert trained by rejection-sampling fine-tuning. Their capabilities are merged into the released model by an unstated method. The launch blog adds an interactive user simulator for multi-turn coding.

Not disclosed

Not disclosed

Same as the blueprintThe 31B pairs with its released drafter (gemma-4-31B-it-assistant): the drafter proposes several tokens and the 31B verifies them in one forward pass. The drafter's model card claims up to 3× faster decoding at the same output quality. Google's example starts at 4 draft tokens and adapts the count to the acceptance rate.

Not present

Post-attention RMSNorm (sandwich norm)The attention output is RMS-normalized with a learned gain before the residual add; there is no output gate or sink.

Not present

Post-feed-forward RMSNorm (sandwich norm)The MLP output is RMS-normalized with a learned gain before it is added to the residual stream.

More comparisons