Skip to content
archdiff

models / compare / minimax-m3-vs-minimax-m2

MiniMax-M3 vs MiniMax-M2.7

How MiniMax-M3 (MiniMax, June 2026) and MiniMax-M2.7 (MiniMax, March 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 7 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ minimax-m3@2026-06+0~4−4

MiniMax-M3

MiniMax, June 2026, 427.0B total, 23.5B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in blocks 0–2 — AttentionFull grouped-query attention…AttentionMiniMax Sparse Attention (MSA) — AttentionMiniMax Sparse Attention (MS…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headBF16 weights plus an official MXFP8 checkpoint — Training & inferenceBF16 weights plus an official MXFP8 ch…Training & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ minimax-m2@2026-03+0~4−7

MiniMax-M2.7

MiniMax, March 2026, 229.9B total, 9.8B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normLayer-wide QK RMSNorm — Residual & normLayer-wide QK RMSNormResidual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in all 62 layers — AttentionFull grouped-query attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 block-scaled checkpoint — Training & inferenceFP8 block-scaled checkpointTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

MiniMax-M3 and MiniMax-M2.7 differ on 7 of 27 components.

Blueprint componentMiniMax-M3MiniMax-M2.7

Same as the blueprintA 32-layer CLIP-style ViT (width 1,280, 16 heads, GELU MLP of 5,120, LayerNorm; 631M parameters) embeds 14 × 14 patches with a Conv3d that spans 2 frames for video, using 3D (time, height, width) RoPE. A GELU MLP lifts each patch to 6,144 and a second fuses each 2 × 2 group into one token (234M projector parameters). MiniMax says M3 trained on mixed modalities from the first step and lists text, image and video inputs.

Not used

Same as the blueprintPer-head Gemma-style RMSNorm (ε = 1e-6) on q and k before RoPE, with one 128-dim gain shared by all 64 query heads and one by the 4 key heads; the indexer's queries and key get their own. This replaces M2's layer-wide norm over the concatenated projections (qk_norm_type per_layer → per_head), so head-sharded tensor parallelism no longer needs a cross-rank reduction for it.

Layer-wide QK RMSNormOne RMSNorm over all query heads (6,144 dims) and one over all key heads (1,024 dims), applied before RoPE.

Full grouped-query attention in blocks 0–2The first three blocks attend over the whole context: 64 query heads share 4 KV heads of dim 128; no indexer, gate or sink.

Full grouped-query attention in all 62 layersCausal softmax over the whole context in every layer: 48 query heads share 8 KV heads of dim 128; no window, sink or gate.

MiniMax Sparse Attention (MSA)Per GQA group, a 2-projection indexer picks the top 16 of 128-token KV blocks; exact softmax over ≤ 2,048 keys, in 57 of 60 layers.

Not used

Same as the blueprintOne always-active shared expert per MoE layer: a clamped-SwiGLU MLP of width 3,072, the size of one routed expert, added ungated to the scaled routed output. M2 had no shared expert.

Not used

BF16 weights plus an official MXFP8 checkpointBF16 release with an FP32 router; MiniMax-M3-MXFP8 stores the text stack's linears as E4M3 with one 8-bit scale per 32 weights.

FP8 block-scaled checkpointOpen weights ship as FP8 E4M3 with 128 × 128 block scales; router, its bias, embeddings and LM head stay in higher precision.

Not disclosed

Same as the blueprintThe 3 MTP modules draft tokens that the main model verifies in one forward pass. During RL they are co-trained with the policy via a top-K KL loss so acceptance stays high as the policy shifts.

More comparisons