Skip to content
archdiff

models / compare / minimax-m2-vs-qwen3-5

MiniMax-M2.7 vs Qwen3.5-397B-A17B

How MiniMax-M2.7 (MiniMax, March 2026) and Qwen3.5-397B-A17B (Alibaba Qwen, February 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 8 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ minimax-m2@2026-03+0~4−7

MiniMax-M2.7

MiniMax, March 2026, 229.9B total, 9.8B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normLayer-wide QK RMSNorm — Residual & normLayer-wide QK RMSNormResidual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in all 62 layers — AttentionFull grouped-query attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 block-scaled checkpoint — Training & inferenceFP8 block-scaled checkpointTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ qwen3-5@2026-02+0~3−3

Qwen3.5-397B-A17B

Alibaba Qwen, February 2026, 397B total, 17B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGated grouped-query attention — AttentionGated grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

MiniMax-M2.7 and Qwen3.5-397B-A17B differ on 8 of 27 components.

Blueprint componentMiniMax-M2.7Qwen3.5-397B-A17B

Not used

Same as the blueprintA 27-layer ViT (width 1,152, 16 heads, MLP 4,304) with 16-px patches; video frames are patched in pairs (temporal patch 2). A 2×2 spatial merge and an MLP project into d = 4,096, and the DeepStack multi-layer injection is disabled (empty index list). Trained by early fusion on trillions of multimodal tokens.

Layer-wide QK RMSNormOne RMSNorm over all query heads (6,144 dims) and one over all key heads (1,024 dims), applied before RoPE.

Same as the blueprintPer-head RMSNorm (zero-centred gain) over the 256-dim heads on q and k in the 15 gated-attention layers, applied before RoPE. The Gated DeltaNet layers instead L2-normalize q and k inside the kernel.

Not used

Same as the blueprintGated DeltaNet in 45 of 60 layers, three before every attention layer. 16 query/key heads and 64 value heads of dim 128 (each q/k head serves 4 value heads); short causal conv (kernel 4) + SiLU on q, k, v; per-head scalar decay g = −eᴬ·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 64 × 128 × 128 state per sequence, held in FP32.

Full grouped-query attention in all 62 layersCausal softmax over the whole context in every layer: 48 query heads share 8 KV heads of dim 128; no window, sink or gate.

Gated grouped-query attentionEvery fourth layer: full causal GQA with 32 query heads sharing 2 KV heads of dim 256, QK-Norm and a sigmoid output gate.

Not used

Same as the blueprintIn the attention layers q_proj emits 2 × 32 × 256 values per token; the second half is an elementwise sigmoid gate, computed from the layer input, applied to the attention output before W_o. Gated DeltaNet layers gate their output with SiLU(z) after a per-head RMSNorm. No sink logits.

Same as the blueprintSigmoid scores computed in FP32 over 256 experts; a per-expert bias (e_score_correction_bias) is added only to choose the top 8, and the 8 unbiased scores are renormalized to sum to 1. The report calls the biases learnable and says they let the auxiliary balance loss be greatly reduced, not removed (the M2 config lists router_aux_loss_coef = 0.001).

Softmax top-k routerA linear router scores 512 experts; softmax, keep the top 10, renormalize. No selection bias.

Not used

Same as the blueprintOne shared SwiGLU expert per layer (intermediate 1,024) runs on every token, but its output is scaled by a learned per-token sigmoid gate σ(w·x), w ∈ ℝ^4096, before it is added to the routed sum. Unlike DeepSeek-style shared experts it is not ungated.

FP8 block-scaled checkpointOpen weights ship as FP8 E4M3 with 128 × 128 block scales; router, its bias, embeddings and LM head stay in higher precision.

Same as the blueprintPretraining ran a native FP8 pipeline (activations, MoE routing and GEMMs in FP8, with runtime monitoring keeping sensitive layers in BF16), which Qwen credits with ~50% less activation memory and >10% speedup; RL also runs FP8 end to end. The official FP8 checkpoint (128 × 128 weight blocks) keeps embeddings, LM head, routers, shared-expert gates, the DeltaNet conv and decay/β projections and the vision encoder in BF16.

More comparisons