Skip to content
archdiff

models / compare / qwen3-5-vs-minimax-m3

Qwen3.5-397B-A17B vs MiniMax-M3

How Qwen3.5-397B-A17B (Alibaba Qwen, February 2026) and MiniMax-M3 (MiniMax, June 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 7 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ qwen3-5@2026-02+0~3−3

Qwen3.5-397B-A17B

Alibaba Qwen, February 2026, 397B total, 17B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGated grouped-query attention — AttentionGated grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ minimax-m3@2026-06+0~4−4

MiniMax-M3

MiniMax, June 2026, 427.0B total, 23.5B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in blocks 0–2 — AttentionFull grouped-query attention…AttentionMiniMax Sparse Attention (MSA) — AttentionMiniMax Sparse Attention (MS…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headBF16 weights plus an official MXFP8 checkpoint — Training & inferenceBF16 weights plus an official MXFP8 ch…Training & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Qwen3.5-397B-A17B and MiniMax-M3 differ on 7 of 27 components.

Blueprint componentQwen3.5-397B-A17BMiniMax-M3

Same as the blueprintGated DeltaNet in 45 of 60 layers, three before every attention layer. 16 query/key heads and 64 value heads of dim 128 (each q/k head serves 4 value heads); short causal conv (kernel 4) + SiLU on q, k, v; per-head scalar decay g = −eᴬ·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 64 × 128 × 128 state per sequence, held in FP32.

Not used

Gated grouped-query attentionEvery fourth layer: full causal GQA with 32 query heads sharing 2 KV heads of dim 256, QK-Norm and a sigmoid output gate.

Full grouped-query attention in blocks 0–2The first three blocks attend over the whole context: 64 query heads share 4 KV heads of dim 128; no indexer, gate or sink.

Not used

MiniMax Sparse Attention (MSA)Per GQA group, a 2-projection indexer picks the top 16 of 128-token KV blocks; exact softmax over ≤ 2,048 keys, in 57 of 60 layers.

Same as the blueprintIn the attention layers q_proj emits 2 × 32 × 256 values per token; the second half is an elementwise sigmoid gate, computed from the layer input, applied to the attention output before W_o. Gated DeltaNet layers gate their output with SiLU(z) after a per-head RMSNorm. No sink logits.

Not used

Softmax top-k routerA linear router scores 512 experts; softmax, keep the top 10, renormalize. No selection bias.

Same as the blueprintAn FP32 linear router scores 128 experts with sigmoids. A per-expert bias (e_score_correction_bias, FP32) is added only to choose the top 4, the 4 unbiased scores are renormalized to sum to 1, and the routed output is multiplied by 2.0 (routed_scaling_factor) before the shared expert is added. MiniMax has not described how M3's biases are updated or whether an auxiliary balance loss is used.

Same as the blueprintPretraining ran a native FP8 pipeline (activations, MoE routing and GEMMs in FP8, with runtime monitoring keeping sensitive layers in BF16), which Qwen credits with ~50% less activation memory and >10% speedup; RL also runs FP8 end to end. The official FP8 checkpoint (128 × 128 weight blocks) keeps embeddings, LM head, routers, shared-expert gates, the DeltaNet conv and decay/β projections and the vision encoder in BF16.

BF16 weights plus an official MXFP8 checkpointBF16 release with an FP32 router; MiniMax-M3-MXFP8 stores the text stack's linears as E4M3 with one 8-bit scale per 32 weights.

Same as the blueprintMTP self-speculation is the recommended serving mode: SGLang NEXTN with 3 steps and 4 draft tokens, or vLLM qwen3_next_mtp with 2 speculative tokens. Qwen's RL rollout engine also uses speculative decoding.

Not disclosed

More comparisons