Skip to content
archdiff

models / compare / minimax-m3-vs-qwen3-8

MiniMax-M3 vs Qwen3.8-2.4T-A95B

How MiniMax-M3 (MiniMax, June 2026) and Qwen3.8-2.4T-A95B (Alibaba Qwen, August 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 8 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ minimax-m3@2026-06+0~4−4

MiniMax-M3

MiniMax, June 2026, 427.0B total, 23.5B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in blocks 0–2 — AttentionFull grouped-query attention…AttentionMiniMax Sparse Attention (MSA) — AttentionMiniMax Sparse Attention (MS…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headBF16 weights plus an official MXFP8 checkpoint — Training & inferenceBF16 weights plus an official MXFP8 ch…Training & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ qwen3-8@2026-08+0~3−4

Qwen3.8-2.4T-A95B

Alibaba Qwen, August 2026, 2.4T total, 95B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGated grouped-query attention — AttentionGated grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

MiniMax-M3 and Qwen3.8-2.4T-A95B differ on 8 of 27 components.

Blueprint componentMiniMax-M3Qwen3.8-2.4T-A95B

Same as the blueprintA 32-layer CLIP-style ViT (width 1,280, 16 heads, GELU MLP of 5,120, LayerNorm; 631M parameters) embeds 14 × 14 patches with a Conv3d that spans 2 frames for video, using 3D (time, height, width) RoPE. A GELU MLP lifts each patch to 6,144 and a second fuses each 2 × 2 group into one token (234M projector parameters). MiniMax says M3 trained on mixed modalities from the first step and lists text, image and video inputs.

Not used

Not used

Same as the blueprintGated DeltaNet in 69 of 92 layers, three before every attention layer. 16 query/key heads and 128 value heads of dim 128, so each q/k head serves 8 value heads (4 in Qwen3.5). A depthwise causal conv (kernel 4) + SiLU runs over q, k and v; each value head has a scalar decay g = −exp(A_log)·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 128 × 128 × 128 state per sequence, which the config holds in FP32.

Full grouped-query attention in blocks 0–2The first three blocks attend over the whole context: 64 query heads share 4 KV heads of dim 128; no indexer, gate or sink.

Gated grouped-query attentionEvery fourth layer: full causal GQA with 64 query heads sharing 4 KV heads of dim 256, QK-Norm and a sigmoid output gate.

MiniMax Sparse Attention (MSA)Per GQA group, a 2-projection indexer picks the top 16 of 128-token KV blocks; exact softmax over ≤ 2,048 keys, in 57 of 60 layers.

Not used

Not used

Same as the blueprintIn the attention layers q_proj emits 2 × 64 × 256 values per token; each head's second 256 values are an elementwise sigmoid gate on its attention output before W_o. The Gated DeltaNet layers gate their output with SiLU(z) after a per-head RMSNorm (output_gate_type: swish). No sink logits.

Same as the blueprintAn FP32 linear router scores 128 experts with sigmoids. A per-expert bias (e_score_correction_bias, FP32) is added only to choose the top 4, the 4 unbiased scores are renormalized to sum to 1, and the routed output is multiplied by 2.0 (routed_scaling_factor) before the shared expert is added. MiniMax has not described how M3's biases are updated or whether an auxiliary balance loss is used.

Softmax top-k routerA linear router scores 512 experts; softmax, keep the top 10, renormalize. No selection bias.

BF16 weights plus an official MXFP8 checkpointBF16 release with an FP32 router; MiniMax-M3-MXFP8 stores the text stack's linears as E4M3 with one 8-bit scale per 32 weights.

Same as the blueprintQwen has not published the training precision. The weights ship in BF16 (4.89 TB). The official FP8 checkpoint uses 128 × 128 weight blocks with dynamic activation scales, but converts only the routed experts: embeddings, LM head, routers, shared experts and their gates, every attention and Gated DeltaNet projection, and the MTP input projection stay BF16. Qwen3.5's FP8 release also quantized the attention, DeltaNet and shared-expert projections.

Not disclosed

Same as the blueprintThe serving recipes linked from the model card use MTP self-speculation. vLLM runs method mtp with 3 speculative tokens, which its recipe measures at roughly 2.3× per-user output rate; it calls MTP-1 (64.8% acceptance) not worth it. SGLang runs NEXTN with 3 steps, top-k 1 and 4 draft tokens. SGLang also offers DSpark with a separately trained third-party draft model (RadixArk).

More comparisons