Skip to content
archdiff

models / compare / minimax-m3-vs-gpt-oss-120b

MiniMax-M3 vs gpt-oss-120b

How MiniMax-M3 (MiniMax, June 2026) and gpt-oss-120b (OpenAI, August 2025) are built, drawn on the same November 2026 blueprint. They make different choices on 13 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ minimax-m3@2026-06+0~4−4

MiniMax-M3

MiniMax, June 2026, 427.0B total, 23.5B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in blocks 0–2 — AttentionFull grouped-query attention…AttentionMiniMax Sparse Attention (MSA) — AttentionMiniMax Sparse Attention (MS…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headBF16 weights plus an official MXFP8 checkpoint — Training & inferenceBF16 weights plus an official MXFP8 ch…Training & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ gpt-oss-120b@2025-08+0~4−9

gpt-oss-120b

OpenAI, August 2025, 117B total, 5.1B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kFull RoPE with YaRN scaling — AttentionFull RoPE with YaRN scalingAttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersDense grouped-query attention — AttentionDense grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

MiniMax-M3 and gpt-oss-120b differ on 13 of 27 components.

Blueprint componentMiniMax-M3gpt-oss-120b

Same as the blueprintA 32-layer CLIP-style ViT (width 1,280, 16 heads, GELU MLP of 5,120, LayerNorm; 631M parameters) embeds 14 × 14 patches with a Conv3d that spans 2 frames for video, using 3D (time, height, width) RoPE. A GELU MLP lifts each patch to 6,144 and a second fuses each 2 × 2 group into one token (234M projector parameters). MiniMax says M3 trained on mixed modalities from the first step and lists text, image and video inputs.

Not used

Same as the blueprintPer-head Gemma-style RMSNorm (ε = 1e-6) on q and k before RoPE, with one 128-dim gain shared by all 64 query heads and one by the 4 key heads; the indexer's queries and key get their own. This replaces M2's layer-wide norm over the concatenated projections (qk_norm_type per_layer → per_head), so head-sharded tensor parallelism no longer needs a cross-rank reduction for it.

Not used

Same as the blueprintPartial RoPE in all 60 layers, unchanged from M2: the first 64 of each head's 128 dims rotate (rotate-half layout) and the other 64 carry no position, with base θ = 5,000,000. The indexer's queries and key are rotated the same way. The config allows 1,048,576 positions with no YaRN or other scaling.

Full RoPE with YaRN scalingRotary embeddings on all 64 head dims, θ = 150,000, stretched 32× with YaRN.

Full grouped-query attention in blocks 0–2The first three blocks attend over the whole context: 64 query heads share 4 KV heads of dim 128; no indexer, gate or sink.

Dense grouped-query attentionFull causal attention with 64 query heads sharing 8 KV heads, in every other block.

MiniMax Sparse Attention (MSA)Per GQA group, a 2-projection indexer picks the top 16 of 128-token KV blocks; exact softmax over ≤ 2,048 keys, in 57 of 60 layers.

Not used

Not used

Same as the blueprintEvery other block (18 of 36) attends only to a banded window of the last 128 tokens, alternating with the dense blocks; same GQA head layout.

Not used

Same as the blueprintNo output gate. Each head instead has a learned sink logit added to the softmax denominator, so it can assign attention mass to nothing.

Same as the blueprintAn FP32 linear router scores 128 experts with sigmoids. A per-expert bias (e_score_correction_bias, FP32) is added only to choose the top 4, the 4 unbiased scores are renormalized to sum to 1, and the routed output is multiplied by 2.0 (routed_scaling_factor) before the shared expert is added. MiniMax has not described how M3's biases are updated or whether an auxiliary balance loss is used.

Softmax top-k routerA linear router picks the top 4 of 128 experts; softmax over those 4 logits gives the weights.

Same as the blueprintOne always-active shared expert per MoE layer: a clamped-SwiGLU MLP of width 3,072, the size of one routed expert, added ungated to the scaled routed output. M2 had no shared expert.

Not used

Same as the blueprintNot in the released weights: neither the BF16 nor the MXFP8 checkpoint contains MTP tensors, and the Transformers port discards mtp.* keys on load. The configs disagree on the module count (num_mtp_modules = 7 with num_nextn_predict_layers = 1 in the BF16 config, 1 in the MXFP8 config), and MiniMax has not described M3's MTP. M2 used 3 modules.

Not used

BF16 weights plus an official MXFP8 checkpointBF16 release with an FP32 router; MiniMax-M3-MXFP8 stores the text stack's linears as E4M3 with one 8-bit scale per 32 weights.

Same as the blueprintMoE weights were post-trained and are released in MXFP4 (block-scaled 4-bit, about 4.25 bits per parameter), which is what lets the 120b run on one 80 GB GPU. Attention, router, embeddings and head are not quantized.

Not disclosed

Not used

Same as the blueprintNo KV compression: MSA selects what to read, not what to store. Per token the stack caches 60 × 1,024 K/V values plus a 128-dim index key in each of the 57 sparse layers, 134.25 KiB in BF16 (M2: 248 KiB), about 144 GB at 1,048,576 tokens. Each decode step reads at most 2,048 tokens per KV group in sparse layers but scans every index key. SGLang's HiSparse offloads the sparse layers' K/V to pinned host memory.

Not used

More comparisons