Skip to content
archdiff

models / compare / gpt-oss-120b-vs-minimax-m2

gpt-oss-120b vs MiniMax-M2.7

How gpt-oss-120b (OpenAI, August 2025) and MiniMax-M2.7 (MiniMax, March 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 10 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ gpt-oss-120b@2025-08+0~4−9

gpt-oss-120b

OpenAI, August 2025, 117B total, 5.1B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kFull RoPE with YaRN scaling — AttentionFull RoPE with YaRN scalingAttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersDense grouped-query attention — AttentionDense grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ minimax-m2@2026-03+0~4−7

MiniMax-M2.7

MiniMax, March 2026, 229.9B total, 9.8B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normLayer-wide QK RMSNorm — Residual & normLayer-wide QK RMSNormResidual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in all 62 layers — AttentionFull grouped-query attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 block-scaled checkpoint — Training & inferenceFP8 block-scaled checkpointTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

gpt-oss-120b and MiniMax-M2.7 differ on 10 of 27 components.

Blueprint componentgpt-oss-120bMiniMax-M2.7

Not used

Layer-wide QK RMSNormOne RMSNorm over all query heads (6,144 dims) and one over all key heads (1,024 dims), applied before RoPE.

Full RoPE with YaRN scalingRotary embeddings on all 64 head dims, θ = 150,000, stretched 32× with YaRN.

Same as the blueprintPartial RoPE in all 62 layers: the first 64 of each head's 128 dims are rotated (rotate-half layout), the other 64 carry no position, with base θ = 5,000,000. No YaRN or other RoPE scaling is configured.

Dense grouped-query attentionFull causal attention with 64 query heads sharing 8 KV heads, in every other block.

Full grouped-query attention in all 62 layersCausal softmax over the whole context in every layer: 48 query heads share 8 KV heads of dim 128; no window, sink or gate.

Same as the blueprintEvery other block (18 of 36) attends only to a banded window of the last 128 tokens, alternating with the dense blocks; same GQA head layout.

Not used

Same as the blueprintNo output gate. Each head instead has a learned sink logit added to the softmax denominator, so it can assign attention mass to nothing.

Not used

Softmax top-k routerA linear router picks the top 4 of 128 experts; softmax over those 4 logits gives the weights.

Same as the blueprintSigmoid scores computed in FP32 over 256 experts; a per-expert bias (e_score_correction_bias) is added only to choose the top 8, and the 8 unbiased scores are renormalized to sum to 1. The report calls the biases learnable and says they let the auxiliary balance loss be greatly reduced, not removed (the M2 config lists router_aux_loss_coef = 0.001).

Not used

Same as the blueprintDeepSeek-V3-style sequential MTP, each module one Transformer layer. Pretraining used a single module with loss weight 0.3, annealed to 0.1; in the decay phase it was expanded to 3 modules initialized by copying main-model weights, trained alone first and then jointly. The released M2 and M2.7 checkpoints contain no MTP weights.

Same as the blueprintMoE weights were post-trained and are released in MXFP4 (block-scaled 4-bit, about 4.25 bits per parameter), which is what lets the 120b run on one 80 GB GPU. Attention, router, embeddings and head are not quantized.

FP8 block-scaled checkpointOpen weights ship as FP8 E4M3 with 128 × 128 block scales; router, its bias, embeddings and LM head stay in higher precision.

Not used

Same as the blueprintThe 3 MTP modules draft tokens that the main model verifies in one forward pass. During RL they are co-trained with the policy via a top-K KL loss so acceptance stays high as the policy shifts.

Not used

Same as the blueprintNo architectural KV reduction beyond GQA-8: 2,048 values per token per layer in all 62 layers. For RL rollouts, Forge adds a distributed, DFS-backed global KV-cache pool with cost-aware routing to raise prefix-cache hit rates, plus prefill/decode disaggregation.

More comparisons