Skip to content
archdiff

models / compare / gpt-oss-120b-vs-qwen3-5

gpt-oss-120b vs Qwen3.5-397B-A17B

How gpt-oss-120b (OpenAI, August 2025) and Qwen3.5-397B-A17B (Alibaba Qwen, February 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 10 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ gpt-oss-120b@2025-08+0~4−9

gpt-oss-120b

OpenAI, August 2025, 117B total, 5.1B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kFull RoPE with YaRN scaling — AttentionFull RoPE with YaRN scalingAttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersDense grouped-query attention — AttentionDense grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ qwen3-5@2026-02+0~3−3

Qwen3.5-397B-A17B

Alibaba Qwen, February 2026, 397B total, 17B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGated grouped-query attention — AttentionGated grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

gpt-oss-120b and Qwen3.5-397B-A17B differ on 10 of 27 components.

Blueprint componentgpt-oss-120bQwen3.5-397B-A17B

Not used

Same as the blueprintA 27-layer ViT (width 1,152, 16 heads, MLP 4,304) with 16-px patches; video frames are patched in pairs (temporal patch 2). A 2×2 spatial merge and an MLP project into d = 4,096, and the DeepStack multi-layer injection is disabled (empty index list). Trained by early fusion on trillions of multimodal tokens.

Not used

Same as the blueprintPer-head RMSNorm (zero-centred gain) over the 256-dim heads on q and k in the 15 gated-attention layers, applied before RoPE. The Gated DeltaNet layers instead L2-normalize q and k inside the kernel.

Full RoPE with YaRN scalingRotary embeddings on all 64 head dims, θ = 150,000, stretched 32× with YaRN.

Same as the blueprintPartial, interleaved multimodal RoPE (M-RoPE) in the attention layers only: 64 of 256 head dims rotate (partial_rotary_factor 0.25), θ = 10,000,000, with the 32 frequency pairs split 11/11/10 between time, height and width. The Gated DeltaNet layers carry no explicit position.

Not used

Same as the blueprintGated DeltaNet in 45 of 60 layers, three before every attention layer. 16 query/key heads and 64 value heads of dim 128 (each q/k head serves 4 value heads); short causal conv (kernel 4) + SiLU on q, k, v; per-head scalar decay g = −eᴬ·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 64 × 128 × 128 state per sequence, held in FP32.

Dense grouped-query attentionFull causal attention with 64 query heads sharing 8 KV heads, in every other block.

Gated grouped-query attentionEvery fourth layer: full causal GQA with 32 query heads sharing 2 KV heads of dim 256, QK-Norm and a sigmoid output gate.

Same as the blueprintEvery other block (18 of 36) attends only to a banded window of the last 128 tokens, alternating with the dense blocks; same GQA head layout.

Not used

Not used

Same as the blueprintOne shared SwiGLU expert per layer (intermediate 1,024) runs on every token, but its output is scaled by a learned per-token sigmoid gate σ(w·x), w ∈ ℝ^4096, before it is added to the routed sum. Unlike DeepSeek-style shared experts it is not ungated.

Not used

Same as the blueprintOne MTP layer: a full gated-attention + 512-expert MoE block whose input is a linear projection (8,192 → 4,096) of the RMSNorm'd next-token embedding concatenated with the RMSNorm'd trunk state. It reuses the main embedding and LM head and, per the model card, was trained with multiple steps, so the one layer is applied repeatedly to draft several tokens.

Not used

Same as the blueprintMTP self-speculation is the recommended serving mode: SGLang NEXTN with 3 steps and 4 draft tokens, or vLLM qwen3_next_mtp with 2 speculative tokens. Qwen's RL rollout engine also uses speculative decoding.

Not used

Same as the blueprintNo KV quantization or tiered cache is published; the saving is architectural. Only the 15 attention layers cache K and V: 2 KV heads × 256 dims × 2 × 15 = 15,360 values per token (30 KiB in BF16). The 45 Gated DeltaNet layers each hold a fixed 64 × 128 × 128 FP32 state (4 MiB per layer, ~180 MiB per sequence) regardless of length.

More comparisons