Skip to content
archdiff

models / compare / qwen3-8-vs-gpt-oss-120b

Qwen3.8-2.4T-A95B vs gpt-oss-120b

How Qwen3.8-2.4T-A95B (Alibaba Qwen, August 2026) and gpt-oss-120b (OpenAI, August 2025) are built, drawn on the same November 2026 blueprint. They make different choices on 9 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ qwen3-8@2026-08+0~3−4

Qwen3.8-2.4T-A95B

Alibaba Qwen, August 2026, 2.4T total, 95B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGated grouped-query attention — AttentionGated grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ gpt-oss-120b@2025-08+0~4−9

gpt-oss-120b

OpenAI, August 2025, 117B total, 5.1B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kFull RoPE with YaRN scaling — AttentionFull RoPE with YaRN scalingAttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersDense grouped-query attention — AttentionDense grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Qwen3.8-2.4T-A95B and gpt-oss-120b differ on 9 of 27 components.

Blueprint componentQwen3.8-2.4T-A95Bgpt-oss-120b

Same as the blueprintPer-head RMSNorm (zero-centred gain) over the 256-dim heads on q and k in the 23 gated-attention layers, applied before RoPE. The Gated DeltaNet layers instead L2-normalize q and k inside the kernel.

Not used

Same as the blueprintPartial RoPE in the attention layers only: 64 of 256 head dims rotate (partial_rotary_factor 0.25), θ = 10,000,000. The text-only config drops Qwen3.5's M-RoPE section fields; with text input every rotary frequency sees the same 1D position. The Gated DeltaNet layers carry no explicit position.

Full RoPE with YaRN scalingRotary embeddings on all 64 head dims, θ = 150,000, stretched 32× with YaRN.

Same as the blueprintGated DeltaNet in 69 of 92 layers, three before every attention layer. 16 query/key heads and 128 value heads of dim 128, so each q/k head serves 8 value heads (4 in Qwen3.5). A depthwise causal conv (kernel 4) + SiLU runs over q, k and v; each value head has a scalar decay g = −exp(A_log)·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 128 × 128 × 128 state per sequence, which the config holds in FP32.

Not used

Gated grouped-query attentionEvery fourth layer: full causal GQA with 64 query heads sharing 4 KV heads of dim 256, QK-Norm and a sigmoid output gate.

Dense grouped-query attentionFull causal attention with 64 query heads sharing 8 KV heads, in every other block.

Not used

Same as the blueprintEvery other block (18 of 36) attends only to a banded window of the last 128 tokens, alternating with the dense blocks; same GQA head layout.

Same as the blueprintOne shared SwiGLU expert per layer (intermediate 2,048) runs on every token. Its output is scaled by a learned per-token sigmoid gate σ(w·x), w ∈ ℝ^8192, before it is added to the routed sum, so unlike a DeepSeek-style shared expert it is gated.

Not used

Same as the blueprintOne MTP layer (~26.4B parameters): a full gated-attention + 512-expert MoE block. Its input is a linear projection (16,384 → 8,192) of the RMSNorm'd next-token embedding concatenated with the RMSNorm'd trunk state. It reuses the main embedding and LM head (mtp_use_dedicated_embeddings: false). Per the model card it was trained with multiple steps, and vLLM applies the one layer repeatedly to draft several tokens.

Not used

Same as the blueprintThe serving recipes linked from the model card use MTP self-speculation. vLLM runs method mtp with 3 speculative tokens, which its recipe measures at roughly 2.3× per-user output rate; it calls MTP-1 (64.8% acceptance) not worth it. SGLang runs NEXTN with 3 steps, top-k 1 and 4 draft tokens. SGLang also offers DSpark with a separately trained third-party draft model (RadixArk).

Not used

Same as the blueprintQwen publishes no KV compression; the saving is architectural. Only the 23 attention layers cache K and V: 4 KV heads × 256 dims × 2 × 23 = 47,104 values per token (92 KiB in BF16). The 69 Gated DeltaNet layers each hold a fixed 128 × 128 × 128 state (8 MiB in FP32, 552 MiB per sequence). SGLang's cookbook calls this state, not the KV cache, the scarce resource. vLLM's low-latency recipes add an FP8 KV cache.

Not used

More comparisons