Skip to content
archdiff

models / compare / deepseek-v4-1-flash-vs-gpt-oss-120b

DeepSeek-V4.1-Flash vs gpt-oss-120b

How DeepSeek-V4.1-Flash (DeepSeek, September 2026) and gpt-oss-120b (OpenAI, August 2025) are built, drawn on the same November 2026 blueprint. They make different choices on 14 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ deepseek-v4-1-flash@2026-09+0~3−1

DeepSeek-V4.1-Flash

DeepSeek, September 2026, 552B + 196B Engram total, 16B (8B in prefill) active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersCED decoder attention (CSA2 over encoder-projected KV) — AttentionCED decoder attention (CSA2…AttentionCSA2 encoder layers (2× compression, Full + Reuse) — AttentionCSA2 encoder layers (2× comp…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsDSpark drafter (trained after pretraining) — Input & outputDSpark drafter (trained after pretrain…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ gpt-oss-120b@2025-08+0~4−9

gpt-oss-120b

OpenAI, August 2025, 117B total, 5.1B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kFull RoPE with YaRN scaling — AttentionFull RoPE with YaRN scalingAttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersDense grouped-query attention — AttentionDense grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

DeepSeek-V4.1-Flash and gpt-oss-120b differ on 14 of 27 components.

Blueprint componentDeepSeek-V4.1-Flashgpt-oss-120b

Same as the blueprintDeepSeek-ViT, trained from scratch: 32 layers, width 1,024, 16 heads, 14-px patches embedded by a linear projection, 2D-RoPE, RMSNorm and SwiGLU. A 3×3 pixel-unshuffle cuts visual tokens 9× before a 2-layer MLP projector maps them to 5,120 dims. Images only, up to about 1344×1344 px.

Not used

Same as the blueprintTwo Engram modules (196B parameters, FP8) at layers 1 and 14: n-gram orders 2–4, 8 hash heads per order, 2,048 dims per order, about 16M rows per head table. A sigmoid gate on each mHC stream's normalized dot product with its own key adds a shared value. The short causal conv was dropped. Rows are prefetched from host memory over RDMA.

Not used

Same as the blueprintSingle-Pass mHC: n = 4 streams of width 5,120 and 20 Sinkhorn iterations as in V4, but each sublayer reads its input with the mixing coefficients predicted by the previous sublayer. Removing that dependency lets the Mega-mHC kernel read and write the residual once, halving activation traffic, with negligible quality loss reported.

Single residual streamOne d = 2,880 residual stream with pre-norm sublayers; no hyper-connections.

Same as the blueprintRMSNorm on the single 512-dim KV entry, which the report uses to bound every FP4 KV channel to about √512 ≈ 22.6, and on the 1,280-dim query latent. Unlike V4's reference code, V4.1's applies no separate per-head RMS normalization to queries.

Not used

Same as the blueprintPartial RoPE on the last 64 of 512 dims, with the inverse rotation on each head's output as in V4. CSA2 layers use θ = 160,000 with YaRN (factor 16 over 65,536 tokens); the two pure-SWA layers use θ = 10,000 without YaRN. The FP4 KV cache is quantized after RoPE.

Full RoPE with YaRN scalingRotary embeddings on all 64 head dims, θ = 150,000, stretched 32× with YaRN.

CED decoder attention (CSA2 over encoder-projected KV)Layers 20–39 take uncompressed global KV from the encoder's last hidden state and re-select with a hierarchical indexer.

Dense grouped-query attentionFull causal attention with 64 query heads sharing 8 KV heads, in every other block.

CSA2 encoder layers (2× compression, Full + Reuse)Layers 2–19: KV pooled 2→1 and computed in only 3 of 18 layers; the rest reuse that KV and its top-512 selection.

Not used

Same as the blueprint√softplus scores over 384 experts, top-6, with auxiliary-loss-free bias balancing kept separately for text and image tokens (each bias set updated from its own modality's load, speed 0.001) plus a 0.0001 sequence-level loss. Selected scores are normalized and scaled by 1.5. V4's hash-routed early layers are gone.

Softmax top-k routerA linear router picks the top 4 of 128 experts; softmax over those 4 logits gives the weights.

Same as the blueprintOne always-active shared SwiGLU expert of intermediate size 2,304 in every MoE layer.

Not used

DSpark drafter (trained after pretraining)Three 128-token-window blocks draft 5 tokens per pass from the backbone's last 3 layers; no MTP loss in pretraining.

Not used

Same as the blueprintMuon (momentum 0.95, weight decay 0.1, update RMS 0.18) for backbone linear weights, Engram projections and the vision projector, applied per head to query and key weights. Embeddings, the head and Engram tables use a momentum update with Sinkhorn row/column balancing (Engram learning rate 5×); AdamW covers norms and other non-matrix parameters.

Not disclosed

Same as the blueprintThe final stage is full-vocabulary on-policy distillation over all domains from more than 40 teachers, which may differ architecturally from the student, run with asynchronous generation.

Not disclosed

Same as the blueprintServed with the DSpark drafter: 5 draft tokens per pass, with the verification length chosen per request from predicted acceptance and profiled throughput under current load. DSpark also accelerates RL and OPD rollouts.

Not used

Same as the blueprintGlobal KV is 890 bytes per token (always in HBM), about 1/4 of V4-Flash's, from CSA2 sharing plus FP4 main KV. The persistent prefix cache (SSD, kept at least 72 h) holds only global KV, about 1/8 of V4-Flash's footprint; SWA KV sits in a minutes-TTL pool using 10% of host DRAM and is rebuilt by bounded replay on a miss.

Not used

More comparisons