Skip to content
archdiff

models / compare / deepseek-v4-vs-inkling

DeepSeek-V4-Pro vs Inkling

How DeepSeek-V4-Pro (DeepSeek, April 2026) and Inkling (Thinking Machines Lab, July 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 9 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ deepseek-v4@2026-04+0~1−3

DeepSeek-V4-Pro

DeepSeek, April 2026, 1.6T total, 49B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersHeavily Compressed Attention (HCA) — AttentionHeavily Compressed Attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ inkling@2026-07+0~5−3

Inkling

Thinking Machines Lab, July 2026, 975B total, 41B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single pre-norm residual stream — Residual & normSingle pre-norm residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kLearned relative-position bias (no RoPE) — AttentionLearned relative-position bias (no RoP…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with log-length scaling — AttentionGlobal GQA with log-length s…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowShort causal convolutions (SConv) — AttentionShort causal convolutions (SConv)Attention⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with shared-expert sink — Mixture of expertsSigmoid router with shared-expert sinkMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

DeepSeek-V4-Pro and Inkling differ on 9 of 27 components.

Blueprint componentDeepSeek-V4-ProInkling

Not used

Same as the blueprintEncoder-free. Images and video are cut into 40 × 40-pixel patches, 2 frames deep. A 4-layer hierarchical MLP (three linear + RMSNorm + GELU stages that fold space and time into channels, then a linear projection) maps each patch to one 6,144-dim token. 16 kHz audio becomes dMel frames: 80 mel bins, each quantized to 16 levels, embedded by lookup and summed per frame.

Same as the blueprintmHC with n = 4 streams of width 7,168 around every attention and MoE sublayer. The residual-mixing matrix is projected to doubly stochastic with 20 Sinkhorn–Knopp iterations; read-in uses σ(·) and write-back 2σ(·). DeepSeek reports a 6.7% wall-time overhead per overlapped 1F1B pipeline stage.

Single pre-norm residual streamOne d = 6,144 residual stream through 66 pre-norm blocks; no hyper-connections, branch outputs pass a short conv.

Same as the blueprintPartial RoPE on the last 64 of 512 dims of queries and KV entries. Because each entry is both key and value, the same 64 dims of every head's output are rotated by −i so each entry's contribution depends on relative distance. All 61 layers are compressed and use θ = 160,000 with YaRN (factor 16 over a 65,536-token base) to reach 1,048,576 positions.

Learned relative-position bias (no RoPE)Each head adds a bias by backward distance: a query-dependent mix of 16 learned profiles, over 512 (local) or 1,024 (global) tokens.

Heavily Compressed Attention (HCA)Every 128 tokens pooled into one KV entry; dense shared-KV attention over all of them plus a 128-token window.

Global GQA with log-length scaling11 full-attention layers (every 6th, incl. the last): 64 Q / 8 KV heads, d_h = 128; logits scaled up logarithmically past 128K.

Same as the blueprintCompressed Sparse Attention in 30 of 61 layers (even layers 2–60), interleaved with HCA. Each KV entry pools 8 overlapping tokens at stride m = 4 with a per-channel softmax. A lightning indexer (64 heads × 128 dims, ReLU scores weighted per head) keeps the top 1,024 entries, which the 128 query heads attend to together with the SWA entries.

Not used

Same as the blueprintNo output gate. Each of the 128 heads has a learnable sink logit whose exponential is added to the softmax denominator in both CSA and HCA, so its attention weights can sum to less than 1.

Short causal convolutions (SConv)Depthwise causal convs (kernel 4, with skip) on K, V and on both sublayer outputs before they rejoin the residual.

Same as the blueprint√softplus affinities over 384 experts with a per-expert bias used only to pick the top 6 (auxiliary-loss-free balancing, bias speed 0.001) plus a 0.0001 sequence-wise balance loss. The selected scores are normalized and scaled by 2.5. V3's node limit is removed, and the first 3 MoE layers pick experts by a fixed token-id hash. Anticipatory Routing (indices from Δt-step-old parameters) was switched on only around loss spikes.

Sigmoid router with shared-expert sinkScores 256 routed + 2 shared experts; bias-steered top-6; weights normalized jointly over the 8 active experts, then ×8.

Same as the blueprintMore than ten specialist teachers are merged into one student by on-policy distillation with a weighted, full-vocabulary reverse-KL loss. Teacher logits are rebuilt on the fly from cached last-layer hidden states instead of being stored.

Not disclosed

Not disclosed

Same as the blueprintThe 8 MTP layers act as drafters for speculative decoding, verified by the full model so outputs match plain decoding. Thinking Machines has not published acceptance rates.

More comparisons