Skip to content
archdiff

models / compare / qwen3-8-vs-inkling

Qwen3.8-2.4T-A95B vs Inkling

How Qwen3.8-2.4T-A95B (Alibaba Qwen, August 2026) and Inkling (Thinking Machines Lab, July 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 9 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ qwen3-8@2026-08+0~3−4

Qwen3.8-2.4T-A95B

Alibaba Qwen, August 2026, 2.4T total, 95B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGated grouped-query attention — AttentionGated grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ inkling@2026-07+0~5−3

Inkling

Thinking Machines Lab, July 2026, 975B total, 41B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single pre-norm residual stream — Residual & normSingle pre-norm residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kLearned relative-position bias (no RoPE) — AttentionLearned relative-position bias (no RoP…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with log-length scaling — AttentionGlobal GQA with log-length s…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowShort causal convolutions (SConv) — AttentionShort causal convolutions (SConv)Attention⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with shared-expert sink — Mixture of expertsSigmoid router with shared-expert sinkMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Qwen3.8-2.4T-A95B and Inkling differ on 9 of 27 components.

Blueprint componentQwen3.8-2.4T-A95BInkling

Not used

Same as the blueprintEncoder-free. Images and video are cut into 40 × 40-pixel patches, 2 frames deep. A 4-layer hierarchical MLP (three linear + RMSNorm + GELU stages that fold space and time into channels, then a linear projection) maps each patch to one 6,144-dim token. 16 kHz audio becomes dMel frames: 80 mel bins, each quantized to 16 levels, embedded by lookup and summed per frame.

Single residual streamOne d = 8,192 pre-norm residual stream with zero-centred RMSNorm gains; no hyper-connections or depth attention.

Single pre-norm residual streamOne d = 6,144 residual stream through 66 pre-norm blocks; no hyper-connections, branch outputs pass a short conv.

Same as the blueprintPartial RoPE in the attention layers only: 64 of 256 head dims rotate (partial_rotary_factor 0.25), θ = 10,000,000. The text-only config drops Qwen3.5's M-RoPE section fields; with text input every rotary frequency sees the same 1D position. The Gated DeltaNet layers carry no explicit position.

Learned relative-position bias (no RoPE)Each head adds a bias by backward distance: a query-dependent mix of 16 learned profiles, over 512 (local) or 1,024 (global) tokens.

Same as the blueprintGated DeltaNet in 69 of 92 layers, three before every attention layer. 16 query/key heads and 128 value heads of dim 128, so each q/k head serves 8 value heads (4 in Qwen3.5). A depthwise causal conv (kernel 4) + SiLU runs over q, k and v; each value head has a scalar decay g = −exp(A_log)·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 128 × 128 × 128 state per sequence, which the config holds in FP32.

Not used

Gated grouped-query attentionEvery fourth layer: full causal GQA with 64 query heads sharing 4 KV heads of dim 256, QK-Norm and a sigmoid output gate.

Global GQA with log-length scaling11 full-attention layers (every 6th, incl. the last): 64 Q / 8 KV heads, d_h = 128; logits scaled up logarithmically past 128K.

Not used

Same as the blueprint55 of 66 layers (all except 5, 11, …, 65) attend to the last 512 tokens with 64 query heads and 16 KV heads of dimension 128. Keys and values pass through a kernel-4 causal convolution before caching, and the relative-position bias spans the whole window.

Same as the blueprintIn the attention layers q_proj emits 2 × 64 × 256 values per token; each head's second 256 values are an elementwise sigmoid gate on its attention output before W_o. The Gated DeltaNet layers gate their output with SiLU(z) after a per-head RMSNorm (output_gate_type: swish). No sink logits.

Short causal convolutions (SConv)Depthwise causal convs (kernel 4, with skip) on K, V and on both sublayer outputs before they rejoin the residual.

Softmax top-k routerA linear router scores 512 experts; softmax, keep the top 10, renormalize. No selection bias.

Sigmoid router with shared-expert sinkScores 256 routed + 2 shared experts; bias-steered top-6; weights normalized jointly over the 8 active experts, then ×8.

Not disclosed

Same as the blueprintMuon for large matrix weights and Adam for all other parameters, with weight decay coupled to the square of the learning rate. Hyperparameter schedules draw on Thinking Machines' modular-manifolds research. Newton–Schulz settings are not published.

More comparisons