Skip to content
archdiff

models / compare / gemma-4-vs-qwen3-8

Gemma 4 31B vs Qwen3.8-2.4T-A95B

How Gemma 4 31B (Google DeepMind, April 2026) and Qwen3.8-2.4T-A95B (Alibaba Qwen, August 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 17 of 29 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ gemma-4@2026-04+2~7−6

Gemma 4 31B

Google DeepMind, April 2026, 30.7B

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioSentencePiece BPE with byte fallback — Input & outputSentencePiece BPE with byte fallbackInput & outputVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream with per-layer scalars — Residual & normSingle residual stream with per-layer…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kRoPE by layer type: full RoPE local, p-RoPE global — AttentionRoPE by layer type: full RoPE local, p…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with keys reused as values — AttentionGlobal GQA with keys reused…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sinkPost-attention RMSNorm (sandwich norm) — Residual & normPost-attention RMSNorm (sandwich norm)Residual & norm⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normDense GeGLU feed-forward — Mixture of expertsDense GeGLU feed-forwardMixture of expertsPost-feed-forward RMSNorm (sandwich norm) — Residual & normPost-feed-forward RMSNorm (sandwich no…Residual & normRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormTied LM head with logit soft-capping — Input & outputTied LM head with logit soft-cappingInput & outputAutoregressive MTP drafter (4 layers, reads trunk KV) — Input & outputAutoregressive MTP drafter (4 layers,…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ qwen3-8@2026-08+0~3−4

Qwen3.8-2.4T-A95B

Alibaba Qwen, August 2026, 2.4T total, 95B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGated grouped-query attention — AttentionGated grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Gemma 4 31B and Qwen3.8-2.4T-A95B differ on 17 of 29 components.

Blueprint componentGemma 4 31BQwen3.8-2.4T-A95B

SentencePiece BPE with byte fallbackGemini's 262,144-entry SentencePiece vocabulary: character-level BPE, split digits, bytes only as a fallback.

Same as the blueprintByte-level BPE (Qwen2Tokenizer), unchanged from Qwen3.5: vocab.json and merges.txt are byte-identical to the Qwen3.5 release (248,044 BPE entries), with the same 33 special tokens (ids 248,044–248,076), padded to 248,320 rows. The pre-tokenizer splits numbers into single digits.

Same as the blueprintA ~550M-parameter ViT (27 layers, width 1,152, MLP 4,304, 16 heads, 16-px patches) with 2D RoPE and 2D absolute position embeddings, kept frozen during pretraining. Its outputs are pooled 3 × 3, so each soft token covers 48 × 48 px, under a per-image budget of 70, 140, 280 (default), 560 or 1,120 tokens. Video enters as frames; the 31B has no audio encoder.

Not used

Single residual stream with per-layer scalarsOne d = 5,376 stream, RMSNorm before and after each sublayer, and a stored scalar that rescales the stream after every block.

Single residual streamOne d = 8,192 pre-norm residual stream with zero-centred RMSNorm gains; no hyper-connections or depth attention.

RoPE by layer type: full RoPE local, p-RoPE globalLocal layers rotate all 256 dims (θ = 10⁴); global layers keep only the 64 fastest of 256 frequency pairs (θ = 10⁶).

Same as the blueprintPartial RoPE in the attention layers only: 64 of 256 head dims rotate (partial_rotary_factor 0.25), θ = 10,000,000. The text-only config drops Qwen3.5's M-RoPE section fields; with text input every rotary frequency sees the same 1D position. The Gated DeltaNet layers carry no explicit position.

Not used

Same as the blueprintGated DeltaNet in 69 of 92 layers, three before every attention layer. 16 query/key heads and 128 value heads of dim 128, so each q/k head serves 8 value heads (4 in Qwen3.5). A depthwise causal conv (kernel 4) + SiLU runs over q, k and v; each value head has a scalar decay g = −exp(A_log)·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 128 × 128 × 128 state per sequence, which the config holds in FP32.

Global GQA with keys reused as valuesEvery sixth layer: full causal attention, 32 query heads and 4 KV heads of dim 512, V taken from the key projection.

Gated grouped-query attentionEvery fourth layer: full causal GQA with 64 query heads sharing 4 KV heads of dim 256, QK-Norm and a sigmoid output gate.

Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

Not used

Not used

Same as the blueprintIn the attention layers q_proj emits 2 × 64 × 256 values per token; each head's second 256 values are an elementwise sigmoid gate on its attention output before W_o. The Gated DeltaNet layers gate their output with SiLU(z) after a per-head RMSNorm (output_gate_type: swish). No sink logits.

Post-attention RMSNorm (sandwich norm)The attention output is RMS-normalized with a learned gain before the residual add; there is no output gate or sink.

Not present

Dense GeGLU feed-forwardOne gated MLP per layer, 5,376 → 21,504 → 5,376 with tanh-approximated GELU; all 30.7B parameters are active.

Softmax top-k routerA linear router scores 512 experts; softmax, keep the top 10, renormalize. No selection bias.

Post-feed-forward RMSNorm (sandwich norm)The MLP output is RMS-normalized with a learned gain before it is added to the residual stream.

Not present

Not used

Same as the blueprint512 SwiGLU experts per layer with intermediate size 2,048 (twice Qwen3.5's) at the full width 8,192, stored as a fused [512, 4,096, 8,192] gate/up tensor; 10 are active per token, about 2% of the pool. Routed experts hold 2.37T of the 2.45T parameters and are the only weights converted in the official FP8 checkpoint.

Not used

Same as the blueprintOne shared SwiGLU expert per layer (intermediate 2,048) runs on every token. Its output is scaled by a learned per-token sigmoid gate σ(w·x), w ∈ ℝ^8192, before it is added to the routed sum, so unlike a DeepSeek-style shared expert it is gated.

Tied LM head with logit soft-cappingLogits come from the input embedding matrix (no separate head) and are squashed by 30·tanh(z/30).

Same as the blueprintUntied 8,192 → 248,320 output projection (~2.03B parameters), also used by the MTP layer; kept in BF16 in the FP8 release.

Autoregressive MTP drafter (4 layers, reads trunk KV)A separate ~500M-parameter, 4-layer Transformer drafts tokens one at a time while attending to the 31B's own KV cache.

Same as the blueprintOne MTP layer (~26.4B parameters): a full gated-attention + 512-expert MoE block. Its input is a linear projection (16,384 → 8,192) of the RMSNorm'd next-token embedding concatenated with the RMSNorm'd trunk state. It reuses the main embedding and LM head (mtp_use_dedicated_embeddings: false). Per the model card it was trained with multiple steps, and vLLM applies the one layer repeatedly to draft several tokens.

Not disclosed

Same as the blueprint262,144 tokens natively. The model card says the model extends to 1,010,000 tokens but, unlike Qwen3.5's card, gives no YaRN recipe. The vLLM recipe it links reaches 1,010,000 by raising max_position_embeddings alone, with no rope scaling. The API model Qwen3.8-Max defaults to 1M. The pretraining context schedule is not published.

Not disclosed

Same as the blueprintQwen's blog describes scaling RL environments and compute together across agent harnesses (QwenWork, Claude Code, Codex, OpenClaw, Hermes). It has three parts. Environments grow along task, workspace and harness axes. One reward system combines execution checks, rubric-based judging of text and rendered visuals, and agentic inspection. An online balancer evens each batch over tasks, difficulty, workspaces and harnesses to cut inter-batch gradient variance. The policy-optimization algorithm is not published.

More comparisons