Skip to content
archdiff

models / compare / gemma-4-vs-kimi-k3-1-pro

Gemma 4 31B vs Kimi K3.1 Pro

How Gemma 4 31B (Google DeepMind, April 2026) and Kimi K3.1 Pro (Moonshot AI, October 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 19 of 29 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ gemma-4@2026-04+2~7−6

Gemma 4 31B

Google DeepMind, April 2026, 30.7B

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioSentencePiece BPE with byte fallback — Input & outputSentencePiece BPE with byte fallbackInput & outputVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream with per-layer scalars — Residual & normSingle residual stream with per-layer…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kRoPE by layer type: full RoPE local, p-RoPE global — AttentionRoPE by layer type: full RoPE local, p…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with keys reused as values — AttentionGlobal GQA with keys reused…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sinkPost-attention RMSNorm (sandwich norm) — Residual & normPost-attention RMSNorm (sandwich norm)Residual & norm⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normDense GeGLU feed-forward — Mixture of expertsDense GeGLU feed-forwardMixture of expertsPost-feed-forward RMSNorm (sandwich norm) — Residual & normPost-feed-forward RMSNorm (sandwich no…Residual & normRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormTied LM head with logit soft-capping — Input & outputTied LM head with logit soft-cappingInput & outputAutoregressive MTP drafter (4 layers, reads trunk KV) — Input & outputAutoregressive MTP drafter (4 layers,…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ kimi-k3-1-pro@2026-10+0~5−2

Kimi K3.1 Pro

Moonshot AI, October 2026, 2.8T total, 104B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Block Attention Residuals (AttnRes) — Residual & normBlock Attention Residuals (AttnRes)Residual & normRMSNorm — pre-normRMSNormpre-normQK-Clip and normalized latents; LayerNorm on the indexer key — Residual & normQK-Clip and normalized latents; LayerN…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with Quantile Balancing — Mixture of expertsSigmoid router with Quantile BalancingMixture of expertsStable LatentMoE routed experts (SiTU-GLU) — Mixture of expertsStable LatentMoE routed experts (SiTU-…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsNo MTP layer in the open checkpoint — Input & outputNo MTP layer in the open checkpointInput & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Gemma 4 31B and Kimi K3.1 Pro differ on 19 of 29 components.

Blueprint componentGemma 4 31BKimi K3.1 Pro

SentencePiece BPE with byte fallbackGemini's 262,144-entry SentencePiece vocabulary: character-level BPE, split digits, bytes only as a fallback.

Same as the blueprintUnchanged from K3 (the tiktoken.model file is byte-identical): byte-level BPE with 163,584 merge ranks plus 256 reserved special tokens, 163,840 ids in total. The pre-tokenizer keeps runs of Han characters together and splits digits into groups of 1–3.

Single residual stream with per-layer scalarsOne d = 5,376 stream, RMSNorm before and after each sublayer, and a stored scalar that rescales the stream after every block.

Block Attention Residuals (AttnRes)Each sublayer reads a softmax-weighted mix of the embedding and earlier 12-layer block summaries, as in K3.

Same as the blueprintPer-head RMSNorm with a learned gain on queries and keys (256-dim heads in local layers, 512-dim in global ones), so the logits use scale 1 instead of 1/√d. Values get a gain-free RMSNorm. There is no attention-logit soft-cap.

QK-Clip and normalized latents; LayerNorm on the indexer keyNo per-head QK-Norm: K2-style weight clipping for MLA, L2-normalized q/k in KDA, a LayerNorm on the DSA indexer key.

RoPE by layer type: full RoPE local, p-RoPE globalLocal layers rotate all 256 dims (θ = 10⁴); global layers keep only the 64 fastest of 256 frequency pairs (θ = 10⁶).

Same as the blueprintNoPE throughout. MLA applies no rotary encoding (the 64-dim decoupled key slot is kept but never rotated), and the new indexer sets index_rope_dim = 0, whereas DeepSeek-V3.2's indexer rotates 64 of its 128 dims. KDA's decay and short convolutions carry order, so the 1M window needs no RoPE rescaling or YaRN.

Not used

Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1), unchanged from K3: 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + Swish on q, k, v; L2-normalized q, k. The log-decay is bounded as g = −5·σ(eᴬ·z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

Global GQA with keys reused as valuesEvery sixth layer: full causal attention, 32 query heads and 4 KV heads of dim 512, V taken from the key projection.

Same as the blueprintGated MLA in the same 24 layers as K3 (4, 8, …, 92 plus 93), every projection the same shape: 96 heads, queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer), q/k head dim 192, v head dim 128. The difference is the key set: each query now attends only to the 2,048 tokens its layer's DSA indexer selects, not the full prefix.

Not used

Same as the blueprintNew in K3.1: a DSA lightning indexer in each of the 24 MLA layers, each with its own weights (vLLM: no index sharing). 32 heads × 128 dims; queries from the 1,536-dim query latent (1,536 → 4,096), one shared 128-dim key from the hidden state with a LayerNorm, per-head weights from a 7,168 → 32 projection. Scores I = Σₕ wₕ·ReLU(qₕ·k) pick the top 2,048 tokens per query. 7.44M BF16 parameters per layer. DeepSeek-V3.2's indexer has 64 heads. How it was trained is unpublished.

Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

Not used

Not used

Same as the blueprintBoth mixers keep K3's input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In gated DSA it multiplies the sparse MLA output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. No sink logits.

Post-attention RMSNorm (sandwich norm)The attention output is RMS-normalized with a learned gain before the residual add; there is no output gate or sink.

Not present

Dense GeGLU feed-forwardOne gated MLP per layer, 5,376 → 21,504 → 5,376 with tanh-approximated GELU; all 30.7B parameters are active.

Sigmoid router with Quantile BalancingSigmoid scores pick 16 of 896 experts; each expert's selection bias is set every step from a quantile of routing margins.

Post-feed-forward RMSNorm (sandwich norm)The MLP output is RMS-normalized with a learned gain before it is added to the residual stream.

Not present

Not used

Stable LatentMoE routed experts (SiTU-GLU)896 SiTU-GLU experts run in a 3,584-dim latent (half of d); 16 fire per token; an RMSNorm precedes the up-projection.

Not used

Same as the blueprintTwo full-width shared experts per MoE layer, stored fused as one SiTU-GLU MLP with intermediate size 6,144 (2 × 3,072) at d = 7,168. They read the un-projected input, are ungated and stay in BF16.

Tied LM head with logit soft-cappingLogits come from the input embedding matrix (no separate head) and are squashed by 30·tanh(z/30).

Same as the blueprintUntied 7,168 → 163,840 output projection, kept in BF16 (excluded from MXFP4 quantization).

Autoregressive MTP drafter (4 layers, reads trunk KV)A separate ~500M-parameter, 4-layer Transformer drafts tokens one at a time while attending to the 31B's own KV cache.

No MTP layer in the open checkpointThe checkpoint ships without MTP weights; Moonshot has not described a K3.1 drafter, so serving engines bring their own.

Not disclosed

Same as the blueprintInherited from K3's pretraining: Per-Head Muon, which orthogonalizes each head's block of the Q, K and V momentum separately, plus Kimi K2's weight clipping, a cosine LR schedule with 1% linear warmup and weight decay 0.1. Moonshot has not said which optimizer trained the new indexer or K3.1's own training stages.

Not disclosed

Same as the blueprintThe 1,048,576-token window comes from K3's four-stage curriculum (8K → 64K in pretraining, 256K → 1M in cooldown), with no positional encoding to rescale. DSA now caps each MLA query at 2,048 attended tokens, while the indexer still scores the whole prefix. The context lengths the indexer was trained at are not published.

Not disclosed

Same as the blueprintPost-trained from K3 with large-scale joint RL across hundreds of downstream domains and substantially more post-training compute than K3. For the open weights, cyber-related data was removed from the mid-training corpus and the RL environment mix, which cut xdev-bench-hard pass@3 from 90.5% to 81.0%. Reasoning effort is low, high or max (default max).

More comparisons