Gemma 4 31B vs DeepSeek-V4-Pro

=Same as the blueprintThe DeepSeek-V3 byte-level BPE tokenizer (128K vocabulary, 129,280 rows in the config) with a few added special tokens for context construction. V3's token-splitting and fill-in-the-middle strategies are kept.

=Same as the blueprintA ~550M-parameter ViT (27 layers, width 1,152, MLP 4,304, 16 heads, 16-px patches) with 2D RoPE and 2D absolute position embeddings, kept frozen during pretraining. Its outputs are pooled 3 × 3, so each soft token covers 48 × 48 px, under a per-image budget of 70, 140, 280 (default), 560 or 1,120 tokens. Video enters as frames; the 31B has no audio encoder.

=Same as the blueprintmHC with n = 4 streams of width 7,168 around every attention and MoE sublayer. The residual-mixing matrix is projected to doubly stochastic with 20 Sinkhorn–Knopp iterations; read-in uses σ(·) and write-back 2σ(·). DeepSeek reports a 6.7% wall-time overhead per overlapped 1F1B pipeline stage.

=Same as the blueprintPartial RoPE on the last 64 of 512 dims of queries and KV entries. Because each entry is both key and value, the same 64 dims of every head's output are rotated by −i so each entry's contribution depends on relative distance. All 61 layers are compressed and use θ = 160,000 with YaRN (factor 16 over a 65,536-token base) to reach 1,048,576 positions.

=Same as the blueprintCompressed Sparse Attention in 30 of 61 layers (even layers 2–60), interleaved with HCA. Each KV entry pools 8 overlapping tokens at stride m = 4 with a per-channel softmax. A lightning indexer (64 heads × 128 dims, ReLU scores weighted per head) keeps the top 1,024 entries, which the 128 query heads attend to together with the SWA entries.

=Same as the blueprintNo output gate. Each of the 128 heads has a learnable sink logit whose exponential is added to the softmax denominator in both CSA and HCA, so its attention weights can sum to less than 1.

=Same as the blueprint√softplus affinities over 384 experts with a per-expert bias used only to pick the top 6 (auxiliary-loss-free balancing, bias speed 0.001) plus a 0.0001 sequence-wise balance loss. The selected scores are normalized and scaled by 2.5. V3's node limit is removed, and the first 3 MoE layers pick experts by a fixed token-id hash. Anticipatory Routing (indices from Δt-step-old parameters) was switched on only around loss spikes.

=Same as the blueprint384 SwiGLU experts of intermediate size 3,072, 6 active per token, in all 61 layers. The linear branch is clamped to [−10, 10] and the gate capped at 10. Expert weights went through MXFP4 quantization-aware training in post-training and ship in FP4.

=Same as the blueprintOne always-active shared SwiGLU expert of intermediate size 3,072 in every MoE layer, with the same clamping; unlike the routed experts it is not stored in FP4.

=Same as the blueprintUntied 7,168 → 129,280 output projection, trained with AdamW and shared with the MTP module.

=Same as the blueprintOne MTP module (depth 1), unchanged from DeepSeek-V3: an extra block that combines the normalized next-token embedding with the trunk's normalized state and shares the embedding and LM head. Its attention is sliding-window only. Loss weight 0.3, lowered to 0.1 when learning-rate decay starts.

?Not disclosed

=Same as the blueprintMuon for most matrices: 10 hybrid Newton–Schulz steps (8 with coefficients (3.4445, −4.7750, 2.0315), then 2 with (2, −1.5, 0.5)), Nesterov momentum 0.95, weight decay 0.1, update RMS rescaled to 0.18. AdamW (β = 0.9 / 0.95) trains the embedding, prediction head, RMSNorm gains and mHC static biases and gates. No QK-Clip.

?Not disclosed

=Same as the blueprintSequence length grows 4K → 16K → 64K → 1M during pretraining on 33T tokens. Attention is dense at first (a longer dense stage than Flash's 1T tokens), and sparse CSA starts at 64K after a short indexer warm-up. Packed documents use sample-level attention masking.

?Not disclosed

=Same as the blueprintDomain specialists (mathematics, coding, agent, instruction following and others) are each trained with SFT then GRPO. Hard-to-verify tasks use rubric-guided data scored by a generative reward model that is the actor itself; there is no scalar reward model. Non-think, High and Max effort modes come from different length penalties and context windows in RL.

?Not disclosed

=Same as the blueprintMore than ten specialist teachers are merged into one student by on-policy distillation with a weighted, full-vocabulary reverse-KL loss. Teacher logits are rebuilt on the fly from cached last-layer hidden states instead of being stored.

=Same as the blueprintThe 31B pairs with its released drafter (gemma-4-31B-it-assistant): the drafter proposes several tokens and the 31B verifies them in one forward pass. The drafter's model card claims up to 3× faster decoding at the same output quality. Google's example starts at 4 draft tokens and adapts the count to the acceptance rate.

?Not disclosed