Gemma 4 31B is the largest model in Google DeepMind's Gemma 4 family and a dense decoder: 60 layers of width 5,376, 30.7B parameters plus a ~550M-parameter vision encoder. Fifty layers attend over a 1,024-token sliding window and ten are global, in a fixed pattern of five local then one global that ends on a global layer. The global layers carry the long-context design: 512-dim heads, 4 KV heads, keys reused as values (there is no value projection) and p-RoPE, which keeps only the 64 highest-frequency rotary pairs (128 of 512 dims). Local layers use full RoPE. The report credits these choices with a global KV cache up to 37.5% smaller.
The rest of the block is Gemma's own recipe: RMSNorm both before and after every sublayer, QK-norm with an attention scale of 1 plus a gain-free norm on values, a 21,504-wide GeGLU MLP, and a stored per-layer scalar that rescales the residual stream. The 262,144-entry embedding is tied to the output head, whose logits are soft-capped at 30. A separate 4-layer drafter that reads the trunk's KV cache provides speculative decoding. There is no MoE, output gate, attention sink, linear or sparse attention, or widened residual stream.
The smaller variants differ in more than width. E2B (35 layers, 4:1 local/global) and E4B (42 layers, 5:1) use 512-token windows, keep separate K and V, and add two on-device mechanisms. Per-layer embeddings give every layer its own 256-dim token embedding (2.34B and 2.82B lookup parameters, hence "effective" 2.3B and 4.5B). Cross-layer KV sharing makes the last 20 of 35 (E2B) or 18 of 42 (E4B) layers reuse the KV of the last earlier layer of the same type, and E2B doubles the MLP width in those layers. Both carry a 150M vision and a 305M audio encoder. The 26B-A4B (30 layers, 3.8B active) runs a dense 2,112-wide MLP beside 128 experts (top-8, 704 wide), which the model card counts as one shared expert. The 12B Unified, added after launch, drops the encoders and projects raw 48 × 48 image patches and 40 ms audio chunks straight into the embedding space.