Skip to content
archdiff

models / compare / inkling-vs-gemma-4

Inkling vs Gemma 4 31B

How Inkling (Thinking Machines Lab, July 2026) and Gemma 4 31B (Google DeepMind, April 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 15 of 29 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ inkling@2026-07+0~5−3

Inkling

Thinking Machines Lab, July 2026, 975B total, 41B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single pre-norm residual stream — Residual & normSingle pre-norm residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kLearned relative-position bias (no RoPE) — AttentionLearned relative-position bias (no RoP…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with log-length scaling — AttentionGlobal GQA with log-length s…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowShort causal convolutions (SConv) — AttentionShort causal convolutions (SConv)Attention⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with shared-expert sink — Mixture of expertsSigmoid router with shared-expert sinkMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ gemma-4@2026-04+2~7−6

Gemma 4 31B

Google DeepMind, April 2026, 30.7B

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioSentencePiece BPE with byte fallback — Input & outputSentencePiece BPE with byte fallbackInput & outputVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream with per-layer scalars — Residual & normSingle residual stream with per-layer…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kRoPE by layer type: full RoPE local, p-RoPE global — AttentionRoPE by layer type: full RoPE local, p…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with keys reused as values — AttentionGlobal GQA with keys reused…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sinkPost-attention RMSNorm (sandwich norm) — Residual & normPost-attention RMSNorm (sandwich norm)Residual & norm⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normDense GeGLU feed-forward — Mixture of expertsDense GeGLU feed-forwardMixture of expertsPost-feed-forward RMSNorm (sandwich norm) — Residual & normPost-feed-forward RMSNorm (sandwich no…Residual & normRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormTied LM head with logit soft-capping — Input & outputTied LM head with logit soft-cappingInput & outputAutoregressive MTP drafter (4 layers, reads trunk KV) — Input & outputAutoregressive MTP drafter (4 layers,…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Inkling and Gemma 4 31B differ on 15 of 29 components.

Blueprint componentInklingGemma 4 31B

Same as the blueprintA 199,998-token byte-level BPE with the same size and pre-tokenizer regex as OpenAI's o200k_base, plus 60 special tokens, many of them reserved, for chat roles, thinking, tool calls and image/audio spans (200,058 ids). The embedding is padded to 201,024 rows.

SentencePiece BPE with byte fallbackGemini's 262,144-entry SentencePiece vocabulary: character-level BPE, split digits, bytes only as a fallback.

Single pre-norm residual streamOne d = 6,144 residual stream through 66 pre-norm blocks; no hyper-connections, branch outputs pass a short conv.

Single residual stream with per-layer scalarsOne d = 5,376 stream, RMSNorm before and after each sublayer, and a stored scalar that rescales the stream after every block.

Learned relative-position bias (no RoPE)Each head adds a bias by backward distance: a query-dependent mix of 16 learned profiles, over 512 (local) or 1,024 (global) tokens.

RoPE by layer type: full RoPE local, p-RoPE globalLocal layers rotate all 256 dims (θ = 10⁴); global layers keep only the 64 fastest of 256 frequency pairs (θ = 10⁶).

Global GQA with log-length scaling11 full-attention layers (every 6th, incl. the last): 64 Q / 8 KV heads, d_h = 128; logits scaled up logarithmically past 128K.

Global GQA with keys reused as valuesEvery sixth layer: full causal attention, 32 query heads and 4 KV heads of dim 512, V taken from the key projection.

Short causal convolutions (SConv)Depthwise causal convs (kernel 4, with skip) on K, V and on both sublayer outputs before they rejoin the residual.

Not used

Sigmoid router with shared-expert sinkScores 256 routed + 2 shared experts; bias-steered top-6; weights normalized jointly over the 8 active experts, then ×8.

Dense GeGLU feed-forwardOne gated MLP per layer, 5,376 → 21,504 → 5,376 with tanh-approximated GELU; all 30.7B parameters are active.

Same as the blueprint256 routed SwiGLU experts of intermediate size 3,072, 6 active per token, in blocks 2–65 (64 MoE layers). Blocks 0 and 1 use a dense SwiGLU MLP of width 24,576 instead.

Not used

Same as the blueprintTwo shared SwiGLU experts of intermediate size 3,072 run on every token. Unlike the blueprint's ungated shared expert, each is weighted by its own router score, normalized jointly with the 6 routed scores. They stay in BF16 in the NVFP4 checkpoint.

Not used

Same as the blueprintUntied unembedding over d = 6,144 with 200,058 rows, the real vocabulary without padding. It stays in BF16 in the NVFP4 checkpoint.

Tied LM head with logit soft-cappingLogits come from the input embedding matrix (no separate head) and are squashed by 30·tanh(z/30).

Same as the blueprint8 MTP layers ship in a separate file (mtp.safetensors). Each concatenates the previous hidden state with the next token's embedding and runs one decoder block with a dense MLP; MTP layers 1 and 3 use global attention, the other six sliding-window. Whether MTP was a pretraining loss is not stated.

Autoregressive MTP drafter (4 layers, reads trunk KV)A separate ~500M-parameter, 4-layer Transformer drafts tokens one at a time while attending to the 31B's own KV cache.

Same as the blueprintMuon for large matrix weights and Adam for all other parameters, with weight decay coupled to the square of the learning rate. Hyperparameter schedules draw on Thinking Machines' modular-manifolds research. Newton–Schulz settings are not published.

Not disclosed

Same as the blueprintContext reaches 1,048,576 tokens without RoPE rescaling: the positional bias is bounded to 1,024 tokens, and global layers apply log-length scaling past 128K positions. The context-extension schedule and data are not published.

Not disclosed

Same as the blueprintA small SFT stage on synthetic data from open-weight models (including Kimi K2.5), then most compute on asynchronous RL in synthetic and human-built environments: over 30M rollouts in two long continuous runs. Effort is set through the system message with a per-token cost during RL. Calibration RL uses proper scoring rules.

Not disclosed

Not present

Post-attention RMSNorm (sandwich norm)The attention output is RMS-normalized with a learned gain before the residual add; there is no output gate or sink.

Not present

Post-feed-forward RMSNorm (sandwich norm)The MLP output is RMS-normalized with a learned gain before it is added to the residual stream.

More comparisons