Skip to content
archdiff

models / compare / inkling-vs-kimi-k3

Inkling vs Kimi K3

How Inkling (Thinking Machines Lab, July 2026) and Kimi K3 (Moonshot AI, July 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 11 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ inkling@2026-07+0~5−3

Inkling

Thinking Machines Lab, July 2026, 975B total, 41B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single pre-norm residual stream — Residual & normSingle pre-norm residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kLearned relative-position bias (no RoPE) — AttentionLearned relative-position bias (no RoP…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with log-length scaling — AttentionGlobal GQA with log-length s…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowShort causal convolutions (SConv) — AttentionShort causal convolutions (SConv)Attention⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with shared-expert sink — Mixture of expertsSigmoid router with shared-expert sinkMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ kimi-k3@2026-07+0~5−3

Kimi K3

Moonshot AI, July 2026, 2.8T total, 104B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Block Attention Residuals (AttnRes) — Residual & normBlock Attention Residuals (AttnRes)Residual & normRMSNorm — pre-normRMSNormpre-normQK-Clip and normalized latents (no per-head QK-Norm) — Residual & normQK-Clip and normalized latents (no per…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with Quantile Balancing — Mixture of expertsSigmoid router with Quantile BalancingMixture of expertsStable LatentMoE routed experts (SiTU-GLU) — Mixture of expertsStable LatentMoE routed experts (SiTU-…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP layer turned EAGLE-3 drafter (not in open weights) — Input & outputMTP layer turned EAGLE-3 drafter (not…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

Component by component

Inkling and Kimi K3 differ on 11 of 27 components.

Blueprint componentInklingKimi K3

Single pre-norm residual streamOne d = 6,144 residual stream through 66 pre-norm blocks; no hyper-connections, branch outputs pass a short conv.

Block Attention Residuals (AttnRes)Each sublayer reads a softmax-weighted mix of the embedding and earlier 12-layer block summaries, not one running sum.

Same as the blueprintPer-head RMSNorm on q and k (head dim 128) in all 66 layers, applied to keys after their short convolution. Because q and k are normalized, the softmax scale is 1/d_h = 1/128 instead of 1/√d_h.

QK-Clip and normalized latents (no per-head QK-Norm)No per-head RMSNorm on q and k: MLA logits are held by K2's weight clipping, KDA's q and k are L2-normalized.

Learned relative-position bias (no RoPE)Each head adds a bias by backward distance: a query-dependent mix of 16 learned profiles, over 512 (local) or 1,024 (global) tokens.

Same as the blueprintNoPE throughout. The 24 MLA layers apply no rotary or other positional encoding (the 64-dim decoupled key slot of MLA is kept but never rotated), and KDA's decay and short convolutions carry order. Context therefore extends to 1M tokens without RoPE rescaling or YaRN.

Not used

Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1): 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + SiLU on q, k, v; L2-normalized q, k. K3 bounds the per-channel log-decay as g = −5·σ(eᴬ·z) instead of Kimi Linear's −eᴬ·softplus(z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

Global GQA with log-length scaling11 full-attention layers (every 6th, incl. the last): 64 Q / 8 KV heads, d_h = 128; logits scaled up logarithmically past 128K.

Same as the blueprintGated MLA in 24 layers: every fourth layer (4, 8, …, 92) plus layer 93. 96 heads; queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer); q/k head dim 192, v head dim 128. The attention output is kept in FP32 during training to avoid flash attention's biased rounding.

Same as the blueprint55 of 66 layers (all except 5, 11, …, 65) attend to the last 512 tokens with 64 query heads and 16 KV heads of dimension 128. Keys and values pass through a kernel-4 causal convolution before caching, and the relative-position bias spans the whole window.

Not used

Short causal convolutions (SConv)Depthwise causal convs (kernel 4, with skip) on K, V and on both sublayer outputs before they rejoin the residual.

Same as the blueprintBoth mixers end in an input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In MLA it multiplies the attention output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. K3 replaced Kimi Linear's low-rank KDA gate with this full-rank one. No sink logits.

Sigmoid router with shared-expert sinkScores 256 routed + 2 shared experts; bias-steered top-6; weights normalized jointly over the 8 active experts, then ×8.

Sigmoid router with Quantile BalancingSigmoid scores pick 16 of 896 experts; each expert's selection bias is reset every step from a quantile of routing margins.

Same as the blueprint256 routed SwiGLU experts of intermediate size 3,072, 6 active per token, in blocks 2–65 (64 MoE layers). Blocks 0 and 1 use a dense SwiGLU MLP of width 24,576 instead.

Stable LatentMoE routed experts (SiTU-GLU)896 SiTU-GLU experts run in a 3,584-dim latent (half of d); 16 fire per token; an RMSNorm precedes the up-projection.

Same as the blueprint8 MTP layers ship in a separate file (mtp.safetensors). Each concatenates the previous hidden state with the next token's embedding and runs one decoder block with a dense MLP; MTP layers 1 and 3 use global attention, the other six sliding-window. Whether MTP was a pretraining loss is not stated.

MTP layer turned EAGLE-3 drafter (not in open weights)One pretrained MTP layer is fine-tuned into an EAGLE-3-style draft model that reads features from three depths.

Not disclosed

Same as the blueprintMulti-teacher on-policy distillation merges the nine domain × effort experts: for a sampled domain and effort level, the matching teacher scores each student token with the clipped log-ratio clip(sg(log π_teacher − log π_θ), −R_max, R_max), used as a dense reward inside the RL framework. Top-k distillation objectives showed no advantage.

More comparisons