Skip to content
archdiff

models / compare / kimi-k3-1-pro-vs-kimi-k3

Kimi K3.1 Pro vs Kimi K3

How Kimi K3.1 Pro (Moonshot AI, October 2026) and Kimi K3 (Moonshot AI, July 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 4 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ kimi-k3-1-pro@2026-10+0~5−2

Kimi K3.1 Pro

Moonshot AI, October 2026, 2.8T total, 104B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Block Attention Residuals (AttnRes) — Residual & normBlock Attention Residuals (AttnRes)Residual & normRMSNorm — pre-normRMSNormpre-normQK-Clip and normalized latents; LayerNorm on the indexer key — Residual & normQK-Clip and normalized latents; LayerN…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with Quantile Balancing — Mixture of expertsSigmoid router with Quantile BalancingMixture of expertsStable LatentMoE routed experts (SiTU-GLU) — Mixture of expertsStable LatentMoE routed experts (SiTU-…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsNo MTP layer in the open checkpoint — Input & outputNo MTP layer in the open checkpointInput & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ kimi-k3@2026-07+0~5−3

Kimi K3

Moonshot AI, July 2026, 2.8T total, 104B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Block Attention Residuals (AttnRes) — Residual & normBlock Attention Residuals (AttnRes)Residual & normRMSNorm — pre-normRMSNormpre-normQK-Clip and normalized latents (no per-head QK-Norm) — Residual & normQK-Clip and normalized latents (no per…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with Quantile Balancing — Mixture of expertsSigmoid router with Quantile BalancingMixture of expertsStable LatentMoE routed experts (SiTU-GLU) — Mixture of expertsStable LatentMoE routed experts (SiTU-…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP layer turned EAGLE-3 drafter (not in open weights) — Input & outputMTP layer turned EAGLE-3 drafter (not…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

Component by component

Kimi K3.1 Pro and Kimi K3 differ on 4 of 27 components.

Blueprint componentKimi K3.1 ProKimi K3

QK-Clip and normalized latents; LayerNorm on the indexer keyNo per-head QK-Norm: K2-style weight clipping for MLA, L2-normalized q/k in KDA, a LayerNorm on the DSA indexer key.

QK-Clip and normalized latents (no per-head QK-Norm)No per-head RMSNorm on q and k: MLA logits are held by K2's weight clipping, KDA's q and k are L2-normalized.

Same as the blueprintNew in K3.1: a DSA lightning indexer in each of the 24 MLA layers, each with its own weights (vLLM: no index sharing). 32 heads × 128 dims; queries from the 1,536-dim query latent (1,536 → 4,096), one shared 128-dim key from the hidden state with a LayerNorm, per-head weights from a 7,168 → 32 projection. Scores I = Σₕ wₕ·ReLU(qₕ·k) pick the top 2,048 tokens per query. 7.44M BF16 parameters per layer. DeepSeek-V3.2's indexer has 64 heads. How it was trained is unpublished.

Not used

No MTP layer in the open checkpointThe checkpoint ships without MTP weights; Moonshot has not described a K3.1 drafter, so serving engines bring their own.

MTP layer turned EAGLE-3 drafter (not in open weights)One pretrained MTP layer is fine-tuned into an EAGLE-3-style draft model that reads features from three depths.

Not disclosed

Same as the blueprintMulti-teacher on-policy distillation merges the nine domain × effort experts: for a sampled domain and effort level, the matching teacher scores each student token with the clipped log-ratio clip(sg(log π_teacher − log π_θ), −R_max, R_max), used as a dense reward inside the RL framework. Top-k distillation objectives showed no advantage.

More comparisons