Skip to content
archdiff

models / compare / qwen3-8-vs-kimi-k3-1-pro

Qwen3.8-2.4T-A95B vs Kimi K3.1 Pro

How Qwen3.8-2.4T-A95B (Alibaba Qwen, August 2026) and Kimi K3.1 Pro (Moonshot AI, October 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 9 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ qwen3-8@2026-08+0~3−4

Qwen3.8-2.4T-A95B

Alibaba Qwen, August 2026, 2.4T total, 95B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGated grouped-query attention — AttentionGated grouped-query attentionAttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSoftmax top-k router — Mixture of expertsSoftmax top-k routerMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ kimi-k3-1-pro@2026-10+0~5−2

Kimi K3.1 Pro

Moonshot AI, October 2026, 2.8T total, 104B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Block Attention Residuals (AttnRes) — Residual & normBlock Attention Residuals (AttnRes)Residual & normRMSNorm — pre-normRMSNormpre-normQK-Clip and normalized latents; LayerNorm on the indexer key — Residual & normQK-Clip and normalized latents; LayerN…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with Quantile Balancing — Mixture of expertsSigmoid router with Quantile BalancingMixture of expertsStable LatentMoE routed experts (SiTU-GLU) — Mixture of expertsStable LatentMoE routed experts (SiTU-…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsNo MTP layer in the open checkpoint — Input & outputNo MTP layer in the open checkpointInput & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Qwen3.8-2.4T-A95B and Kimi K3.1 Pro differ on 9 of 27 components.

Blueprint componentQwen3.8-2.4T-A95BKimi K3.1 Pro

Not used

Same as the blueprintMoonViT-V2, as in K3: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch jointly with the language model. Images and video share weights; a 2×2 pixel shuffle cuts tokens 4× before an MLP projector to 7,168 dims. The model card lists text and image input; Moonshot's API docs also list video.

Single residual streamOne d = 8,192 pre-norm residual stream with zero-centred RMSNorm gains; no hyper-connections or depth attention.

Block Attention Residuals (AttnRes)Each sublayer reads a softmax-weighted mix of the embedding and earlier 12-layer block summaries, as in K3.

Same as the blueprintPer-head RMSNorm (zero-centred gain) over the 256-dim heads on q and k in the 23 gated-attention layers, applied before RoPE. The Gated DeltaNet layers instead L2-normalize q and k inside the kernel.

QK-Clip and normalized latents; LayerNorm on the indexer keyNo per-head QK-Norm: K2-style weight clipping for MLA, L2-normalized q/k in KDA, a LayerNorm on the DSA indexer key.

Gated grouped-query attentionEvery fourth layer: full causal GQA with 64 query heads sharing 4 KV heads of dim 256, QK-Norm and a sigmoid output gate.

Same as the blueprintGated MLA in the same 24 layers as K3 (4, 8, …, 92 plus 93), every projection the same shape: 96 heads, queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer), q/k head dim 192, v head dim 128. The difference is the key set: each query now attends only to the 2,048 tokens its layer's DSA indexer selects, not the full prefix.

Not used

Same as the blueprintNew in K3.1: a DSA lightning indexer in each of the 24 MLA layers, each with its own weights (vLLM: no index sharing). 32 heads × 128 dims; queries from the 1,536-dim query latent (1,536 → 4,096), one shared 128-dim key from the hidden state with a LayerNorm, per-head weights from a 7,168 → 32 projection. Scores I = Σₕ wₕ·ReLU(qₕ·k) pick the top 2,048 tokens per query. 7.44M BF16 parameters per layer. DeepSeek-V3.2's indexer has 64 heads. How it was trained is unpublished.

Softmax top-k routerA linear router scores 512 experts; softmax, keep the top 10, renormalize. No selection bias.

Sigmoid router with Quantile BalancingSigmoid scores pick 16 of 896 experts; each expert's selection bias is set every step from a quantile of routing margins.

Same as the blueprint512 SwiGLU experts per layer with intermediate size 2,048 (twice Qwen3.5's) at the full width 8,192, stored as a fused [512, 4,096, 8,192] gate/up tensor; 10 are active per token, about 2% of the pool. Routed experts hold 2.37T of the 2.45T parameters and are the only weights converted in the official FP8 checkpoint.

Stable LatentMoE routed experts (SiTU-GLU)896 SiTU-GLU experts run in a 3,584-dim latent (half of d); 16 fire per token; an RMSNorm precedes the up-projection.

Same as the blueprintOne MTP layer (~26.4B parameters): a full gated-attention + 512-expert MoE block. Its input is a linear projection (16,384 → 8,192) of the RMSNorm'd next-token embedding concatenated with the RMSNorm'd trunk state. It reuses the main embedding and LM head (mtp_use_dedicated_embeddings: false). Per the model card it was trained with multiple steps, and vLLM applies the one layer repeatedly to draft several tokens.

No MTP layer in the open checkpointThe checkpoint ships without MTP weights; Moonshot has not described a K3.1 drafter, so serving engines bring their own.

Not disclosed

Same as the blueprintInherited from K3's pretraining: Per-Head Muon, which orthogonalizes each head's block of the Q, K and V momentum separately, plus Kimi K2's weight clipping, a cosine LR schedule with 1% linear warmup and weight decay 0.1. Moonshot has not said which optimizer trained the new indexer or K3.1's own training stages.

More comparisons