Skip to content
archdiff

models / compare / kimi-k3-vs-minimax-m3

Kimi K3 vs MiniMax-M3

How Kimi K3 (Moonshot AI, July 2026) and MiniMax-M3 (MiniMax, June 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 13 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ kimi-k3@2026-07+0~5−3

Kimi K3

Moonshot AI, July 2026, 2.8T total, 104B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Block Attention Residuals (AttnRes) — Residual & normBlock Attention Residuals (AttnRes)Residual & normRMSNorm — pre-normRMSNormpre-normQK-Clip and normalized latents (no per-head QK-Norm) — Residual & normQK-Clip and normalized latents (no per…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with Quantile Balancing — Mixture of expertsSigmoid router with Quantile BalancingMixture of expertsStable LatentMoE routed experts (SiTU-GLU) — Mixture of expertsStable LatentMoE routed experts (SiTU-…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP layer turned EAGLE-3 drafter (not in open weights) — Input & outputMTP layer turned EAGLE-3 drafter (not…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ minimax-m3@2026-06+0~4−4

MiniMax-M3

MiniMax, June 2026, 427.0B total, 23.5B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in blocks 0–2 — AttentionFull grouped-query attention…AttentionMiniMax Sparse Attention (MSA) — AttentionMiniMax Sparse Attention (MS…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headBF16 weights plus an official MXFP8 checkpoint — Training & inferenceBF16 weights plus an official MXFP8 ch…Training & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Kimi K3 and MiniMax-M3 differ on 13 of 27 components.

Blueprint componentKimi K3MiniMax-M3

Block Attention Residuals (AttnRes)Each sublayer reads a softmax-weighted mix of the embedding and earlier 12-layer block summaries, not one running sum.

Single residual streamOne d = 6,144 residual stream with pre-norm sublayers; no hyper-connections.

QK-Clip and normalized latents (no per-head QK-Norm)No per-head RMSNorm on q and k: MLA logits are held by K2's weight clipping, KDA's q and k are L2-normalized.

Same as the blueprintPer-head Gemma-style RMSNorm (ε = 1e-6) on q and k before RoPE, with one 128-dim gain shared by all 64 query heads and one by the 4 key heads; the indexer's queries and key get their own. This replaces M2's layer-wide norm over the concatenated projections (qk_norm_type per_layer → per_head), so head-sharded tensor parallelism no longer needs a cross-rank reduction for it.

Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1): 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + SiLU on q, k, v; L2-normalized q, k. K3 bounds the per-channel log-decay as g = −5·σ(eᴬ·z) instead of Kimi Linear's −eᴬ·softplus(z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

Not used

Same as the blueprintGated MLA in 24 layers: every fourth layer (4, 8, …, 92) plus layer 93. 96 heads; queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer); q/k head dim 192, v head dim 128. The attention output is kept in FP32 during training to avoid flash attention's biased rounding.

Full grouped-query attention in blocks 0–2The first three blocks attend over the whole context: 64 query heads share 4 KV heads of dim 128; no indexer, gate or sink.

Not used

MiniMax Sparse Attention (MSA)Per GQA group, a 2-projection indexer picks the top 16 of 128-token KV blocks; exact softmax over ≤ 2,048 keys, in 57 of 60 layers.

Same as the blueprintBoth mixers end in an input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In MLA it multiplies the attention output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. K3 replaced Kimi Linear's low-rank KDA gate with this full-rank one. No sink logits.

Not used

Sigmoid router with Quantile BalancingSigmoid scores pick 16 of 896 experts; each expert's selection bias is reset every step from a quantile of routing margins.

Same as the blueprintAn FP32 linear router scores 128 experts with sigmoids. A per-expert bias (e_score_correction_bias, FP32) is added only to choose the top 4, the 4 unbiased scores are renormalized to sum to 1, and the routed output is multiplied by 2.0 (routed_scaling_factor) before the shared expert is added. MiniMax has not described how M3's biases are updated or whether an auxiliary balance loss is used.

Stable LatentMoE routed experts (SiTU-GLU)896 SiTU-GLU experts run in a 3,584-dim latent (half of d); 16 fire per token; an RMSNorm precedes the up-projection.

Same as the blueprint128 experts of intermediate size 3,072, top-4 active, in blocks 3–59 (M2: 256 of width 1,536, top-8). Blocks 0–2 use a dense MLP of width 12,288 instead. Every FFN uses gpt-oss's clamped SwiGLU: gate clamped at 7, linear branch to ±7, output (x_lin + 1) · x_glu · σ(1.702 · x_glu). Routed experts hold 413.1B of the 427.0B parameters.

MTP layer turned EAGLE-3 drafter (not in open weights)One pretrained MTP layer is fine-tuned into an EAGLE-3-style draft model that reads features from three depths.

Same as the blueprintNot in the released weights: neither the BF16 nor the MXFP8 checkpoint contains MTP tensors, and the Transformers port discards mtp.* keys on load. The configs disagree on the module count (num_mtp_modules = 7 with num_nextn_predict_layers = 1 in the BF16 config, 1 in the MXFP8 config), and MiniMax has not described M3's MTP. M2 used 3 modules.

Same as the blueprintPer-Head Muon: momentum of the Q, K and V projections is split along the head dimension and each head's block is orthogonalized separately, equalizing update scale across heads; other matrices use standard Muon. Pretraining also keeps Kimi K2's weight-clipping mechanism, a cosine LR schedule with 1% linear warmup and weight decay 0.1.

Not disclosed

Same as the blueprintQuantization-aware training through all of post-training (SFT and RL): routed-expert weights in MXFP4 (one shared 8-bit scale per 32 values) with MXFP8 activations. Attention, latent MoE projections, shared experts, router, the dense layer and the LM head stay in BF16. Rollouts and training share the scheme, so RL sees no train–inference precision mismatch.

BF16 weights plus an official MXFP8 checkpointBF16 release with an FP32 router; MiniMax-M3-MXFP8 stores the text stack's linears as E4M3 with one 8-bit scale per 32 weights.

Same as the blueprintMulti-teacher on-policy distillation merges the nine domain × effort experts: for a sampled domain and effort level, the matching teacher scores each student token with the clipped log-ratio clip(sg(log π_teacher − log π_θ), −R_max, R_max), used as a dense reward inside the RL framework. Top-k distillation objectives showed no advantage.

Not disclosed

Same as the blueprintSpeculative decoding uses the EAGLE-3-style drafter fine-tuned from the MTP layer. For KDA rollback, the serving kernel caches only the draft tokens' projected inputs and replays them on-chip to rebuild the accepted state, instead of snapshotting the state at every draft position.

Not disclosed

More comparisons