Skip to content
archdiff

models / compare / nemotron-3-ultra-vs-kimi-k3-1-pro

Nemotron 3 Ultra vs Kimi K3.1 Pro

How Nemotron 3 Ultra (NVIDIA, June 2026) and Kimi K3.1 Pro (Moonshot AI, October 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 12 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ nemotron-3-ultra@2026-06+0~4−7

Nemotron 3 Ultra

NVIDIA, June 2026, 550B total, 55B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream, one sublayer per layer — Residual & normSingle residual stream, one sublayer p…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneMamba-2 selective state-space layers — AttentionMamba-2 selective state-spac…AttentionGlobal GQA attention without positional encoding — AttentionGlobal GQA attention without…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingLatentMoE routed experts (ReLU², 2,048-dim latent) — Mixture of expertsLatentMoE routed experts (ReLU², 2,048…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ kimi-k3-1-pro@2026-10+0~5−2

Kimi K3.1 Pro

Moonshot AI, October 2026, 2.8T total, 104B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Block Attention Residuals (AttnRes) — Residual & normBlock Attention Residuals (AttnRes)Residual & normRMSNorm — pre-normRMSNormpre-normQK-Clip and normalized latents; LayerNorm on the indexer key — Residual & normQK-Clip and normalized latents; LayerN…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with Quantile Balancing — Mixture of expertsSigmoid router with Quantile BalancingMixture of expertsStable LatentMoE routed experts (SiTU-GLU) — Mixture of expertsStable LatentMoE routed experts (SiTU-…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsNo MTP layer in the open checkpoint — Input & outputNo MTP layer in the open checkpointInput & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

Nemotron 3 Ultra and Kimi K3.1 Pro differ on 12 of 27 components.

Blueprint componentNemotron 3 UltraKimi K3.1 Pro

Not used

Same as the blueprintMoonViT-V2, as in K3: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch jointly with the language model. Images and video share weights; a 2×2 pixel shuffle cuts tokens 4× before an MLP projector to 7,168 dims. The model card lists text and image input; Moonshot's API docs also list video.

Single residual stream, one sublayer per layer108 pre-norm layers on one d = 8,192 residual stream; each layer is one mixer or one MoE, not an attention+FFN pair.

Block Attention Residuals (AttnRes)Each sublayer reads a softmax-weighted mix of the embedding and earlier 12-layer block summaries, as in K3.

Not used

QK-Clip and normalized latents; LayerNorm on the indexer keyNo per-head QK-Norm: K2-style weight clipping for MLA, L2-normalized q/k in KDA, a LayerNorm on the DSA indexer key.

Mamba-2 selective state-space layers48 of the 60 mixer layers: SSD with 256 heads × 64 dims, state size 128, 8 B/C groups, causal conv of kernel 4.

Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1), unchanged from K3: 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + Swish on q, k, v; L2-normalized q, k. The log-decay is bounded as g = −5·σ(eᴬ·z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

Global GQA attention without positional encoding12 layers of dense causal attention: 64 query heads share 2 KV heads (d_h = 128); no RoPE, QK-norm, gate or sink.

Same as the blueprintGated MLA in the same 24 layers as K3 (4, 8, …, 92 plus 93), every projection the same shape: 96 heads, queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer), q/k head dim 192, v head dim 128. The difference is the key set: each query now attends only to the 2,048 tokens its layer's DSA indexer selects, not the full prefix.

Not used

Same as the blueprintNew in K3.1: a DSA lightning indexer in each of the 24 MLA layers, each with its own weights (vLLM: no index sharing). 32 heads × 128 dims; queries from the 1,536-dim query latent (1,536 → 4,096), one shared 128-dim key from the hidden state with a LayerNorm, per-head weights from a 7,168 → 32 projection. Scores I = Σₕ wₕ·ReLU(qₕ·k) pick the top 2,048 tokens per query. 7.44M BF16 parameters per layer. DeepSeek-V3.2's indexer has 64 heads. How it was trained is unpublished.

Not used

Same as the blueprintBoth mixers keep K3's input-dependent, full-rank, channel-wise sigmoid gate σ(W_g·x), W_g: 7,168 → 12,288 (96 × 128). In gated DSA it multiplies the sparse MLA output; in KDA it multiplies the head-wise RMSNorm of the recurrent output. No sink logits.

Same as the blueprintLinear router at the full 8,192 width: sigmoid scores over 512 experts, a per-expert bias steers top-22 selection, and the selected scores are renormalized and scaled by 5.0. Balancing follows the Nemotron 3 Super recipe: aux-loss-free bias updates (rate 1e-3) plus a standard load-balancing loss (coefficient 1e-4).

Sigmoid router with Quantile BalancingSigmoid scores pick 16 of 896 experts; each expert's selection bias is set every step from a quantile of routing margins.

LatentMoE routed experts (ReLU², 2,048-dim latent)512 non-gated squared-ReLU experts of width 5,120 run in a 2,048-dim latent space; 22 active per token.

Stable LatentMoE routed experts (SiTU-GLU)896 SiTU-GLU experts run in a 3,584-dim latent (half of d); 16 fire per token; an RMSNorm precedes the up-projection.

Same as the blueprintOne MTP block (an attention layer followed by a LatentMoE layer) whose weights are shared across steps and applied recursively. It ran 2 steps in pretraining (total loss weight 0.1, 0.05 per step). MTP Boosting then trains only this head with a forward-KL loss to the frozen backbone's logits over 7 steps, to match the inputs it sees when drafting.

No MTP layer in the open checkpointThe checkpoint ships without MTP weights; Moonshot has not described a K3.1 drafter, so serving engines bring their own.

Not used

Same as the blueprintInherited from K3's pretraining: Per-Head Muon, which orthogonalizes each head's block of the Q, K and V momentum separately, plus Kimi K2's weight clipping, a cosine LR schedule with 1% linear warmup and weight decay 0.1. Moonshot has not said which optimizer trained the new indexer or K3.1's own training stages.

Same as the blueprintMulti-teacher on-policy distillation (MOPD) in two iterations. More than ten domain-specialized teachers (STEM, coding, SWE, search, terminal use, office work, chat, safety and others) give dense token-level reverse-KL signals on the student's own rollouts. The RLVR student acts as a self-teacher for uncovered domains.

Not disclosed

More comparisons