Skip to content
archdiff

models / compare / inkling-vs-nemotron-3-ultra

Inkling vs Nemotron 3 Ultra

How Inkling (Thinking Machines Lab, July 2026) and Nemotron 3 Ultra (NVIDIA, June 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 12 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ inkling@2026-07+0~5−3

Inkling

Thinking Machines Lab, July 2026, 975B total, 41B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single pre-norm residual stream — Residual & normSingle pre-norm residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kLearned relative-position bias (no RoPE) — AttentionLearned relative-position bias (no RoP…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with log-length scaling — AttentionGlobal GQA with log-length s…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowShort causal convolutions (SConv) — AttentionShort causal convolutions (SConv)Attention⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normSigmoid router with shared-expert sink — Mixture of expertsSigmoid router with shared-expert sinkMixture of expertsRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ nemotron-3-ultra@2026-06+0~4−7

Nemotron 3 Ultra

NVIDIA, June 2026, 550B total, 55B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream, one sublayer per layer — Residual & normSingle residual stream, one sublayer p…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneMamba-2 selective state-space layers — AttentionMamba-2 selective state-spac…AttentionGlobal GQA attention without positional encoding — AttentionGlobal GQA attention without…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingLatentMoE routed experts (ReLU², 2,048-dim latent) — Mixture of expertsLatentMoE routed experts (ReLU², 2,048…Mixture of expertsShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

Component by component

Inkling and Nemotron 3 Ultra differ on 12 of 27 components.

Blueprint componentInklingNemotron 3 Ultra

Same as the blueprintEncoder-free. Images and video are cut into 40 × 40-pixel patches, 2 frames deep. A 4-layer hierarchical MLP (three linear + RMSNorm + GELU stages that fold space and time into channels, then a linear projection) maps each patch to one 6,144-dim token. 16 kHz audio becomes dMel frames: 80 mel bins, each quantized to 16 levels, embedded by lookup and summed per frame.

Not used

Single pre-norm residual streamOne d = 6,144 residual stream through 66 pre-norm blocks; no hyper-connections, branch outputs pass a short conv.

Single residual stream, one sublayer per layer108 pre-norm layers on one d = 8,192 residual stream; each layer is one mixer or one MoE, not an attention+FFN pair.

Same as the blueprintPer-head RMSNorm on q and k (head dim 128) in all 66 layers, applied to keys after their short convolution. Because q and k are normalized, the softmax scale is 1/d_h = 1/128 instead of 1/√d_h.

Not used

Learned relative-position bias (no RoPE)Each head adds a bias by backward distance: a query-dependent mix of 16 learned profiles, over 512 (local) or 1,024 (global) tokens.

Same as the blueprintNoPE: the 12 attention layers apply no positional encoding (Nemotron 3 Super omits positional embeddings, and Ultra keeps the same architecture). Token order reaches them only through the Mamba-2 layers' causal convolutions and recurrent state. config.json still lists rope_theta = 10,000, but the Hugging Face implementation never applies it.

Not used

Mamba-2 selective state-space layers48 of the 60 mixer layers: SSD with 256 heads × 64 dims, state size 128, 8 B/C groups, causal conv of kernel 4.

Global GQA with log-length scaling11 full-attention layers (every 6th, incl. the last): 64 Q / 8 KV heads, d_h = 128; logits scaled up logarithmically past 128K.

Global GQA attention without positional encoding12 layers of dense causal attention: 64 query heads share 2 KV heads (d_h = 128); no RoPE, QK-norm, gate or sink.

Same as the blueprint55 of 66 layers (all except 5, 11, …, 65) attend to the last 512 tokens with 64 query heads and 16 KV heads of dimension 128. Keys and values pass through a kernel-4 causal convolution before caching, and the relative-position bias spans the whole window.

Not used

Short causal convolutions (SConv)Depthwise causal convs (kernel 4, with skip) on K, V and on both sublayer outputs before they rejoin the residual.

Not used

Sigmoid router with shared-expert sinkScores 256 routed + 2 shared experts; bias-steered top-6; weights normalized jointly over the 8 active experts, then ×8.

Same as the blueprintLinear router at the full 8,192 width: sigmoid scores over 512 experts, a per-expert bias steers top-22 selection, and the selected scores are renormalized and scaled by 5.0. Balancing follows the Nemotron 3 Super recipe: aux-loss-free bias updates (rate 1e-3) plus a standard load-balancing loss (coefficient 1e-4).

Same as the blueprint256 routed SwiGLU experts of intermediate size 3,072, 6 active per token, in blocks 2–65 (64 MoE layers). Blocks 0 and 1 use a dense SwiGLU MLP of width 24,576 instead.

LatentMoE routed experts (ReLU², 2,048-dim latent)512 non-gated squared-ReLU experts of width 5,120 run in a 2,048-dim latent space; 22 active per token.

Same as the blueprintMuon for large matrix weights and Adam for all other parameters, with weight decay coupled to the square of the learning rate. Hyperparameter schedules draw on Thinking Machines' modular-manifolds research. Newton–Schulz settings are not published.

Not used

Not disclosed

Same as the blueprintMulti-teacher on-policy distillation (MOPD) in two iterations. More than ten domain-specialized teachers (STEM, coding, SWE, search, terminal use, office work, chat, safety and others) give dense token-level reverse-KL signals on the student's own rollouts. The RLVR student acts as a self-teacher for uncovered domains.

More comparisons