Nemotron 3 Ultra vs Gemma 4 31B

=Same as the blueprintByte-level BPE with 131,072 entries, the first 1,000 of which are reserved special and control tokens; the pre-tokenizer splits every digit into its own piece. The model is text-only.

=Same as the blueprintA ~550M-parameter ViT (27 layers, width 1,152, MLP 4,304, 16 heads, 16-px patches) with 2D RoPE and 2D absolute position embeddings, kept frozen during pretraining. Its outputs are pooled 3 × 3, so each soft token covers 48 × 48 px, under a per-image budget of 70, 140, 280 (default), 560 or 1,120 tokens. Video enters as frames; the 31B has no audio encoder.

=Same as the blueprintPer-head RMSNorm with a learned gain on queries and keys (256-dim heads in local layers, 512-dim in global ones), so the logits use scale 1 instead of 1/√d. Values get a gain-free RMSNorm. There is no attention-logit soft-cap.

=Same as the blueprintNoPE: the 12 attention layers apply no positional encoding (Nemotron 3 Super omits positional embeddings, and Ultra keeps the same architecture). Token order reaches them only through the Mamba-2 layers' causal convolutions and recurrent state. config.json still lists rope_theta = 10,000, but the Hugging Face implementation never applies it.

=Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

=Same as the blueprintLinear router at the full 8,192 width: sigmoid scores over 512 experts, a per-expert bias steers top-22 selection, and the selected scores are renormalized and scaled by 5.0. Balancing follows the Nemotron 3 Super recipe: aux-loss-free bias updates (rate 1e-3) plus a standard load-balancing loss (coefficient 1e-4).

=Same as the blueprintOne shared expert per MoE layer at full width: a non-gated squared-ReLU MLP, 8,192 → 10,240 → 8,192, added to the routed output without a router weight. The released checkpoint stores it in FP8 (per-tensor), not NVFP4.

=Same as the blueprintUntied 8,192 × 131,072 output projection, shared with the MTP block and kept in BF16. Its gradient accumulation must stay in FP32: lowering it to BF16 erased the MTP heads' small gradient contribution and caused a pretraining divergence at about 8T tokens.

=Same as the blueprintOne MTP block (an attention layer followed by a LatentMoE layer) whose weights are shared across steps and applied recursively. It ran 2 steps in pretraining (total loss weight 0.1, 0.05 per step). MTP Boosting then trains only this head with a forward-KL loss to the frozen backbone's logits over 7 steps, to match the inputs it sees when drafting.

?Not disclosed

=Same as the blueprintPretraining used 8,192-token sequences (Nemotron 3 Super recipe). A 33B-token continued-pretraining phase then trained at 1,048,576 tokens for 92% of iterations and at 4,096 for the rest, at a constant LR of 2.5e-6. With no RoPE, no YaRN-style rescaling is involved.

?Not disclosed

=Same as the blueprintAfter a two-stage SFT, a single unified RLVR stage ran asynchronous GRPO across agentic, coding, math, STEM, safety, chat, instruction-following and long-context environments. The maximum generation length grew from 48K to 64K tokens, and RLHF used a released GenRM. MTP drafting with 5 tokens sped up rollout generation 1.46×.

?Not disclosed

=Same as the blueprintMulti-teacher on-policy distillation (MOPD) in two iterations. More than ten domain-specialized teachers (STEM, coding, SWE, search, terminal use, office work, chat, safety and others) give dense token-level reverse-KL signals on the student's own rollouts. The RLVR student acts as a self-teacher for uncovered domains.

?Not disclosed