Qwen3.5-397B-A17B vs Gemma 4 31B

=Same as the blueprintByte-level BPE (Qwen2Tokenizer): 248,044 BPE entries plus 33 special tokens, padded to 248,320 rows. The pre-tokenizer splits numbers into single digits. Qwen says the ~250K vocabulary (vs. ~150K in Qwen3) boosts encoding/decoding efficiency by 10–60% across most languages.

=Same as the blueprintPartial, interleaved multimodal RoPE (M-RoPE) in the attention layers only: 64 of 256 head dims rotate (partial_rotary_factor 0.25), θ = 10,000,000, with the 32 frequency pairs split 11/11/10 between time, height and width. The Gated DeltaNet layers carry no explicit position.

=Same as the blueprintGated DeltaNet in 45 of 60 layers, three before every attention layer. 16 query/key heads and 64 value heads of dim 128 (each q/k head serves 4 value heads); short causal conv (kernel 4) + SiLU on q, k, v; per-head scalar decay g = −eᴬ·softplus(a + dt_bias) and write strength β = σ(b). Each layer keeps a 64 × 128 × 128 state per sequence, held in FP32.

=Same as the blueprint50 of 60 layers, in runs of five between global layers. Each uses a 1,024-token window with 32 query heads and 16 KV heads of dim 256 (2 query heads per KV head), separate K and V projections and full RoPE with θ = 10,000. Their KV cache is bounded at 1,024 tokens per layer.

=Same as the blueprintIn the attention layers q_proj emits 2 × 32 × 256 values per token; the second half is an elementwise sigmoid gate, computed from the layer input, applied to the attention output before W_o. Gated DeltaNet layers gate their output with SiLU(z) after a per-head RMSNorm. No sink logits.

=Same as the blueprint512 SwiGLU experts with intermediate size 1,024 at the full model width 4,096 (no latent projection); 10 are active per token, about 2% of the pool. Weights ship in BF16, with an official FP8 variant.

=Same as the blueprintOne shared SwiGLU expert per layer (intermediate 1,024) runs on every token, but its output is scaled by a learned per-token sigmoid gate σ(w·x), w ∈ ℝ^4096, before it is added to the routed sum. Unlike DeepSeek-style shared experts it is not ungated.

=Same as the blueprintUntied 4,096 → 248,320 output projection (~1.02B parameters), also used by the MTP layer; kept in BF16 in the FP8 release.

=Same as the blueprintOne MTP layer: a full gated-attention + 512-expert MoE block whose input is a linear projection (8,192 → 4,096) of the RMSNorm'd next-token embedding concatenated with the RMSNorm'd trunk state. It reuses the main embedding and LM head and, per the model card, was trained with multiple steps, so the one layer is applied repeatedly to draft several tokens.

=Same as the blueprint262,144 tokens natively. The model card extends this to 1,010,000 tokens with static YaRN (factor 4 over 262,144) on the rotary dims, and warns that static YaRN can hurt performance on shorter texts. The pretraining context schedule is not published.

?Not disclosed

=Same as the blueprintQwen describes RL scaled across million-scale agent scaffolds and environments on an asynchronous, fully disaggregated training–inference framework, using FP8 end-to-end training, rollout router replay, speculative decoding and multi-turn rollout locking for throughput and train–inference consistency (3–5× end-to-end speedup). The policy-optimization algorithm is not published.

?Not disclosed