Qwen3.5-397B-A17B scales the Qwen3-Next design to 397B parameters. Its 60 layers repeat one pattern 15 times: three Gated DeltaNet layers, then one gated-attention layer, each followed by an MoE block. Only the 15 attention layers keep a KV cache, and they use 2 KV heads of dimension 256, so the model caches about 30 KiB per token in BF16; the 45 linear layers hold a fixed recurrent state. Qwen reports 8.6× and 19.0× the decode throughput of Qwen3-Max at 32K and 256K context.
The attention layers use per-head QK-Norm, partial interleaved M-RoPE on 64 of 256 head dims (θ = 10⁷) and an elementwise sigmoid output gate. The Gated DeltaNet layers use 16 query/key heads shared by 64 value heads, a scalar per-head decay and a SiLU-gated output norm.
The FFN half is a fine-grained MoE with 512 experts of width 1,024: 10 routed experts plus one shared expert whose output is scaled by a learned sigmoid gate. Routing is a plain softmax top-k with renormalization and an auxiliary balance loss, not the sigmoid-and-bias scheme of DeepSeek and Kimi. The residual is a single pre-norm stream, and one MTP layer provides self-speculative drafts.
The model is natively vision-language (early fusion with a 27-layer ViT) and moves to a ~250K vocabulary. Qwen has not published a technical report for it, so the optimizer, token counts, context schedule and RL algorithm are undisclosed.