Kimi K3.1 Pro vs gpt-oss-120b

=Same as the blueprintMoonViT-V2, as in K3: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch jointly with the language model. Images and video share weights; a 2×2 pixel shuffle cuts tokens 4× before an MLP projector to 7,168 dims. The model card lists text and image input; Moonshot's API docs also list video.

=Same as the blueprintNoPE throughout. MLA applies no rotary encoding (the 64-dim decoupled key slot is kept but never rotated), and the new indexer sets index_rope_dim = 0, whereas DeepSeek-V3.2's indexer rotates 64 of its 128 dims. KDA's decay and short convolutions carry order, so the 1M window needs no RoPE rescaling or YaRN.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1), unchanged from K3: 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + Swish on q, k, v; L2-normalized q, k. The log-decay is bounded as g = −5·σ(eᴬ·z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in the same 24 layers as K3 (4, 8, …, 92 plus 93), every projection the same shape: 96 heads, queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer), q/k head dim 192, v head dim 128. The difference is the key set: each query now attends only to the 2,048 tokens its layer's DSA indexer selects, not the full prefix.

=Same as the blueprintNew in K3.1: a DSA lightning indexer in each of the 24 MLA layers, each with its own weights (vLLM: no index sharing). 32 heads × 128 dims; queries from the 1,536-dim query latent (1,536 → 4,096), one shared 128-dim key from the hidden state with a LayerNorm, per-head weights from a 7,168 → 32 projection. Scores I = Σₕ wₕ·ReLU(qₕ·k) pick the top 2,048 tokens per query. 7.44M BF16 parameters per layer. DeepSeek-V3.2's indexer has 64 heads. How it was trained is unpublished.

=Same as the blueprintEvery other block (18 of 36) attends only to a banded window of the last 128 tokens, alternating with the dense blocks; same GQA head layout.

=Same as the blueprint128 experts, top-4 active, each a SwiGLU with intermediate size 2,880. The gate is clamped at 7.0 and the linear branch to ±7.0, combined as (x_lin + 1) · x_glu · σ(1.702 · x_glu). Expert weights ship in MXFP4.

=Same as the blueprintTwo full-width shared experts per MoE layer, stored fused as one SiTU-GLU MLP with intermediate size 6,144 (2 × 3,072) at d = 7,168. They read the un-projected input, are ungated and stay in BF16.

=Same as the blueprintInherited from K3's pretraining: Per-Head Muon, which orthogonalizes each head's block of the Q, K and V momentum separately, plus Kimi K2's weight clipping, a cosine LR schedule with 1% linear warmup and weight decay 0.1. Moonshot has not said which optimizer trained the new indexer or K3.1's own training stages.

?Not disclosed

=Same as the blueprintMoonshot has not described speculative decoding for K3.1. The vLLM recipe drafts with Inferact's 5-layer DSpark preview model and SGLang with RadixArk's DFlash2 preview (block 8). Neither drafter comes from Moonshot, and Moonshot has not explained the 2–3× speed of the hosted Highspeed variant.

=Same as the blueprintOnly the 24 DSA layers keep per-token state: 576 MLA values plus a 128-dim indexer key, 704 per layer and 16,896 per token (K3: 13,824). SGLang stores both in FP8 (page size 64), about 18.9 KB per token, and keeps the 69 KDA states (96 × 128 × 128 per layer) in FP32, about 52 MB per request per rank at TP8. Decode context parallelism shards the MLA and indexer-key caches across GPUs.