DeepSeek-V4-Pro vs Kimi K3.1 Pro

=Same as the blueprintMoonViT-V2, as in K3: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch jointly with the language model. Images and video share weights; a 2×2 pixel shuffle cuts tokens 4× before an MLP projector to 7,168 dims. The model card lists text and image input; Moonshot's API docs also list video.

=Same as the blueprintmHC with n = 4 streams of width 7,168 around every attention and MoE sublayer. The residual-mixing matrix is projected to doubly stochastic with 20 Sinkhorn–Knopp iterations; read-in uses σ(·) and write-back 2σ(·). DeepSeek reports a 6.7% wall-time overhead per overlapped 1F1B pipeline stage.

=Same as the blueprintAn RMS normalization of each of the 128 query heads and an RMSNorm on the single 512-dim KV entry, applied just before core attention in both CSA and HCA. DeepSeek credits this for training with Muon without QK-Clip.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1), unchanged from K3: 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + Swish on q, k, v; L2-normalized q, k. The log-decay is bounded as g = −5·σ(eᴬ·z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in the same 24 layers as K3 (4, 8, …, 92 plus 93), every projection the same shape: 96 heads, queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer), q/k head dim 192, v head dim 128. The difference is the key set: each query now attends only to the 2,048 tokens its layer's DSA indexer selects, not the full prefix.

=Same as the blueprintV4-Pro has no pure sliding-window layers (V4-Flash makes its first two layers pure SWA). Every CSA and HCA layer adds 128 uncompressed KV entries for the most recent tokens to the same softmax, which covers the tokens in the query's own not-yet-compressed block.

=Same as the blueprint√softplus affinities over 384 experts with a per-expert bias used only to pick the top 6 (auxiliary-loss-free balancing, bias speed 0.001) plus a 0.0001 sequence-wise balance loss. The selected scores are normalized and scaled by 2.5. V3's node limit is removed, and the first 3 MoE layers pick experts by a fixed token-id hash. Anticipatory Routing (indices from Δt-step-old parameters) was switched on only around loss spikes.

=Same as the blueprint384 SwiGLU experts of intermediate size 3,072, 6 active per token, in all 61 layers. The linear branch is clamped to [−10, 10] and the gate capped at 10. Expert weights went through MXFP4 quantization-aware training in post-training and ship in FP4.

=Same as the blueprintOne MTP module (depth 1), unchanged from DeepSeek-V3: an extra block that combines the normalized next-token embedding with the trunk's normalized state and shares the embedding and LM head. Its attention is sliding-window only. Loss weight 0.3, lowered to 0.1 when learning-rate decay starts.

=Same as the blueprintMore than ten specialist teachers are merged into one student by on-policy distillation with a weighted, full-vocabulary reverse-KL loss. Teacher logits are rebuilt on the fly from cached last-layer hidden states instead of being stored.

?Not disclosed

?Not disclosed

=Same as the blueprintMoonshot has not described speculative decoding for K3.1. The vLLM recipe drafts with Inferact's 5-layer DSpark preview model and SGLang with RadixArk's DFlash2 preview (block 8). Neither drafter comes from Moonshot, and Moonshot has not explained the 2–3× speed of the hosted Highspeed variant.