Kimi K3 vs DeepSeek-V4-Pro

=Same as the blueprintMoonViT-V2: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch with the language-model loss instead of a SigLIP initialization. Images and video share weights (factorized spatial/temporal attention, temporal pooling); a 2×2 pixel shuffle cuts tokens 4× before an MLP projector, allowing inputs up to 3,584 × 3,584 px.

=Same as the blueprintmHC with n = 4 streams of width 7,168 around every attention and MoE sublayer. The residual-mixing matrix is projected to doubly stochastic with 20 Sinkhorn–Knopp iterations; read-in uses σ(·) and write-back 2σ(·). DeepSeek reports a 6.7% wall-time overhead per overlapped 1F1B pipeline stage.

=Same as the blueprintAn RMS normalization of each of the 128 query heads and an RMSNorm on the single 512-dim KV entry, applied just before core attention in both CSA and HCA. DeepSeek credits this for training with Muon without QK-Clip.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1): 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + SiLU on q, k, v; L2-normalized q, k. K3 bounds the per-channel log-decay as g = −5·σ(eᴬ·z) instead of Kimi Linear's −eᴬ·softplus(z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in 24 layers: every fourth layer (4, 8, …, 92) plus layer 93. 96 heads; queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer); q/k head dim 192, v head dim 128. The attention output is kept in FP32 during training to avoid flash attention's biased rounding.

=Same as the blueprintCompressed Sparse Attention in 30 of 61 layers (even layers 2–60), interleaved with HCA. Each KV entry pools 8 overlapping tokens at stride m = 4 with a per-channel softmax. A lightning indexer (64 heads × 128 dims, ReLU scores weighted per head) keeps the top 1,024 entries, which the 128 query heads attend to together with the SWA entries.

=Same as the blueprintV4-Pro has no pure sliding-window layers (V4-Flash makes its first two layers pure SWA). Every CSA and HCA layer adds 128 uncompressed KV entries for the most recent tokens to the same softmax, which covers the tokens in the query's own not-yet-compressed block.

=Same as the blueprint√softplus affinities over 384 experts with a per-expert bias used only to pick the top 6 (auxiliary-loss-free balancing, bias speed 0.001) plus a 0.0001 sequence-wise balance loss. The selected scores are normalized and scaled by 2.5. V3's node limit is removed, and the first 3 MoE layers pick experts by a fixed token-id hash. Anticipatory Routing (indices from Δt-step-old parameters) was switched on only around loss spikes.

=Same as the blueprint384 SwiGLU experts of intermediate size 3,072, 6 active per token, in all 61 layers. The linear branch is clamped to [−10, 10] and the gate capped at 10. Expert weights went through MXFP4 quantization-aware training in post-training and ship in FP4.

=Same as the blueprintOne MTP module (depth 1), unchanged from DeepSeek-V3: an extra block that combines the normalized next-token embedding with the trunk's normalized state and shares the embedding and LM head. Its attention is sliding-window only. Loss weight 0.3, lowered to 0.1 when learning-rate decay starts.

=Same as the blueprintSpeculative decoding uses the EAGLE-3-style drafter fine-tuned from the MTP layer. For KDA rollback, the serving kernel caches only the draft tokens' projected inputs and replays them on-chip to rebuild the accepted state, instead of snapshotting the state at every draft position.

?Not disclosed