Kimi K3 is a 2.8T-parameter MoE (104B active) whose 93 layers interleave Kimi Delta Attention (KDA) with gated MLA at 3:1, plus one extra MLA layer at the top so the last layer is always global (69 KDA + 24 MLA). There is no positional encoding anywhere: the MLA layers are NoPE and KDA's channel-wise decay carries order, so the 1,048,576-token context needed no RoPE rescaling. K3 also changes KDA itself: the log-decay is bounded to (−5, 0) so every intra-chunk tile runs as a tensor-core matmul, and the output gate becomes full-rank.
Depth is handled by Block Attention Residuals instead of a plain running sum: before each sublayer, a learned pseudo-query takes a softmax over the token embedding, the summaries of earlier 12-layer blocks and the current block's partial sum.
Width is Stable LatentMoE: 896 routed experts, 16 active, working in a 3,584-dim latent (half of d = 7,168), plus two full-width shared experts. Three changes make that sparsity trainable: an RMSNorm on the routed sum before projecting back up, SiTU-GLU (a tanh-soft-capped SwiGLU bounded at 100) and Quantile Balancing, which sets each expert's routing bias from a quantile of routing margins instead of a fixed-step update.
Routed-expert weights (about 2.72T of the 2.78T parameters) ship in MXFP4 after quantization-aware training through SFT and RL; everything else stays in BF16. Vision comes from MoonViT-V2, trained from scratch with the language-model loss. The pretrained MTP layer is turned into an EAGLE-3-style drafter, which is not part of the open checkpoint.