Kimi K3 vs gpt-oss-120b

=Same as the blueprintMoonViT-V2: a 27-layer, 401M-parameter ViT (14-px patches, width 1,024, 12 heads) with RMSNorm and no bias terms, trained from scratch with the language-model loss instead of a SigLIP initialization. Images and video share weights (factorized spatial/temporal attention, temporal pooling); a 2×2 pixel shuffle cuts tokens 4× before an MLP projector, allowing inputs up to 3,584 × 3,584 px.

=Same as the blueprintNoPE throughout. The 24 MLA layers apply no rotary or other positional encoding (the 64-dim decoupled key slot of MLA is kept but never rotated), and KDA's decay and short convolutions carry order. Context therefore extends to 1M tokens without RoPE rescaling or YaRN.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1): 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + SiLU on q, k, v; L2-normalized q, k. K3 bounds the per-channel log-decay as g = −5·σ(eᴬ·z) instead of Kimi Linear's −eᴬ·softplus(z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in 24 layers: every fourth layer (4, 8, …, 92) plus layer 93. 96 heads; queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer); q/k head dim 192, v head dim 128. The attention output is kept in FP32 during training to avoid flash attention's biased rounding.

=Same as the blueprintEvery other block (18 of 36) attends only to a banded window of the last 128 tokens, alternating with the dense blocks; same GQA head layout.

=Same as the blueprint128 experts, top-4 active, each a SwiGLU with intermediate size 2,880. The gate is clamped at 7.0 and the linear branch to ±7.0, combined as (x_lin + 1) · x_glu · σ(1.702 · x_glu). Expert weights ship in MXFP4.

=Same as the blueprintTwo full-width shared experts per MoE layer, stored fused as one SiTU-GLU MLP with intermediate size 6,144 (2 × 3,072) at d = 7,168. They read the un-projected input, are ungated and stay in BF16.

=Same as the blueprintPer-Head Muon: momentum of the Q, K and V projections is split along the head dimension and each head's block is orthogonalized separately, equalizing update scale across heads; other matrices use standard Muon. Pretraining also keeps Kimi K2's weight-clipping mechanism, a cosine LR schedule with 1% linear warmup and weight decay 0.1.

?Not disclosed

=Same as the blueprintMulti-teacher on-policy distillation merges the nine domain × effort experts: for a sampled domain and effort level, the matching teacher scores each student token with the clipped log-ratio clip(sg(log π_teacher − log π_θ), −R_max, R_max), used as a dense reward inside the RL framework. Top-k distillation objectives showed no advantage.

?Not disclosed

=Same as the blueprintSpeculative decoding uses the EAGLE-3-style drafter fine-tuned from the MTP layer. For KDA rollback, the serving kernel caches only the draft tokens' projected inputs and replays them on-chip to rebuild the accepted state, instead of snapshotting the state at every draft position.

=Same as the blueprintOnly the 24 MLA layers keep a per-token cache: 576 values per token per layer, 13,824 per token in total (~27 KiB in BF16). Each KDA layer holds a fixed 96 × 128 × 128 state per sequence. Serving packs KDA states into the same paged pool as MLA pages, matches prefixes at 512-token granularity and checkpoints KDA states at a sparse subset of those boundaries; RL rollouts spill idle prefixes to a CPU-DRAM pool.