Kimi K3 vs DeepSeek-V4.1-Flash

=Same as the blueprintTwo Engram modules (196B parameters, FP8) at layers 1 and 14: n-gram orders 2–4, 8 hash heads per order, 2,048 dims per order, about 16M rows per head table. A sigmoid gate on each mHC stream's normalized dot product with its own key adds a shared value. The short causal conv was dropped. Rows are prefetched from host memory over RDMA.

=Same as the blueprintSingle-Pass mHC: n = 4 streams of width 5,120 and 20 Sinkhorn iterations as in V4, but each sublayer reads its input with the mixing coefficients predicted by the previous sublayer. Removing that dependency lets the Mega-mHC kernel read and write the residual once, halving activation traffic, with negligible quality loss reported.

=Same as the blueprintRMSNorm on the single 512-dim KV entry, which the report uses to bound every FP4 KV channel to about √512 ≈ 22.6, and on the 1,280-dim query latent. Unlike V4's reference code, V4.1's applies no separate per-head RMS normalization to queries.

=Same as the blueprintKDA in 69 of 93 layers (three of every four, starting at layer 1): 96 heads with 128-dim q, k, v and a 128 × 128 state per head; short conv (kernel 4) + SiLU on q, k, v; L2-normalized q, k. K3 bounds the per-channel log-decay as g = −5·σ(eᴬ·z) instead of Kimi Linear's −eᴬ·softplus(z), which keeps 16-token-tile rescaling inside BF16 range so every intra-chunk tile runs as a tensor-core matmul.

=Same as the blueprintGated MLA in 24 layers: every fourth layer (4, 8, …, 92) plus layer 93. 96 heads; queries via a 1,536-dim latent, keys and values from a 512-dim latent plus a 64-dim shared key slot (576 cached values per token per layer); q/k head dim 192, v head dim 128. The attention output is kept in FP32 during training to avoid flash attention's biased rounding.

=Same as the blueprintLayers 0–1 are pure sliding-window layers; every other layer adds the 128 most recent uncompressed tokens (SWA KV kept in FP8) to its sparse selection. SWA KV is never persisted: on a prefix-cache hit it is rebuilt by replaying only the last 128 tokens (SWA Bounded Replay), with negligible measured loss.

=Same as the blueprint√softplus scores over 384 experts, top-6, with auxiliary-loss-free bias balancing kept separately for text and image tokens (each bias set updated from its own modality's load, speed 0.001) plus a 0.0001 sequence-level loss. Selected scores are normalized and scaled by 1.5. V4's hash-routed early layers are gone.

=Same as the blueprint384 SwiGLU experts of intermediate size 2,304, 6 active per token, in all 40 layers, with V4's clamp (linear branch ±10, gate ≤ 10). The release stores them in FP4.