DeepSeek-V4-Pro (1.6T total, 49B active, 61 layers, d = 7,168) keeps V3's DeepSeekMoE feed-forward and its single MTP module but rebuilds attention around sequence compression. Every layer is one of two global types, interleaved: Compressed Sparse Attention (CSA) pools every 4 tokens into one 512-dim KV entry and lets a lightning indexer pick the top 1,024 entries, while Heavily Compressed Attention (HCA) pools every 128 tokens and attends to all of them densely. Both run shared-KV multi-query attention (128 query heads over one KV head), add a 128-token uncompressed sliding-window branch, carry per-head sink logits and RMS-normalize queries and KV entries. At 1M tokens DeepSeek reports 27% of V3.2's per-token FLOPs and 10% of its KV cache.
The residual stream is widened to four streams with mHC (Sinkhorn-projected doubly stochastic mixing), and training uses Muon with hybrid Newton–Schulz iterations and no QK-Clip. Routing switches to √softplus scores, drops V3's node limit and hash-routes the first three MoE layers. Loss spikes traced to MoE outliers were handled with SwiGLU clamping and Anticipatory Routing. Routed experts ship in FP4 after MXFP4 quantization-aware training in post-training.
V4-Flash (284B total, 13B active) is the same design at d = 4,096 with 43 layers, 64 heads, 256 routed experts of width 2,048 and top-512 CSA selection, and its first two layers are pure sliding-window attention where Pro uses HCA. It needs 10% of V3.2's FLOPs and 7% of its KV cache at 1M tokens. Both models were pretrained on 32–33T tokens and post-trained by merging SFT + GRPO specialists through on-policy distillation.