Skip to content
archdiff

models / compare / minimax-m2-vs-deepseek-v4-1-flash

MiniMax-M2.7 vs DeepSeek-V4.1-Flash

How MiniMax-M2.7 (MiniMax, March 2026) and DeepSeek-V4.1-Flash (DeepSeek, September 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 13 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ minimax-m2@2026-03+0~4−7

MiniMax-M2.7

MiniMax, March 2026, 229.9B total, 9.8B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normLayer-wide QK RMSNorm — Residual & normLayer-wide QK RMSNormResidual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in all 62 layers — AttentionFull grouped-query attention…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 block-scaled checkpoint — Training & inferenceFP8 block-scaled checkpointTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

--- frontier-llm@2026-11+++ deepseek-v4-1-flash@2026-09+0~3−1

DeepSeek-V4.1-Flash

DeepSeek, September 2026, 552B + 196B Engram total, 16B (8B in prefill) active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersCED decoder attention (CSA2 over encoder-projected KV) — AttentionCED decoder attention (CSA2…AttentionCSA2 encoder layers (2× compression, Full + Reuse) — AttentionCSA2 encoder layers (2× comp…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsDSpark drafter (trained after pretraining) — Input & outputDSpark drafter (trained after pretrain…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

Component by component

MiniMax-M2.7 and DeepSeek-V4.1-Flash differ on 13 of 27 components.

Blueprint componentMiniMax-M2.7DeepSeek-V4.1-Flash

Not used

Same as the blueprintDeepSeek-ViT, trained from scratch: 32 layers, width 1,024, 16 heads, 14-px patches embedded by a linear projection, 2D-RoPE, RMSNorm and SwiGLU. A 3×3 pixel-unshuffle cuts visual tokens 9× before a 2-layer MLP projector maps them to 5,120 dims. Images only, up to about 1344×1344 px.

Not used

Same as the blueprintTwo Engram modules (196B parameters, FP8) at layers 1 and 14: n-gram orders 2–4, 8 hash heads per order, 2,048 dims per order, about 16M rows per head table. A sigmoid gate on each mHC stream's normalized dot product with its own key adds a shared value. The short causal conv was dropped. Rows are prefetched from host memory over RDMA.

Single residual streamOne d = 3,072 residual stream with pre-norm sublayers; no hyper-connections.

Same as the blueprintSingle-Pass mHC: n = 4 streams of width 5,120 and 20 Sinkhorn iterations as in V4, but each sublayer reads its input with the mixing coefficients predicted by the previous sublayer. Removing that dependency lets the Mega-mHC kernel read and write the residual once, halving activation traffic, with negligible quality loss reported.

Layer-wide QK RMSNormOne RMSNorm over all query heads (6,144 dims) and one over all key heads (1,024 dims), applied before RoPE.

Same as the blueprintRMSNorm on the single 512-dim KV entry, which the report uses to bound every FP4 KV channel to about √512 ≈ 22.6, and on the 1,280-dim query latent. Unlike V4's reference code, V4.1's applies no separate per-head RMS normalization to queries.

Full grouped-query attention in all 62 layersCausal softmax over the whole context in every layer: 48 query heads share 8 KV heads of dim 128; no window, sink or gate.

CED decoder attention (CSA2 over encoder-projected KV)Layers 20–39 take uncompressed global KV from the encoder's last hidden state and re-select with a hierarchical indexer.

Not used

CSA2 encoder layers (2× compression, Full + Reuse)Layers 2–19: KV pooled 2→1 and computed in only 3 of 18 layers; the rest reuse that KV and its top-512 selection.

Not used

Same as the blueprintLayers 0–1 are pure sliding-window layers; every other layer adds the 128 most recent uncompressed tokens (SWA KV kept in FP8) to its sparse selection. SWA KV is never persisted: on a prefix-cache hit it is rebuilt by replaying only the last 128 tokens (SWA Bounded Replay), with negligible measured loss.

Not used

Same as the blueprintNo output gate. Each of the 64 heads has a learnable sink logit in the softmax denominator, per the reference inference code.

Not used

Same as the blueprintOne always-active shared SwiGLU expert of intermediate size 2,304 in every MoE layer.

Same as the blueprintDeepSeek-V3-style sequential MTP, each module one Transformer layer. Pretraining used a single module with loss weight 0.3, annealed to 0.1; in the decay phase it was expanded to 3 modules initialized by copying main-model weights, trained alone first and then jointly. The released M2 and M2.7 checkpoints contain no MTP weights.

DSpark drafter (trained after pretraining)Three 128-token-window blocks draft 5 tokens per pass from the backbone's last 3 layers; no MTP loss in pretraining.

Not disclosed

Same as the blueprintMuon (momentum 0.95, weight decay 0.1, update RMS 0.18) for backbone linear weights, Engram projections and the vision projector, applied per head to query and key weights. Embeddings, the head and Engram tables use a momentum update with Sinkhorn row/column balancing (Engram learning rate 5×); AdamW covers norms and other non-matrix parameters.

FP8 block-scaled checkpointOpen weights ship as FP8 E4M3 with 128 × 128 block scales; router, its bias, embeddings and LM head stay in higher precision.

Same as the blueprintThe release stores most weights in FP8 and routed experts in FP4. Post-training QAT extends FP4 to the main KV cache (E2M1 with one E4M3 scale per 16 channels and no global scale), indexer queries and keys stay FP4 as in V4, and the SWA KV stays FP8 because it is sensitive to quantization.

Not disclosed

Same as the blueprintThe final stage is full-vocabulary on-policy distillation over all domains from more than 40 teachers, which may differ architecturally from the student, run with asynchronous generation.

More comparisons