Layout. Blocks 3–59 keep the GQA main attention of the dense blocks (64 query heads, 4 KV heads, d_h = 128) and add an Index Branch of two projections: W_q_idx (6,144 → 4 × 128, one index-query head per GQA group) and W_k_idx (6,144 → 128, one index key shared by all groups). Both get a per-head RMSNorm and the same partial RoPE as the main q and k.
Selection. For query i and GQA group r:
s(i, j) = q_idx(r, i) · k_idx(j) / √128 for j ≤ i
M(i, b) = max of s(i, j) over the 128 tokens j in block b
I(r, i) = top-16 blocks by M(i, ·), the query's own block always included
o(h, i) = softmax( q(h, i) · K_r[I]ᵀ / √128 ) · V_r[I] for the 16 heads h of group r
Each group chooses independently, so a token can draw on up to 4 different block sets, and every query attends to at most 16 × 128 = 2,048 keys per group. The paper's final recipe forces only the query's own, incomplete block, matching the released config (sparse_local_block = 1, sparse_init_block = 0): the learned indexer selects the first block and the local diagonal on its own, and a gpt-oss-style learnable sink gave no consistent gain.
Training the indexer. Top-k is not differentiable, so the index projections learn only from a per-layer KL loss, L = L_LM + λ · Σ_layers KL( sg(P) ‖ P_idx ), where P is the main-branch attention averaged over the group's 16 heads and both distributions are taken over the selected tokens. A stop-gradient on the indexer's input keeps the KL out of the backbone; without it MiniMax saw loss divergence at larger λ and gradual short-context regressions. A full-attention warm-up (40B tokens in the 109B experiments) trains the indexer before selection is switched on. An index value head that added the indexer's own attention output was dropped once the warm-up was in place (sparse_disable_index_value is set in every sparse layer).
Evidence. On a 109B-A6B multimodal MoE with M3's head layout and 3T training tokens, MSA from scratch and MSA converted from a 2.6T-token GQA checkpoint (400B tokens of sparse continued pretraining) stayed broadly level with full attention: MMLU 67.0 / 67.2 / 66.8 and RULER-32K 75.0 / 77.5 / 75.7 (full / scratch / converted). After about 140B tokens of long-context training the converted model scored 72.12 on RULER-128K against 72.00 and 45.93 on HELMET-128K against 46.53. A FLOP-matched sliding window (first block plus a local window) had higher agent-task perplexity, and block sizes of 32, 64 and 128 at an equal token budget left perplexity nearly unchanged. MiniMax does not say which training route M3 took.
Cost. The paper's FLOP model is 2·H_q·d_h·N² for GQA and H_kv·d_idx·N² + 4·H_q·d_h·N·k·B_k for MSA. With M3's numbers that is 28.4× fewer attention FLOPs at 1M tokens, and the indexer is then about 8/9 of what remains: scoring, not attention, is the quadratic term. The KV cache is not reduced; every key and value stays resident. The kernels iterate KV-outer (each selected block gathers the queries that chose it, packing 128 × 128 MMAs) with an exp-free register top-k, and MiniMax reports 14.2× prefill and 7.6× decode wall-clock speedups over GQA on H800 at 1M tokens.