Skip to content
archdiff

models / compare / glm-5-vs-minimax-m3

GLM-5 vs MiniMax-M3

How GLM-5 (Z.ai (Zhipu AI), February 2026) and MiniMax-M3 (MiniMax, June 2026) are built, drawn on the same November 2026 blueprint. They make different choices on 8 of 27 components. Select a component in either diagram to find it in the table.

--- frontier-llm@2026-11+++ glm-5@2026-02+0~3−5

GLM-5

Z.ai (Zhipu AI), February 2026, 744B total, 40B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normRMSNorm on MLA latents (no per-head QK-Norm) — Residual & normRMSNorm on MLA latents (no per-head QK…Residual & normPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headINT4 QAT in SFT, FP8 rollouts — Training & inferenceINT4 QAT in SFT, FP8 rolloutsTraining & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot used

--- frontier-llm@2026-11+++ minimax-m3@2026-06+0~4−4

MiniMax-M3

MiniMax, June 2026, 427.0B total, 23.5B active

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream — Residual & normSingle residual streamResidual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersFull grouped-query attention in blocks 0–2 — AttentionFull grouped-query attention…AttentionMiniMax Sparse Attention (MSA) — AttentionMiniMax Sparse Attention (MS…AttentionSliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headBF16 weights plus an official MXFP8 checkpoint — Training & inferenceBF16 weights plus an official MXFP8 ch…Training & inferenceLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Component by component

GLM-5 and MiniMax-M3 differ on 8 of 27 components.

Blueprint componentGLM-5MiniMax-M3

Not used

Same as the blueprintA 32-layer CLIP-style ViT (width 1,280, 16 heads, GELU MLP of 5,120, LayerNorm; 631M parameters) embeds 14 × 14 patches with a Conv3d that spans 2 frames for video, using 3D (time, height, width) RoPE. A GELU MLP lifts each patch to 6,144 and a second fuses each 2 × 2 group into one token (234M projector parameters). MiniMax says M3 trained on mixed modalities from the first step and lists text, image and video inputs.

RMSNorm on MLA latents (no per-head QK-Norm)Queries and keys are normalized as compressed latents (2,048 and 512 dims) before up-projection, not per head.

Same as the blueprintPer-head Gemma-style RMSNorm (ε = 1e-6) on q and k before RoPE, with one 128-dim gain shared by all 64 query heads and one by the 4 key heads; the indexer's queries and key get their own. This replaces M2's layer-wide norm over the concatenated projections (qk_norm_type per_layer → per_head), so head-sharded tensor parallelism no longer needs a cross-rank reduction for it.

Same as the blueprintMLA in all 78 layers and the MTP layer, with 64 heads: 2,048-dim query latent, 512-dim KV latent plus a 64-dim rotary key (576 cached values per token per layer), q/k head dim 256 (192 + 64 rotary), v head dim 256. This is the MLA-256 variant: head dim raised from 192 to 256 with one-third fewer heads, keeping training FLOPs and parameters roughly constant while lowering decode compute.

Full grouped-query attention in blocks 0–2The first three blocks attend over the whole context: 64 query heads share 4 KV heads of dim 128; no indexer, gate or sink.

Same as the blueprintDSA in every layer, not alternated with MLA: a lightning indexer (32 heads × 128 dims, ReLU scores with learned per-head weights) selects the top 2,048 tokens per query and MLA attends only to those. It was added after mid-training: 1,000 warm-up steps on 14 × 202,752-token sequences, then 20B tokens of sparse adaptation. The indexer is frozen in RL and uses a deterministic torch.topk. GLM-5.2 runs it in 21 of 78 layers.

MiniMax Sparse Attention (MSA)Per GQA group, a 2-projection indexer picks the top 16 of 128-token KV blocks; exact softmax over ≤ 2,048 keys, in 57 of 60 layers.

Same as the blueprintMuon, following GLM-4.5, with Muon Split: the MLA up-projections W_UQ, W_UK and W_UV are split per head and each block is orthogonalized separately, which brought MLA level with GQA-8 in Z.ai's ablation. The distributed implementation all-gathers only each rank's own shards. LR peaks at 2e-4, decays to 4e-5 in pretraining, then 1e-5 in mid-training.

Not disclosed

INT4 QAT in SFT, FP8 rolloutsINT4 quantization-aware training during SFT with a bitwise-identical kernel; FP8 inference for RL rollouts and release.

BF16 weights plus an official MXFP8 checkpointBF16 release with an FP32 router; MiniMax-M3-MXFP8 stores the text stack's linears as E4M3 with one 8-bit scale per 32 weights.

Same as the blueprintThe last post-training stage is on-policy cross-stage distillation: final checkpoints of earlier SFT and RL stages are teachers, and the advantage is replaced by the stop-gradient log-ratio log π_teacher − log π_train, with group size 1 and batch 1,024. GLM-5.2 merged more than ten expert models this way in about two days.

Not disclosed

Same as the blueprintThe shared-weight MTP layer drafts tokens for self-speculative decoding in serving and in RL rollouts, where small-batch decoding makes it most effective. GLM-5.2's MTP improvements raise acceptance length by up to 20%.

Not disclosed

More comparisons