Inkling

5:1 sliding-window/global GQA with a learned relative-position bias instead of RoPE, short causal convs, encoder-free image and audio.

Compare with another model

Parameters: 975B total, 41B active
Context: 1M tokens (1,048,576)
Depth: 66 blocks
Weights: Open, Apache-2.0
Released: July 2026

--- frontier-llm@2026-11+++ inkling@2026-07+0~5−3

The November 2026 LLM blueprint, marked with Inkling's differences: 18 components the same, 5 done differently, 3 not used. Select any component.

AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Done differently

Short causal convolutions (SConv)

Attention

Depthwise causal convs (kernel 4, with skip) on K, V and on both sublayer outputs before they rejoin the residual.

How Inkling does it

Every block contains four depthwise causal 1-D convolutions of kernel 4, each wrapped in its own skip connection: y = x + conv₄(x). They have no bias and run in FP32.

K and V: on the key and value projections in every attention layer, before k-norm and before caching.
Attention output: after W_o, before the residual add (this box's position).
FFN output: after the MoE (or dense MLP), before its residual add.

Each sublayer therefore writes conv(F(x)) rather than F(x) into the residual stream. A token's update can include a learned per-channel mix of the updates of the previous three tokens. There is no output gate and no attention-sink logit.

Serving cost: each conv keeps a rolling 3-token state per layer (four states per block). Inference engines must cache and checkpoint these alongside the KV cache for prefix caching and speculative-decoding rollback. In the NVFP4 checkpoint all convolutions stay unquantized.

The blueprint's design: Gated attention output & learnable sinks

Softmax forces every head to put total weight 1 somewhere. Heads with nothing to attend to dump it on a sink token, creating massive activations and quantization outliers. Two cheap fixes now ship widely: a query-dependent sigmoid gate on each head's output (Qwen3.5/3.6, Kimi K3 gated MLA, KDA) and a learnable per-head sink logit in the softmax denominator (gpt-oss, DeepSeek-V4).

Read the full explanation

Other models

DeepSeek-V4.1-Flash: same as the blueprint
Kimi K3: same as the blueprint
Nemotron 3 Ultra: not used by this model
DeepSeek-V4-Pro: same as the blueprint
Gemma 4 31B: not used by this model
MiniMax-M2.7: not used by this model
GLM-5: not used by this model
Qwen3.5-397B-A17B: same as the blueprint
gpt-oss-120b: same as the blueprint