FIELD GUIDE 04 / HYBRID SEQUENCES

Qwen3-Next 80B-A3B

A 48-layer hybrid that uses recurrent linear attention three times as often as full attention, then routes every layer through an ultra-sparse expert bank.

Scale
80B total · about 3B active
Shape
48 layers · width 2,048
Native context
262,144 tokens
Pattern
[Delta, Delta, Delta, full] ×12

THE 60-SECOND READ

Follow one text token.

  1. 01

    Three Gated DeltaNet layers update compact recurrent matrix states instead of forming full attention maps.

  2. 02

    A periodic gated full-attention layer restores direct access across the whole causal context.

  3. 03

    Every layer then uses one shared expert plus 10 routed experts selected from 512.

  4. 04

    Native Multi-Token Prediction adds training signals for more than the immediate next token; decoding support depends on the runtime.

OUTSIDE IN

Whole model

Twelve exact four-layer groups produce 48 layers. Each layer pairs its sequence mixer with a Mixture of Experts (MoE) block, so the pattern is better read as three DeltaNet → MoE layers followed by one full attention → MoE layer.

INPUTText tokensWidth 2,048
DeltaNet
→ MoEDeltaNet
→ MoEDeltaNet
→ MoEfull attention
→ MoE×12 groups
OUTPUTToken logits48 layers total
How to read this diagram: one exact four-layer group is rendered left to right and repeated 12 times. Every mixer is followed by expert computation; the group count and order are exact.On narrow screens, pan horizontally to inspect the full pipeline.

MECHANISM CUTAWAYS

Open the blocks

013:1 hybrid stackThree recurrent mixers, then one full-attention checkpoint.

Plain language. Most layers carry forward a compact running memory; every fourth layer performs a direct sequence-wide lookup.

Technical read. The exact schedule is [Gated DeltaNet, Gated DeltaNet, Gated DeltaNet, Gated Attention] × 12. The resulting 48 layers each feed the same 512-routed-plus-one-shared expert design.

Why it matters. Recurrent state makes most sequence mixing linear in sequence length, while periodic full attention repairs information that is difficult to compress into fixed state.

How to read this diagram: all four layer types are shown in execution order for one group; the multiplier applies to the entire group, not to an individual block.
02Gated DeltaNet state updateCorrect memory by writing the value error.

Plain language. The layer asks what value its memory currently associates with key k, measures the error against the new value v, writes a gated correction, then reads the updated memory with query q.

Technical read. The conceptual path uses 16 query (Q) / key (K) heads, 32 value (V) / state / output-gate heads, head width 128, and a width-4 depthwise causal convolution. Each state head maintains a fixed 128×128 matrix. I use a column-vector convention where state S maps a key to a value: read Sk, then write error ⊗ key. The transposed row-vector convention writes key ⊗ error; both describe the same update orientation. Output is Root Mean Square normalized (RMS-normalized), gated by the Sigmoid Linear Unit (SiLU) SiLU(z), and projected.

Why it matters. State size stays fixed as the sequence grows, and the delta rule edits memory toward the new key/value association rather than only accumulating it.

CONCEPTUAL SIGNAL FLOW · EXACT COUNTS · COLUMN-VECTOR S:key→value CONVENTION

16 Q/K heads32 V/state/gate headshead width 128causal conv width 432 states of 128×128
  1. 1decay old stateS′ = decay · S
  2. 2read current associationv̂ = S′k
  3. 3measure value errore = v − v̂
  4. 4gate the correctionβe
  5. 5write error ⊗ keyS″ = S′ + (βe)kT
  6. 6read with queryy = S″q
RMS normalize y → gate with SiLU(z) → output projection
How to read this diagram: the numbered operations teach the recurrence order under the stated column-vector convention; they do not claim to be the implementation’s exact fused computation graph. Counts come from the official configuration and reference implementation; the update mechanism follows Gated Delta Networks.On narrow screens, pan horizontally to inspect all six update operations.
03Periodic Gated Attention with grouped queriesSixteen queries share two key/value pairs.

Plain language. Full-attention layers keep 16 ways to ask questions while sharing just two sets of remembered keys and values.

Technical read. Grouped-Query Attention (GQA) uses 16 query heads and 2 key/value (KV) heads, dimension 256. Each KV pair serves exactly eight Q heads. Rotary Position Embedding (RoPE) applies to the first 25% of a head—64 of 256 dimensions. Query/key normalization (QK-Norm) and a learned sigmoid output gate are separate operations.

Why it matters. Periodic full attention restores direct global lookup, while 8-to-1 KV sharing reduces its cache and the output gate controls how much of each head returns to the residual stream.

EXACT MAP · 16 QUERY HEADS · 2 KEY/VALUE PAIRS · 8 Q PER PAIR

PARTIAL RoPE64 position dims + 192 content-only dims = 256/headQK-NORMnormalize queries and keysOUTPUT GATElearned sigmoid gate after attention
How to read this diagram: each KV pair connects to exactly eight rendered query heads. The annotation row names three additional operations without pretending they are extra heads.
04Ultra-sparse expert routingOne shared expert plus 10 selected from 512.

Plain language. Every token gets one common transformation and ten specialist transformations. The routed branch turns all 512 scores into probabilities before choosing and rebalancing the top ten.

Technical read. The router applies softmax across all 512 routed experts, selects the top 10 probabilities, then renormalizes those selected probabilities because norm_topk_prob=true. Their weighted expert outputs form the routed sum. A separate shared expert always runs; its output is multiplied by its own learned sigmoid gate before the shared and routed sums are added. Routed expert intermediate width is 512.

Why it matters. Renormalization makes the selected routed weights sum to one, while the independent sigmoid gate controls the always-computed shared path rather than participating in top-10 selection.

How to read this diagram: softmax covers the complete routed bank before top-10 selection, and the selected probabilities are renormalized before they weight expert outputs. The shared expert is outside that competition and has its own sigmoid gate.On narrow screens, pan horizontally to inspect every routing stage.
05Native Multi-Token PredictionConfirmed training design; runtime behavior varies.

Plain language. Training asks the system to predict multiple future steps, adding a learning signal beyond the immediate next token.

Technical read. Official Qwen material confirms native Multi-Token Prediction (MTP) and multi-step training. The released checkpoint exposes a one-block MTP module. The sidecar flow below is inferred from official checkpoint keys and the Qwen/Hugging Face reference, and is corroborated by production inference implementations; it is not specified as a universal decoding interface.

Why it matters. Multi-step targets can improve representation learning and may support speculative generation, but an installed runtime may ignore, discard, or differently expose the released MTP path.

IMPLEMENTATION-INFERRED FLOW · NOT UNIVERSAL RUNTIME BEHAVIOR

main hidden statefrom base modelfuture-token embeddingtraining input
MTP projection
1 MTP block
additional future logits
How to read this diagram: this is a cautious reconstruction of the released sidecar from official keys and implementations, not a claim that every serving engine exposes the same forward path.On narrow screens, pan horizontally to inspect the full inferred path.

READ THE ORIGINALS

Primary sources

The diagrams on this page are original teaching renderings. They simplify layout to make relationships readable; the linked paper, configuration, and implementation remain authoritative for exact computation and checkpoint behavior.