FIELD GUIDE 02 / OPEN REASONING
gpt-oss 20B
An open-weight sparse decoder that shares attention memory aggressively and routes each token through four of 32 feed-forward experts.
- Scale
- 20.91B total · 3.61B active
- Shape
- 24 layers · width 2,880
- Context
- 131,072 tokens
- Signature
- 64 query / 8 key / 8 value heads
THE 60-SECOND READ
Follow one text token.
- 01
Sixty-four query heads ask different questions, but every group of eight shares one key head and one value head.
- 02
Local 128-token attention and full causal attention alternate across 24 layers.
- 03
A router scores 32 experts and activates the top four for this token.
- 04
Residual paths carry the result forward until the model emits next-token logits.
OUTSIDE IN
Whole model
The model repeats a pre-normalized decoder block 24 times. Its attention modes follow [local, global] × 12, and every block contains sparse expert computation.
MECHANISM CUTAWAYS
Open the blocks
01Grouped-Query AttentionEight queries share each key/value pair.
Plain language. A query (Q) is what a token is looking for; a key (K) says what a past token can be matched on; a value (V) is the information returned when the match is strong.
Technical read. Grouped-Query Attention (GQA) has 64 query heads, 8 key heads, and 8 value heads, all dimension 64. Each K/V pair is shared by exactly eight Q heads.
Why it matters. The model keeps many query perspectives while storing one-eighth as many K/V head streams as full multi-head attention with 64 independent pairs.
EXACT MAP · 64 QUERY HEADS · 8 KEY/VALUE PAIRS · 8 Q PER PAIR
GROUP 1
GROUP 2
GROUP 3
GROUP 4
GROUP 5
GROUP 6
GROUP 7
GROUP 8
02Banded versus full attentionAlternate nearby reading with sequence-wide reading.
Plain language. Local layers look back only through a recent window; the next global layer can look back through the entire available history.
Technical read. Local causal attention uses a 128-token sliding band. Global attention uses the full lower-triangular causal field. The 24-layer schedule alternates these modes.
Why it matters. Local layers reduce attention work, while regular global layers prevent information from remaining confined to moving neighborhoods.
SCHEMATIC MASKS · EXACT LOCAL WIDTH = 128 TOKENS
128-token causal band
Only a moving recent window is visible.
Full causal field
Every earlier token is visible.
03Sparse Mixture of ExpertsUse four specialists from a bank of 32.
Plain language. A router chooses four feed-forward specialists for each token, then blends their answers.
Technical read. The Mixture of Experts (MoE) router scores all 32 Swish-Gated Linear Unit (SwiGLU) experts, selects the top four, normalizes those selected scores, and forms a weighted sum. Each expert has intermediate width 2,880.
Why it matters. All 32 experts contribute to model capacity, but only four contribute expert compute for one token—part of why 20.91B total becomes 3.61B active.
present in capacity, inactive for this token
04One decoder blockResidual attention, then residual expert computation.
Plain language. Each layer first mixes information across tokens, then transforms each token through selected experts, with skip routes preserving the running representation.
Technical read. Root Mean Square Layer Normalization (RMSNorm) is applied before attention and before MoE. Each sublayer is added back to the residual stream. The attention implementation switches local/global mode by layer index.
Why it matters. Pre-normalization and residual routes stabilize a deep stack, while changing only the mask lets one block design support both attention ranges.
CONCEPTUAL RESIDUAL FLOW
05Position, sinks, and precisionThree details that change attention and storage.
Plain language. Position rotation tells attention where tokens sit, a sink lets a head choose “none of these values,” and compact expert weights reduce storage.
Technical read. Rotary Position Embedding (RoPE) rotates Q and K. Yet another RoPE extensioN (YaRN) uses factor 32 to blend frequency interpolation and extrapolation from 4,096 to 131,072 positions. Every query head has a learned sink logit whose attention probability contributes no value. Microscaling 4-bit floating point (MXFP4) applies to released MoE weights, not every path.
Why it matters. These are different optimizations: long-context position, more expressive attention normalization, and lower expert-weight memory should not be collapsed into one feature label.
Q/K rotation
4,096 → 131,072
factor 32 blend
Can absorb probability
returns no value vector
MoE weights
not a blanket precision claim
READ THE ORIGINALS
Primary sources
- OpenAI gpt-oss introduction Release scope and model positioning.
- Official gpt-oss architecture model card Primary architecture and training description.
- Official reference implementation Head sharing, sinks, routing, and block flow.
- Official original configuration Released checkpoint dimensions.
The diagrams on this page are original teaching renderings. They simplify layout to make relationships readable; the linked paper, configuration, and implementation remain authoritative for exact computation and checkpoint behavior.