FIELD GUIDE 03 / EFFICIENT SCALE

DeepSeek V3

A 671B sparse Transformer that compresses its attention cache into a shared latent state and routes tokens through one shared plus eight selected experts.

Main model
671B total · 37B active
Shape
61 layers · width 7,168
Attention
128-head latent attention
Routing
1 shared + top 8 / 256

THE 60-SECOND READ

Follow one text token.

  1. 01

    Multi-head Latent Attention compresses content memory to 512 values and keeps a separate 64-value position key.

  2. 02

    The first three layers use dense feed-forward networks; the next 58 use fine-grained sparse experts.

  3. 03

    Every sparse layer runs one shared expert and eight routed experts selected from 256.

  4. 04

    A separately released sequential prediction module can learn one farther-ahead token beyond the main next-token path.

OUTSIDE IN

Whole model

Multi-head Latent Attention (MLA) appears throughout the stack. Dense feed-forward network (FFN) layers establish the first three blocks, then 58 DeepSeek Mixture of Experts (DeepSeekMoE) layers supply sparse capacity. The released Multi-Token Prediction (MTP) weights are a separate module beyond the 671B main model.

INPUTText tokensWidth 7,168
LAYERS 1–3MLA + dense FFN3 dense blocks
LAYERS 4–61MLA + DeepSeekMoE58 sparse blocks
MAIN STATEhidden hishared by both outcomes
shared output head→ next-token logits+ actual embedding ei+1→ separate sequential MTP → token i+2
How to read this diagram: the 671B main hidden state feeds the ordinary shared output head. The separately released ~14B MTP continuation also consumes the actual next-token embedding before predicting one step farther; it is not fed by next-token logits. FFN type changes after layer 3.On narrow screens, pan horizontally to inspect the full pipeline.

MECHANISM CUTAWAYS

Open the blocks

01Multi-head Latent AttentionCache compressed content and position separately.

Plain language. Instead of caching a separate full key and value for all 128 heads, MLA remembers one compact content vector plus one compact position key per token and reconstructs per-head views when needed.

Technical read. Queries pass through a low-rank adaptation (LoRA) bottleneck of rank 1,536, then expand per head into 128 content dimensions plus 64 Rotary Position Embedding (RoPE) dimensions. The key/value (KV) path stores a shared 512-dimensional content latent and a separate shared 64-dimensional RoPE key: exactly 576 cached values per token per layer. Content expands to 128-dimensional keys and 128-dimensional values for each of 128 heads.

Why it matters. Cache cost follows the compact shared state rather than 128 complete K/V head streams. This is latent reconstruction, not Grouped-Query Attention.

EXACT CACHE · 512 CONTENT + 64 POSITION = 576 VALUES / TOKEN / LAYER

How to read this diagram: follow the query and KV rows independently. Content boxes and striped position boxes remain separate throughout; only the two highlighted shared boxes constitute the autoregressive cache.On narrow screens, pan horizontally to inspect both full paths.
02Fine-grained expert routingAlways use one shared expert and top 8 of 256 routed experts.

Plain language. Every token follows a common expert path. Its specialist path is screened twice: first four of eight expert groups become eligible, then eight individual experts are chosen from those groups.

Technical read. DeepSeekMoE computes original sigmoid affinities for 256 routed experts arranged as 8 groups of 32. For selection only, it adds learned correction biases; each group is scored by its two strongest bias-adjusted expert scores, and the top 4 groups remain eligible. The router then selects the top 8 bias-adjusted experts inside those groups. Mixture weights come from the selected experts’ original, unbiased sigmoid affinities, renormalized to sum to one and scaled by 2.5. One additional shared expert always runs, and no token is dropped.

Why it matters. Selection bias can balance routing without changing the weights used to blend expert outputs, while group screening limits where a token travels and the shared expert preserves a universal transformation.

How to read this diagram: the dashed boxes change eligibility and ranking only. After four groups and eight experts are selected, the weighting path returns to the original unbiased sigmoid affinities before normalization and the 2.5 scale; the separate shared lane always executes.On narrow screens, pan horizontally to inspect every routing stage.
03Dense-to-sparse depthChange feed-forward strategy after layer 3.

Plain language. The bottom of the network uses ordinary dense feed-forward computation; specialization starts only after three layers.

Technical read. All 61 blocks use MLA. Layers 1–3 pair it with dense FFNs; layers 4–61 pair it with DeepSeekMoE, yielding 58 sparse blocks.

Why it matters. Early dense layers establish shared low-level representations before most depth branches into expert-specialized transformations.

How to read this diagram: the three dense blocks are rendered individually; the long labeled bar covers every sparse layer from 4 through 61.
04Sequential Multi-Token PredictionUse a separate module to learn token i+2.

Plain language. The extra module combines what the main model knows at position i with the actual embedding of the next token, then learns to predict one token farther ahead.

Technical read. Multi-Token Prediction (MTP) applies separate Root Mean Square Layer Normalizations (RMSNorms) to the main hidden state at i and token embedding at i+1, concatenates them to width 2d, projects 2d → d, passes through one Transformer block, and uses the shared output head to predict i+2.

Why it matters. The approximately 14B released MTP weights are separate from the 671B main model. They can be discarded for ordinary inference or used as a speculative draft path where supported.

SEQUENTIAL RELEASED MODULE · SEPARATE ~14B WEIGHTS

main hidden hiRMSNormactual embedding ei+1separate RMSNorm
concatenate2d
projection2d → d
1 Transformer block
shared output head
predict token i+2
How to read this diagram: two separately normalized inputs merge before one sequential prediction block. This is not a parallel sidecar head on the main representation.On narrow screens, pan horizontally to inspect the full sequential path.

READ THE ORIGINALS

Primary sources

The diagrams on this page are original teaching renderings. They simplify layout to make relationships readable; the linked paper, configuration, and implementation remain authoritative for exact computation and checkpoint behavior.