Built-in Codex tools were disabled. Harness execution is separately permitted only in Experiments 3 and 6. Text below is taken from saved records; it is not a new model run. Reasoning blocks are the summaries actually emitted by the CLI.
Answer the user's request.
Exact prompt · Raw CLI events · Settings and audit
Below is a small character-level language model: a 4-layer transformer with 6 attention heads per layer, which reads up to 128 characters and outputs scores (logits) for the next character. This is exactly how it computes (numpy):
```python
import numpy as np
VOCAB = "\n " + "abcdefghijklmnopqrstuvwxyz" # index 0 = newline, 1 = space, 2..27 = a..z
HEADS, LAYERS = 6, 4
def layer_norm(x, w, b): # over the last axis, eps 1e-5
mu = x.mean(-1, keepdims=True)
var = ((x - mu) ** 2).mean(-1, keepdims=True)
return (x - mu) / np.sqrt(var + 1e-5) * w + b
def next_char_logits(text, p):
"""text: up to 128 characters (only the last 128 are used); p: dict of the weight arrays.
Returns the 28 logits for the character that comes next."""
ids = [VOCAB.index(c) for c in text[-128:]]
n, d = len(ids), p["tok_emb"].shape[1]
hd = d // HEADS
x = p["tok_emb"][ids] + p["pos_emb"][:n] # (n, d); positions 0..n-1 from the start of the text
for l in range(LAYERS): # layer l uses the tensors named <name>.<l>
h = layer_norm(x, p[f"ln1_w.{l}"], p[f"ln1_b.{l}"])
q, k, v = h @ p[f"W_q.{l}"].T, h @ p[f"W_k.{l}"].T, h @ p[f"W_v.{l}"].T
out = np.zeros((n, d))
for j in range(HEADS): # head j uses dimensions j*hd .. (j+1)*hd - 1
s = slice(j * hd, (j + 1) * hd)
att = q[:, s] @ k[:, s].T / np.sqrt(hd) # (n, n)
att = att + np.triu(np.full((n, n), -np.inf), 1) # causal: position i sees positions 0..i
att = np.exp(att - att.max(-1, keepdims=True))
att = att / att.sum(-1, keepdims=True)
out[:, s] = att @ v[:, s]
x = x + out @ p[f"W_o.{l}"].T
h = layer_norm(x, p[f"ln2_w.{l}"], p[f"ln2_b.{l}"])
x = x + np.maximum(0.0, h @ p[f"W_fc.{l}"].T) @ p[f"W_proj.{l}"].T # ReLU MLP, hidden 4 d
h = layer_norm(x, p["lnf_w"], p["lnf_b"])
return h[-1] @ p["W_out"].T + p["b_out"] # logits of the next character (28)
```
Training: from random initialization (every matrix entry drawn from a normal distribution with standard deviation 0.02, or 0.007071 for W_o and W_proj; LayerNorm weights 1 and biases 0; b_out 0), next-character prediction with AdamW (30,517 steps of 256 random 128-character windows, 1,000,000,000 characters in total; learning-rate warm-up then cosine decay; weight decay 0.01 on the matrices).
The training text was ONE of these three corpora:
- python: Python source code (about 36,600 files of Python libraries, docstrings and comments removed; 320 million characters)
- romaji: Japanese literature (the Aozora Bunko collection, about 17,000 works) written in romaji (Latin letters, Hepburn-style romanization; 564 million characters)
- english: English text (English Wikipedia articles; 500 million characters)
and it was used EITHER as is OR with the characters of every line randomly shuffled (each line keeps its letters and its length, the order of the characters inside the line is random, the order of the lines is kept).
Normalization (the same for all three corpora, before anything else): lowercased; in each line the leading spaces (indentation) were kept, every other character that is not a letter a-z became a space, runs of spaces were collapsed to one and trailing spaces removed. So the text contains only newlines, spaces and the letters a-z (the 28 characters of VOCAB).
WHAT YOU CAN USE. The weights inside the four transformer blocks, exactly as trained (float32): for layer l = 0, 1, 2 and 3, ln1_w.l, ln1_b.l, W_q.l, W_k.l, W_v.l, W_o.l (attention) and ln2_w.l, ln2_b.l, W_fc.l, W_proj.l (MLP). The input layer (tok_emb, the 28 x d character embeddings, and pos_emb, the 128 x d position embeddings) and the output layer (lnf_w, lnf_b, W_out, b_out) are NOT available. The d = 384 dimensions of the residual stream carry no labels. Head j of layer l uses rows 64j .. 64j+63 of W_q.l, W_k.l and W_v.l and columns 64j .. 64j+63 of W_o.l.
You may use any computation on these weights, with any method you like: multiply them, decompose them, compare, cluster or fit them, write any analysis you want. Two things are NOT allowed:
1. A forward pass: do not run the network or any of its layers (LayerNorm, attention with its softmax, the MLP with its ReLU) on any input, whether real, random or constructed, in code, in your head or on paper. Treating the weight matrices as linear-algebra objects (products, eigenvalues, singular vectors, applying one weight matrix to directions taken from other weights) is fine; simulating what the network does to an input is not.
2. Search: do not look anything up (there is no internet access in any case).
HOW YOU COMPUTE. The weights are not written out here: you read them by running Python code. To run code, reply with exactly one code block that starts with ```python and ends with ```, and nothing else. We run it and reply with its printed output. Available: numpy, scipy, scikit-learn, PyTorch (CPU), pandas, sympy, networkx. Before your code runs, numpy is imported as np and W is a dict of the weight arrays by name: W['W_q.0'] is 384 x 384, W['W_fc.0'] is 1536 x 384, W['W_proj.0'] is 384 x 1536, W['ln1_w.0'] has 384 entries, and so on; matrices are (out, in), as used in the code above. There is nothing else to read (no other files, no network). Each run is a fresh process (nothing is kept between runs), may take up to 600 seconds, and its output is cut at 20,000 characters. You may run code up to 100 times. When you are done, reply WITHOUT a code block, with your final answer in the order given in the question.
QUESTION. What was this network trained on, and how does it use the context? Answer in this order:
Observations: <what you see in the shown weights and what it suggests; quote the weights you rely on>
Heads: <one line per head, L0H0 .. L0H5, L1H0 .. L1H5, L2H0 .. L2H5, L3H0 .. L3H5: what it most likely does, as one of: 'previous' (attends to the previous character), 'local' (a few characters back), 'broad' (spread far back), 'copy' (induction: attends to the character right after an earlier occurrence of the current character or context, so the network can repeat earlier text), or 'unclear'; then your evidence>
Reasoning: <which training text fits these observations, and why>
corpus: <python | romaji | english>
order: <as is | shuffled>
probabilities: python=<p> romaji=<p> english=<p>; as is=<p> shuffled=<p>
Equal probabilities are not allowed. Even if the evidence is weak, give your best judgement.Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import scipy.linalg as la
from threadpoolctl import threadpool_limits
threadpool_limits(4)
def norm(x):
return float(la.norm(x))
print("All calculations below are weight-only linear algebra; no network evaluations.")
for l in range(4):
print("\nLAYER", l)
for name in ["ln1_w", "ln1_b", "ln2_w", "ln2_b"]:
a = W[f"{name}.{l}"].astype(np.float64)
print(name, "min/mean/max/std/norm",
np.round([a.min(), a.mean(), a.max(), a.std(), norm(a)], 5))
for name in ["W_q", "W_k", "W_v", "W_o", "W_fc", "W_proj"]:
a = W[f"{name}.{l}"].astype(np.float64)
s = la.svdvals(a)
print(name, "Frob", round(norm(a), 4),
"top8_sv", np.round(s[:8], 4),
"stable_rank", round(np.sum(s*s)/(s[0]*s[0]), 3))
print("HEAD: QK_frob QK_top4_sv QK_top1_energy QK_sym_fraction QK_trace; "
"OV_frob OV_top4_sv OV_trace OV_eig_posmax OV_eig_negmin")
for j in range(6):
sl = slice(64*j, 64*(j+1))
q = W[f"W_q.{l}"][sl].astype(np.float64)
k = W[f"W_k.{l}"][sl].astype(np.float64)
v = W[f"W_v.{l}"][sl].astype(np.float64)
o = W[f"W_o.{l}"][:, sl].astype(np.float64)
a = q.T @ k
b = o @ v
sa, sb = la.svdvals(a)[:64], la.svdvals(b)[:64]
eb = la.eigvals(v @ o)
print(f"L{l}H{j}",
round(norm(a), 4), np.round(sa[:4], 4),
round(sa[0]**2/np.sum(sa**2), 4),
round(norm((a+a.T)/2)/norm(a), 4), round(np.trace(a), 4),
";", round(norm(b), 4), np.round(sb[:4], 4),
round(np.trace(b), 4), round(eb.real.max(), 4),
round(eb.real.min(), 4))
```
{
"t": 1792713196.3272984,
"s": 19.67,
"code": "import numpy as np\nimport scipy.linalg as la\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\ndef norm(x):\n return float(la.norm(x))\n\nprint(\"All calculations below are weight-only linear algebra; no network evaluations.\")\nfor l in range(4):\n print(\"\\nLAYER\", l)\n for name in [\"ln1_w\", \"ln1_b\", \"ln2_w\", \"ln2_b\"]:\n a = W[f\"{name}.{l}\"].astype(np.float64)\n print(name, \"min/mean/max/std/norm\",\n np.round([a.min(), a.mean(), a.max(), a.std(), norm(a)], 5))\n for name in [\"W_q\", \"W_k\", \"W_v\", \"W_o\", \"W_fc\", \"W_proj\"]:\n a = W[f\"{name}.{l}\"].astype(np.float64)\n s = la.svdvals(a)\n print(name, \"Frob\", round(norm(a), 4),\n \"top8_sv\", np.round(s[:8], 4),\n \"stable_rank\", round(np.sum(s*s)/(s[0]*s[0]), 3))\n print(\"HEAD: QK_frob QK_top4_sv QK_top1_energy QK_sym_fraction QK_trace; \"\n \"OV_frob OV_top4_sv OV_trace OV_eig_posmax OV_eig_negmin\")\n for j in range(6):\n sl = slice(64*j, 64*(j+1))\n q = W[f\"W_q.{l}\"][sl].astype(np.float64)\n k = W[f\"W_k.{l}\"][sl].astype(np.float64)\n v = W[f\"W_v.{l}\"][sl].astype(np.float64)\n o = W[f\"W_o.{l}\"][:, sl].astype(np.float64)\n a = q.T @ k\n b = o @ v\n sa, sb = la.svdvals(a)[:64], la.svdvals(b)[:64]\n eb = la.eigvals(v @ o)\n print(f\"L{l}H{j}\",\n round(norm(a), 4), np.round(sa[:4], 4),\n round(sa[0]**2/np.sum(sa**2), 4),\n round(norm((a+a.T)/2)/norm(a), 4), round(np.trace(a), 4),\n \";\", round(norm(b), 4), np.round(sb[:4], 4),\n round(np.trace(b), 4), round(eb.real.max(), 4),\n round(eb.real.min(), 4))\n",
"out": "All calculations below are weight-only linear algebra; no network evaluations.\n\nLAYER 0\nln1_w min/mean/max/std/norm [-0.001 0.2403 1.0341 0.2436 6.7051]\nln1_b min/mean/max/std/norm [-0.0959 0.0008 0.091 0.0287 0.5635]\nln2_w min/mean/max/std/norm [ 0.0044 0.8006 1.5072 0.3295 16.9648]\nln2_b min/mean/max/std/norm [-0.4986 0.0022 0.3331 0.0683 1.3392]\nW_q Frob 32.2912 top8_sv [8.2519 8.2232 7.963 7.819 7.68 7.6416 7.5194 7.2054] stable_rank 15.313\nW_k Frob 30.3909 top8_sv [7.6464 7.4363 7.3304 6.9118 6.8285 6.6716 6.4724 6.4464] stable_rank 15.797\nW_v Frob 14.836 top8_sv [4.339 3.5292 3.2699 3.1656 2.9824 2.6465 2.589 2.5519] stable_rank 11.691\nW_o Frob 16.3713 top8_sv [7.1033 6.6074 4.2683 2.7702 2.5025 2.285 2.2638 2.2184] stable_rank 5.312\nW_fc Frob 46.2486 top8_sv [18.1759 11.0178 8.7366 7.4517 7.2123 7.0621 7.0112 6.6204] stable_rank 6.474\nW_proj Frob 39.3075 top8_sv [9.8257 7.3508 6.2651 5.8693 5.7965 5.6574 5.5163 5.4501] stable_rank 16.004\nHEAD: QK_frob QK_top4_sv QK_top1_energy QK_sym_fraction QK_trace; OV_frob OV_top4_sv OV_trace OV_eig_posmax OV_eig_negmin\nL0H0 3.0479 [2.5097 0.8949 0.6966 0.4393] 0.678 0.7267 -1.7245 ; 12.9041 [10.2524 5.6035 3.4749 2.8184] -0.1807 1.1732 -0.4325\nL0H1 1.8376 [1.0971 0.7651 0.4518 0.343 ] 0.3564 0.7472 -1.2358 ; 10.1485 [7.662 5.1724 2.5819 2.0305] -0.7493 0.9812 -0.9241\nL0H2 182.4351 [59.8786 57.2786 55.5815 54.3474] 0.1077 0.6963 32.7753 ; 13.223 [4.0009 3.853 3.7432 3.5437] -4.6421 0.4428 -0.6979\nL0H3 11.3515 [6.568 5.9931 3.2141 2.6459] 0.3348 0.7629 -7.9874 ; 7.3079 [4.305 4.0456 2.4933 2.078 ] -2.0478 0.262 -0.6907\nL0H4 6.6069 [6.0218 2.0421 1.078 0.5566] 0.8307 0.7629 -4.7252 ; 7.3117 [4.8786 3.2357 2.5484 2.0826] -0.9671 0.3599 -0.4665\nL0H5 2.5787 [2.1893 0.543 0.3941 0.328 ] 0.7208 0.7288 -0.9664 ; 13.0384 [10.3142 5.4678 3.1482 2.9348] -0.0858 0.827 -0.41\n\nLAYER 1\nln1_w min/mean/max/std/norm [0.1361 0.4575 1.1239 0.1022 9.1868]\nln1_b min/mean/max/std/norm [-1.4712 -0.005 1.1544 0.1672 3.2771]\nln2_w min/mean/max/std/norm [-0.0019 1.0429 1.4856 0.2003 20.8093]\nln2_b min/mean/max/std/norm [-0.3597 0.0005 0.3686 0.1364 2.6732]\nW_q Frob 32.3126 top8_sv [9.9032 7.7974 6.7042 5.9516 5.6721 5.6025 5.4021 5.3125] stable_rank 10.646\nW_k Frob 29.2506 top8_sv [13.3072 9.9327 5.8989 5.677 5.0458 4.622 4.4903 4.4044] stable_rank 4.832\nW_v Frob 27.3529 top8_sv [5.2845 4.9661 4.87 4.7328 4.6743 4.5853 4.4113 4.3216] stable_rank 26.792\nW_o Frob 25.9233 top8_sv [4.9657 3.2412 3.157 3.1119 3.0615 3.0531 2.9942 2.9545] stable_rank 27.253\nW_fc Frob 79.1265 top8_sv [21.4881 13.1349 12.8042 12.6247 12.3279 12.1184 11.8504 11.6908] stable_rank 13.56\nW_proj Frob 75.7103 top8_sv [8.1935 7.8941 7.1992 6.9972 6.671 6.4329 6.2745 6.1333] stable_rank 85.384\nHEAD: QK_frob QK_top4_sv QK_top1_energy QK_sym_fraction QK_trace; OV_frob OV_top4_sv OV_trace OV_eig_posmax OV_eig_negmin\nL1H0 23.6654 [12.6145 11.6953 8.2823 7.0164] 0.2841 0.7351 -26.1053 ; 21.7066 [8.0989 7.5796 6.3982 6.2019] -21.6005 1.6279 -2.7755\nL1H1 35.5126 [19.649 17.056 10.7527 9.5417] 0.3061 0.7424 -44.9928 ; 22.226 [6.5833 6.0091 5.7392 5.681 ] -28.4652 0.5693 -2.0517\nL1H2 41.7812 [23.3819 18.3356 12.0604 11.2132] 0.3132 0.7448 -60.5649 ; 19.5165 [6.1346 5.7881 5.2517 4.9869] -21.8959 0.2894 -1.708\nL1H3 36.8369 [21.1075 17.3804 11.6312 9.1371] 0.3283 0.7402 -44.4839 ; 21.4133 [7.7675 6.8644 6.5273 6.0604] -20.2784 0.2159 -1.8799\nL1H4 37.3819 [22.9586 17.425 11.442 8.6811] 0.3772 0.7366 -46.5798 ; 22.0658 [6.4984 6.2533 5.8966 5.7835] -26.577 0.1462 -2.0645\nL1H5 36.2112 [20.9422 18.3428 10.3483 8.4275] 0.3345 0.7295 -39.7087 ; 22.4638 [6.2081 6.0193 5.803 5.6451] -29.2909 0.2127 -2.1146\n\nLAYER 2\nln1_w min/mean/max/std/norm [ 0.2078 0.7052 1.0144 0.0749 13.8969]\nln1_b min/mean/max/std/norm [-1.3819 -0.001 1.0527 0.1525 2.9893]\nln2_w min/mean/max/std/norm [ 0.0011 1.3123 1.4655 0.1255 25.8329]\nln2_b min/mean/max/std/norm [-0.3761 0.005 0.3886 0.1537 3."
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 1 of at most 100: ``` All calculations below are weight-only linear algebra; no network evaluations. LAYER 0 ln1_w min/mean/max/std/norm [-0.001 0.2403 1.0341 0.2436 6.7051] ln1_b min/mean/max/std/norm [-0.0959 0.0008 0.091 0.0287 0.5635] ln2_w min/mean/max/std/norm [ 0.0044 0.8006 1.5072 0.3295 16.9648] ln2_b min/mean/max/std/norm [-0.4986 0.0022 0.3331 0.0683 1.3392] W_q Frob 32.2912 top8_sv [8.2519 8.2232 7.963 7.819 7.68 7.6416 7.5194 7.2054] stable_rank 15.313 W_k Frob 30.3909 top8_sv [7.6464 7.4363 7.3304 6.9118 6.8285 6.6716 6.4724 6.4464] stable_rank 15.797 W_v Frob 14.836 top8_sv [4.339 3.5292 3.2699 3.1656 2.9824 2.6465 2.589 2.5519] stable_rank 11.691 W_o Frob 16.3713 top8_sv [7.1033 6.6074 4.2683 2.7702 2.5025 2.285 2.2638 2.2184] stable_rank 5.312 W_fc Frob 46.2486 top8_sv [18.1759 11.0178 8.7366 7.4517 7.2123 7.0621 7.0112 6.6204] stable_rank 6.474 W_proj Frob 39.3075 top8_sv [9.8257 7.3508 6.2651 5.8693 5.7965 5.6574 5.5163 5.4501] stable_rank 16.004 HEAD: QK_frob QK_top4_sv QK_top1_energy QK_sym_fraction QK_trace; OV_frob OV_top4_sv OV_trace OV_eig_posmax OV_eig_negmin L0H0 3.0479 [2.5097 0.8949 0.6966 0.4393] 0.678 0.7267 -1.7245 ; 12.9041 [10.2524 5.6035 3.4749 2.8184] -0.1807 1.1732 -0.4325 L0H1 1.8376 [1.0971 0.7651 0.4518 0.343 ] 0.3564 0.7472 -1.2358 ; 10.1485 [7.662 5.1724 2.5819 2.0305] -0.7493 0.9812 -0.9241 L0H2 182.4351 [59.8786 57.2786 55.5815 54.3474] 0.1077 0.6963 32.7753 ; 13.223 [4.0009 3.853 3.7432 3.5437] -4.6421 0.4428 -0.6979 L0H3 11.3515 [6.568 5.9931 3.2141 2.6459] 0.3348 0.7629 -7.9874 ; 7.3079 [4.305 4.0456 2.4933 2.078 ] -2.0478 0.262 -0.6907 L0H4 6.6069 [6.0218 2.0421 1.078 0.5566] 0.8307 0.7629 -4.7252 ; 7.3117 [4.8786 3.2357 2.5484 2.0826] -0.9671 0.3599 -0.4665 L0H5 2.5787 [2.1893 0.543 0.3941 0.328 ] 0.7208 0.7288 -0.9664 ; 13.0384 [10.3142 5.4678 3.1482 2.9348] -0.0858 0.827 -0.41 LAYER 1 ln1_w min/mean/max/std/norm [0.1361 0.4575 1.1239 0.1022 9.1868] ln1_b min/mean/max/std/norm [-1.4712 -0.005 1.1544 0.1672 3.2771] ln2_w min/mean/max/std/norm [-0.0019 1.0429 1.4856 0.2003 20.8093] ln2_b min/mean/max/std/norm [-0.3597 0.0005 0.3686 0.1364 2.6732] W_q Frob 32.3126 top8_sv [9.9032 7.7974 6.7042 5.9516 5.6721 5.6025 5.4021 5.3125] stable_rank 10.646 W_k Frob 29.2506 top8_sv [13.3072 9.9327 5.8989 5.677 5.0458 4.622 4.4903 4.4044] stable_rank 4.832 W_v Frob 27.3529 top8_sv [5.2845 4.9661 4.87 4.7328 4.6743 4.5853 4.4113 4.3216] stable_rank 26.792 W_o Frob 25.9233 top8_sv [4.9657 3.2412 3.157 3.1119 3.0615 3.0531 2.9942 2.9545] stable_rank 27.253 W_fc Frob 79.1265 top8_sv [21.4881 13.1349 12.8042 12.6247 12.3279 12.1184 11.8504 11.6908] stable_rank 13.56 W_proj Frob 75.7103 top8_sv [8.1935 7.8941 7.1992 6.9972 6.671 6.4329 6.2745 6.1333] stable_rank 85.384 HEAD: QK_frob QK_top4_sv QK_top1_energy QK_sym_fraction QK_trace; OV_frob OV_top4_sv OV_trace OV_eig_posmax OV_eig_negmin L1H0 23.6654 [12.6145 11.6953 8.2823 7.0164] 0.2841 0.7351 -26.1053 ; 21.7066 [8.0989 7.5796 6.3982 6.2019] -21.6005 1.6279 -2.7755 L1H1 35.5126 [19.649 17.056 10.7527 9.5417] 0.3061 0.7424 -44.9928 ; 22.226 [6.5833 6.0091 5.7392 5.681 ] -28.4652 0.5693 -2.0517 L1H2 41.7812 [23.3819 18.3356 12.0604 11.2132] 0.3132 0.7448 -60.5649 ; 19.5165 [6.1346 5.7881 5.2517 4.9869] -21.8959 0.2894 -1.708 L1H3 36.8369 [21.1075 17.3804 11.6312 9.1371] 0.3283 0.7402 -44.4839 ; 21.4133 [7.7675 6.8644 6.5273 6.0604] -20.2784 0.2159 -1.8799 L1H4 37.3819 [22.9586 17.425 11.442 8.6811] 0.3772 0.7366 -46.5798 ; 22.0658 [6.4984 6.2533 5.8966 5.7835] -26.577 0.1462 -2.0645 L1H5 36.2112 [20.9422 18.3428 10.3483 8.4275] 0.3345 0.7295 -39.7087 ; 22.4638 [6.2081 6.0193 5.803 5.6451] -29.2909 0.2127 -2.1146 LAYER 2 ln1_w min/mean/max/std/norm [ 0.2078 0.7052 1.0144 0.0749 13.8969] ln1_b min/mean/max/std/norm [-1.3819 -0.001 1.0527 0.1525 2.9893] ln2_w min/mean/max/std/norm [ 0.0011 1.3123 1.4655 0.1255 25.8329] ln2_b min/mean/max/std/norm [-0.3761 0.005 0.3886 0.1537 3.0128] W_q Frob 39.3455 top8_sv [5.7962 5.1962 4.8092 4.6159 4.6003 4.5145 4.4419 4.432 ] stable_rank 46.08 W_k Frob 37.7688 top8_sv [10.6627 8.6244 4.9887 4.3937 4.241 4.0878 3.9564 3.8475] stable_rank 12.547 W_v Frob 33.3048 top8_sv [3.7062 3.6245 3.5788 3.5334 3.5012 3.4679 3.4285 3.4153] stable_rank 80.753 W_o Frob 32.8397 top8_sv [4.1025 3.9708 3.558 3.3813 3.3611 3.2925 3.1918 3.1686] stable_rank 64.077 W_fc Frob 78.9668 top8_sv [15.3634 8.0212 7.8946 7.7247 7.5968 7.4905 7.2574 7.2158] stable_rank 26.419 W_proj Frob 76.8311 top8_sv [9.4036 9.1867 8.7234 8.4703 8.0398 7.7955 7.6145 7.4501] stable_rank 66.756 HEAD: QK_frob QK_top4_sv QK_top1_energy QK_sym_fraction QK_trace; OV_frob OV_top4_sv OV_trace OV_eig_posmax OV_eig_negmin L2H0 38.5746 [11.6585 9.9934 9.1292 8.7027] 0.0913 0.702 1.9543 ; 22.8142 [4.8981 4.6278 4.5529 4.3995] -54.2674 0.056 -2.1124 L2H1 33.1082 [9.2369 7.1417 7.0468 6.9344] 0.0778 0.7128 9.4557 ; 24.8446 [5.4684 5.1567 4.9683 4.7548] -50.1368 0.2665 -2.1816 L2H2 25.2291 [8.8799 7.5578 6.1827 5. ] 0.1239 0.7114 -16.2093 ; 28.7074 [7.8628 7.1802 6.9013 5.9893] -46.7717 0.8422 -2.9627 L2H3 32.8218 [8.5292 7.9451 7.259 7.0144] 0.0675 0.7059 -11.8331 ; 24.5055 [6.1803 5.7997 5.5394 4.9316] -49.7953 0.2369 -2.7194 L2H4 36.8275 [9.5029 8.8386 8.6143 8.1133] 0.0666 0.7045 -7.386 ; 23.2786 [5.0699 4.7745 4.6597 4.505 ] -45.5437 0.2805 -1.9452 L2H5 35.3545 [9.6053 8.5044 7.7528 7.5417] 0.0738 0.7043 -10.2817 ; 23.3416 [4.8404 4.6932 4.4898 4.4832] -50.7511 0.2453 -2.3041 LAYER 3 ln1_w min/mean/max/std/norm [ 0.2944 0.7546 0.8701 0.0542 14.8247] ln1_b min/mean/max/std/norm [-0.6069 0.0121 0.6502 0.0849 1.6801] ln2_w min/mean/max/std/norm [ 0.4364 1.2778 1.4306 0.0808 25.0891] ln2_b min/mean/max/std/norm [-0.6728 0.0043 0.3068 0.1276 2.5016] W_q Frob 32.9482 top8_sv [4.5598 4.2388 4.0054 3.9241 3.8529 3.7087 3.7044 3.6911] stable_rank 52.212 W_k Frob 32.2094 top8_sv [6.0903 4.8989 4.1973 3.4559 3.3531 3.3051 3.2455 3.1608] stable_rank 27.97 W_v Frob 39.7941 top8_sv [5.0598 4.9045 4.8481 4.769 4.6738 4.6053 4.5722 4.5185] stable_rank 61.854 W_o Frob 40.455 top8_sv [9.5668 8.7592 5.3386 5.139 4.9162 4.8063 4.7743 4.7359] stable_rank 17.882 W_fc Frob 77.7758 top8_sv [14.1624 8.1249 7.9152 7.3262 7.1484 6.8556 6.7628 6.6911] stable_rank 30.159 W_proj Frob 82.3952 top8_sv [26.3094 20.6845 19.6986 19.2612 18.5937 18.1973 17.8617 17.6671] stable_rank 9.808 HEAD: QK_frob QK_top4_sv QK_top1_energy QK_sym_fraction QK_trace; OV_frob OV_top4_sv OV_trace OV_eig_posmax OV_eig_negmin L3H0 31.2433 [6.3237 5.9038 5.873 5.6567] 0.041 0.725 47.8937 ; 29.6831 [8.3838 7.5828 6.4196 6.2466] -72.8707 0.1992 -3.1664 L3H1 25.789 [4.5716 4.43 4.4197 4.2853] 0.0314 0.7783 89.2162 ; 53.3976 [13.8537 13.6123 13.3779 12.3595] -70.0842 0.999 -5.3422 L3H2 21.0416 [6.4644 5.7939 5.0121 4.6657] 0.0944 0.7065 3.1927 ; 35.2432 [11.8908 9.927 8.462 8.2846] -45.1262 0.6127 -3.171 L3H3 22.3492 [6.8507 5.6549 4.6934 4.3132] 0.094 0.7018 1.5275 ; 35.9947 [12.4344 10.3968 9.0091 8.0613] -52.1078 0.6827 -3.1082 L3H4 25.9947 [7.3521 5.9014 5.5575 5.2874] 0.08 0.7074 12.8867 ; 31.7775 [9.5258 8.963 7.0112 6.9513] -44.1288 0.5408 -2.9315 L3H5 11.8506 [3.8648 3.3538 2.9899 2.7941] 0.1064 0.7157 1.3911 ; 59.2935 [21.9791 19.2007 17.4205 16.747 ] -60.0223 1.0804 -8.5079 ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import scipy.linalg as la
from threadpoolctl import threadpool_limits
threadpool_limits(4)
A, B = {}, {}
for l in range(4):
w = W[f"ln1_w.{l}"].astype(float)
bias = W[f"ln1_b.{l}"].astype(float)
print("\nLAYER", l, "(gain-weighted matrices; no LayerNorm evaluation)")
for j in range(6):
s = slice(64*j, 64*(j+1))
q0 = W[f"W_q.{l}"][s].astype(float)
k0 = W[f"W_k.{l}"][s].astype(float)
v0 = W[f"W_v.{l}"][s].astype(float)
o = W[f"W_o.{l}"][:,s].astype(float)
q, k, v = q0*w, k0*w, v0*w
a, b = q.T@k, o@v
A[l,j], B[l,j] = a, b
sv = la.svdvals(a)[:64]
qb, kb = q0@bias, k0@bias
key_linear = k.T@qb
query_linear = q.T@kb
print(f"L{l}H{j}",
"QKnorm", round(la.norm(a),4),
"QKsv", np.round(sv[:6],4),
"QKtrace", round(np.trace(a),4),
"OVnorm", round(la.norm(b),4),
"bias_key/query_norm", np.round([la.norm(key_linear),la.norm(query_linear)],4))
print("\nNormalized trace(A_lh @ B_0j), columns L0H0..L0H5")
for l in range(1,4):
for j in range(6):
a = A[l,j]
vals = [np.trace(a@B[0,k])/(la.norm(a)*la.norm(B[0,k])) for k in range(6)]
print(f"L{l}H{j}", np.round(vals,4))
print("\nLargest earlier-head trace compositions (signed normalized trace)")
for l in range(2,4):
for j in range(6):
a = A[l,j]
vals = []
for m in range(l):
for k in range(6):
b = B[m,k]
vals.append((float(np.trace(a@b)/(la.norm(a)*la.norm(b))), f"L{m}H{k}"))
print(f"L{l}H{j}", sorted(vals, reverse=True)[:5])
print("\nL0H2 effective QK spectrum")
a = A[0,2]
u,s,vt = la.svd(a, full_matrices=False)
print("singular values", np.round(s[:40],5))
ev = la.eigvals(a)
ix = np.argsort(abs(ev))[::-1][:24]
print("largest eigenvalues", np.round(ev[ix],4))
print("left/right top-20 subspace overlap", np.round(la.svdvals(u[:,:20].T@vt[:20].T),4))
print("\nL0 query/key and value subspace overlap with L0H2 dominant QK subspace")
p = la.orth(np.column_stack([u[:,:20],vt[:20].T]))
for j in range(6):
s0 = slice(64*j,64*(j+1))
vals=[]
for name in ["W_q","W_k","W_v"]:
m = W[f"{name}.0"][s0].astype(float)*W["ln1_w.0"]
vals.append(la.norm(m@p)**2/la.norm(m)**2)
print(f"L0H{j}", np.round(vals,4))
```
{
"t": 1792713260.155445,
"s": 4.3,
"code": "import numpy as np\nimport scipy.linalg as la\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\nA, B = {}, {}\nfor l in range(4):\n w = W[f\"ln1_w.{l}\"].astype(float)\n bias = W[f\"ln1_b.{l}\"].astype(float)\n print(\"\\nLAYER\", l, \"(gain-weighted matrices; no LayerNorm evaluation)\")\n for j in range(6):\n s = slice(64*j, 64*(j+1))\n q0 = W[f\"W_q.{l}\"][s].astype(float)\n k0 = W[f\"W_k.{l}\"][s].astype(float)\n v0 = W[f\"W_v.{l}\"][s].astype(float)\n o = W[f\"W_o.{l}\"][:,s].astype(float)\n q, k, v = q0*w, k0*w, v0*w\n a, b = q.T@k, o@v\n A[l,j], B[l,j] = a, b\n sv = la.svdvals(a)[:64]\n qb, kb = q0@bias, k0@bias\n key_linear = k.T@qb\n query_linear = q.T@kb\n print(f\"L{l}H{j}\",\n \"QKnorm\", round(la.norm(a),4),\n \"QKsv\", np.round(sv[:6],4),\n \"QKtrace\", round(np.trace(a),4),\n \"OVnorm\", round(la.norm(b),4),\n \"bias_key/query_norm\", np.round([la.norm(key_linear),la.norm(query_linear)],4))\n\nprint(\"\\nNormalized trace(A_lh @ B_0j), columns L0H0..L0H5\")\nfor l in range(1,4):\n for j in range(6):\n a = A[l,j]\n vals = [np.trace(a@B[0,k])/(la.norm(a)*la.norm(B[0,k])) for k in range(6)]\n print(f\"L{l}H{j}\", np.round(vals,4))\n\nprint(\"\\nLargest earlier-head trace compositions (signed normalized trace)\")\nfor l in range(2,4):\n for j in range(6):\n a = A[l,j]\n vals = []\n for m in range(l):\n for k in range(6):\n b = B[m,k]\n vals.append((float(np.trace(a@b)/(la.norm(a)*la.norm(b))), f\"L{m}H{k}\"))\n print(f\"L{l}H{j}\", sorted(vals, reverse=True)[:5])\n\nprint(\"\\nL0H2 effective QK spectrum\")\na = A[0,2]\nu,s,vt = la.svd(a, full_matrices=False)\nprint(\"singular values\", np.round(s[:40],5))\nev = la.eigvals(a)\nix = np.argsort(abs(ev))[::-1][:24]\nprint(\"largest eigenvalues\", np.round(ev[ix],4))\nprint(\"left/right top-20 subspace overlap\", np.round(la.svdvals(u[:,:20].T@vt[:20].T),4))\n\nprint(\"\\nL0 query/key and value subspace overlap with L0H2 dominant QK subspace\")\np = la.orth(np.column_stack([u[:,:20],vt[:20].T]))\nfor j in range(6):\n s0 = slice(64*j,64*(j+1))\n vals=[]\n for name in [\"W_q\",\"W_k\",\"W_v\"]:\n m = W[f\"{name}.0\"][s0].astype(float)*W[\"ln1_w.0\"]\n vals.append(la.norm(m@p)**2/la.norm(m)**2)\n print(f\"L0H{j}\", np.round(vals,4))\n",
"out": "\nLAYER 0 (gain-weighted matrices; no LayerNorm evaluation)\nL0H0 QKnorm 0.4367 QKsv [0.4114 0.0916 0.0567 0.0368 0.0343 0.0289] QKtrace -0.0379 OVnorm 9.0429 bias_key/query_norm [0.1091 0.1694]\nL0H1 QKnorm 0.2362 QKsv [0.1997 0.0761 0.0422 0.0353 0.0279 0.0247] QKtrace -0.1115 OVnorm 6.9661 bias_key/query_norm [0.0267 0.0107]\nL0H2 QKnorm 32.6269 QKsv [11.3199 10.1995 9.7194 9.6486 9.272 8.7533] QKtrace 9.5999 OVnorm 2.54 bias_key/query_norm [1.1448 1.5108]\nL0H3 QKnorm 3.1569 QKsv [2.8706 0.9529 0.621 0.3398 0.261 0.2181] QKtrace 1.5753 OVnorm 1.7662 bias_key/query_norm [0.4283 0.3216]\nL0H4 QKnorm 0.8508 QKsv [0.7136 0.4284 0.1319 0.0522 0.0419 0.0328] QKtrace -0.5674 OVnorm 4.0336 bias_key/query_norm [0.0901 0.0772]\nL0H5 QKnorm 0.6206 QKsv [0.6073 0.0861 0.0369 0.0299 0.0271 0.023 ] QKtrace -0.2776 OVnorm 9.3107 bias_key/query_norm [0.1299 0.2023]\n\nLAYER 1 (gain-weighted matrices; no LayerNorm evaluation)\nL1H0 QKnorm 7.0989 QKsv [4.7823 3.8586 1.9815 1.5518 1.0526 0.8946] QKtrace -5.1542 OVnorm 10.1223 bias_key/query_norm [10.8591 13.1828]\nL1H1 QKnorm 10.5253 QKsv [7.0546 5.3302 3.2915 2.1355 1.6238 1.4283] QKtrace -9.2943 OVnorm 10.3335 bias_key/query_norm [18.2783 18.644 ]\nL1H2 QKnorm 12.41 QKsv [7.9573 6.8828 3.0682 2.603 2.115 1.8171] QKtrace -12.955 OVnorm 9.1806 bias_key/query_norm [20.7031 23.8558]\nL1H3 QKnorm 11.0075 QKsv [7.3565 5.9091 3.2626 2.029 1.764 1.3744] QKtrace -9.2582 OVnorm 10.0512 bias_key/query_norm [18.809 18.7776]\nL1H4 QKnorm 11.6052 QKsv [7.5816 6.9475 2.8775 2.0717 1.5464 1.3804] QKtrace -9.9582 OVnorm 10.401 bias_key/query_norm [18.3219 23.5867]\nL1H5 QKnorm 10.7065 QKsv [7.146 5.914 2.9776 1.8365 1.6321 1.4794] QKtrace -8.1163 OVnorm 10.6202 bias_key/query_norm [14.0211 19.6845]\n\nLAYER 2 (gain-weighted matrices; no LayerNorm evaluation)\nL2H0 QKnorm 19.5056 QKsv [5.5592 5.0376 4.8195 4.5062 4.2407 4.0516] QKtrace -0.173 OVnorm 16.3068 bias_key/query_norm [13.5928 13.0054]\nL2H1 QKnorm 16.5448 QKsv [4.2136 3.5951 3.4598 3.3372 3.2779 3.161 ] QKtrace 4.237 OVnorm 17.7221 bias_key/query_norm [5.9578 7.4963]\nL2H2 QKnorm 12.9062 QKsv [4.8575 4.024 3.2103 2.5372 2.3832 2.2932] QKtrace -8.8427 OVnorm 20.5752 bias_key/query_norm [ 6.0447 12.5128]\nL2H3 QKnorm 16.6966 QKsv [4.4958 4.2052 3.9324 3.5228 3.2925 3.1557] QKtrace -5.9051 OVnorm 17.5062 bias_key/query_norm [11.702 11.0247]\nL2H4 QKnorm 18.7551 QKsv [5.0228 4.8342 4.2578 4.0652 3.7358 3.6109] QKtrace -3.8054 OVnorm 16.624 bias_key/query_norm [12.6211 13.1065]\nL2H5 QKnorm 18.0092 QKsv [5.1073 4.3177 3.95 3.8098 3.7436 3.615 ] QKtrace -5.578 OVnorm 16.6842 bias_key/query_norm [12.0719 11.6318]\n\nLAYER 3 (gain-weighted matrices; no LayerNorm evaluation)\nL3H0 QKnorm 17.8612 QKsv [3.6121 3.3594 3.345 3.2422 3.2067 3.1565] QKtrace 27.2957 OVnorm 22.5984 bias_key/query_norm [2.0652 2.089 ]\nL3H1 QKnorm 14.6476 QKsv [2.6208 2.5393 2.4737 2.4472 2.4061 2.3836] QKtrace 50.449 OVnorm 40.8686 bias_key/query_norm [1.9323 1.0931]\nL3H2 QKnorm 12.0463 QKsv [3.7667 3.2887 2.8452 2.6572 2.4694 2.3229] QKtrace 1.7643 OVnorm 26.8815 bias_key/query_norm [2.3165 2.2163]\nL3H3 QKnorm 12.7936 QKsv [3.9788 3.2199 2.6861 2.4313 2.3645 2.3594] QKtrace 0.8934 OVnorm 27.4312 bias_key/query_norm [2.8828 2.4488]\nL3H4 QKnorm 14.8484 QKsv [4.2301 3.3 3.1849 2.9889 2.8823 2.6888] QKtrace 7.2288 OVnorm 24.1979 bias_key/query_norm [2.4576 2.0855]\nL3H5 QKnorm 6.679 QKsv [2.075 1.8792 1.6976 1.57 1.5251 1.2757] QKtrace 0.7931 OVnorm 45.4008 bias_key/query_norm [1.3167 0.7099]\n\nNormalized trace(A_lh @ B_0j), columns L0H0..L0H5\nL1H0 [ 0.0388 0.0163 -0.0054 -0.0187 0.032 0.0041]\nL1H1 [ 0.0457 0.0329 -0.0001 -0.0143 0.0496 0.0187]\nL1H2 [ 0.0497 0.0415 0.0145 -0.0059 0.054 0.0222]\nL1H3 [ 0.0584 0.0467 0.0147 -0.0099 0.0601 0.0281]\nL1H4 [ 0.0796 0.0707 0.0133 -0.0046 0.0839 0.0459]\nL1H5 [ 0.0386 0.0174 0.0059 -0.0139 0.0363 -0.0111]\nL2H0 [ 0.0156 0.0124 -0.005 -0.0182 0.0183 0.0159]\nL2H1 [ 0.0098 0.0065 -0.0056 -0.0102 0.0104 0.0116]\nL2H2 [ 0.0295"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 2 of at most 100: ``` LAYER 0 (gain-weighted matrices; no LayerNorm evaluation) L0H0 QKnorm 0.4367 QKsv [0.4114 0.0916 0.0567 0.0368 0.0343 0.0289] QKtrace -0.0379 OVnorm 9.0429 bias_key/query_norm [0.1091 0.1694] L0H1 QKnorm 0.2362 QKsv [0.1997 0.0761 0.0422 0.0353 0.0279 0.0247] QKtrace -0.1115 OVnorm 6.9661 bias_key/query_norm [0.0267 0.0107] L0H2 QKnorm 32.6269 QKsv [11.3199 10.1995 9.7194 9.6486 9.272 8.7533] QKtrace 9.5999 OVnorm 2.54 bias_key/query_norm [1.1448 1.5108] L0H3 QKnorm 3.1569 QKsv [2.8706 0.9529 0.621 0.3398 0.261 0.2181] QKtrace 1.5753 OVnorm 1.7662 bias_key/query_norm [0.4283 0.3216] L0H4 QKnorm 0.8508 QKsv [0.7136 0.4284 0.1319 0.0522 0.0419 0.0328] QKtrace -0.5674 OVnorm 4.0336 bias_key/query_norm [0.0901 0.0772] L0H5 QKnorm 0.6206 QKsv [0.6073 0.0861 0.0369 0.0299 0.0271 0.023 ] QKtrace -0.2776 OVnorm 9.3107 bias_key/query_norm [0.1299 0.2023] LAYER 1 (gain-weighted matrices; no LayerNorm evaluation) L1H0 QKnorm 7.0989 QKsv [4.7823 3.8586 1.9815 1.5518 1.0526 0.8946] QKtrace -5.1542 OVnorm 10.1223 bias_key/query_norm [10.8591 13.1828] L1H1 QKnorm 10.5253 QKsv [7.0546 5.3302 3.2915 2.1355 1.6238 1.4283] QKtrace -9.2943 OVnorm 10.3335 bias_key/query_norm [18.2783 18.644 ] L1H2 QKnorm 12.41 QKsv [7.9573 6.8828 3.0682 2.603 2.115 1.8171] QKtrace -12.955 OVnorm 9.1806 bias_key/query_norm [20.7031 23.8558] L1H3 QKnorm 11.0075 QKsv [7.3565 5.9091 3.2626 2.029 1.764 1.3744] QKtrace -9.2582 OVnorm 10.0512 bias_key/query_norm [18.809 18.7776] L1H4 QKnorm 11.6052 QKsv [7.5816 6.9475 2.8775 2.0717 1.5464 1.3804] QKtrace -9.9582 OVnorm 10.401 bias_key/query_norm [18.3219 23.5867] L1H5 QKnorm 10.7065 QKsv [7.146 5.914 2.9776 1.8365 1.6321 1.4794] QKtrace -8.1163 OVnorm 10.6202 bias_key/query_norm [14.0211 19.6845] LAYER 2 (gain-weighted matrices; no LayerNorm evaluation) L2H0 QKnorm 19.5056 QKsv [5.5592 5.0376 4.8195 4.5062 4.2407 4.0516] QKtrace -0.173 OVnorm 16.3068 bias_key/query_norm [13.5928 13.0054] L2H1 QKnorm 16.5448 QKsv [4.2136 3.5951 3.4598 3.3372 3.2779 3.161 ] QKtrace 4.237 OVnorm 17.7221 bias_key/query_norm [5.9578 7.4963] L2H2 QKnorm 12.9062 QKsv [4.8575 4.024 3.2103 2.5372 2.3832 2.2932] QKtrace -8.8427 OVnorm 20.5752 bias_key/query_norm [ 6.0447 12.5128] L2H3 QKnorm 16.6966 QKsv [4.4958 4.2052 3.9324 3.5228 3.2925 3.1557] QKtrace -5.9051 OVnorm 17.5062 bias_key/query_norm [11.702 11.0247] L2H4 QKnorm 18.7551 QKsv [5.0228 4.8342 4.2578 4.0652 3.7358 3.6109] QKtrace -3.8054 OVnorm 16.624 bias_key/query_norm [12.6211 13.1065] L2H5 QKnorm 18.0092 QKsv [5.1073 4.3177 3.95 3.8098 3.7436 3.615 ] QKtrace -5.578 OVnorm 16.6842 bias_key/query_norm [12.0719 11.6318] LAYER 3 (gain-weighted matrices; no LayerNorm evaluation) L3H0 QKnorm 17.8612 QKsv [3.6121 3.3594 3.345 3.2422 3.2067 3.1565] QKtrace 27.2957 OVnorm 22.5984 bias_key/query_norm [2.0652 2.089 ] L3H1 QKnorm 14.6476 QKsv [2.6208 2.5393 2.4737 2.4472 2.4061 2.3836] QKtrace 50.449 OVnorm 40.8686 bias_key/query_norm [1.9323 1.0931] L3H2 QKnorm 12.0463 QKsv [3.7667 3.2887 2.8452 2.6572 2.4694 2.3229] QKtrace 1.7643 OVnorm 26.8815 bias_key/query_norm [2.3165 2.2163] L3H3 QKnorm 12.7936 QKsv [3.9788 3.2199 2.6861 2.4313 2.3645 2.3594] QKtrace 0.8934 OVnorm 27.4312 bias_key/query_norm [2.8828 2.4488] L3H4 QKnorm 14.8484 QKsv [4.2301 3.3 3.1849 2.9889 2.8823 2.6888] QKtrace 7.2288 OVnorm 24.1979 bias_key/query_norm [2.4576 2.0855] L3H5 QKnorm 6.679 QKsv [2.075 1.8792 1.6976 1.57 1.5251 1.2757] QKtrace 0.7931 OVnorm 45.4008 bias_key/query_norm [1.3167 0.7099] Normalized trace(A_lh @ B_0j), columns L0H0..L0H5 L1H0 [ 0.0388 0.0163 -0.0054 -0.0187 0.032 0.0041] L1H1 [ 0.0457 0.0329 -0.0001 -0.0143 0.0496 0.0187] L1H2 [ 0.0497 0.0415 0.0145 -0.0059 0.054 0.0222] L1H3 [ 0.0584 0.0467 0.0147 -0.0099 0.0601 0.0281] L1H4 [ 0.0796 0.0707 0.0133 -0.0046 0.0839 0.0459] L1H5 [ 0.0386 0.0174 0.0059 -0.0139 0.0363 -0.0111] L2H0 [ 0.0156 0.0124 -0.005 -0.0182 0.0183 0.0159] L2H1 [ 0.0098 0.0065 -0.0056 -0.0102 0.0104 0.0116] L2H2 [ 0.0295 0.0238 0.001 -0.0173 0.0355 0.01 ] L2H3 [ 0.0227 0.0162 0.0029 -0.012 0.023 0.0216] L2H4 [ 0.0293 0.0219 -0.008 -0.0169 0.0299 0.021 ] L2H5 [ 0.0222 0.0146 0.0003 -0.0172 0.0214 0.0168] L3H0 [ 0.0046 0.0048 -0.0022 -0.0031 0.007 0.0069] L3H1 [-0.0023 -0.0035 0.0949 0.001 -0.0052 -0.0028] L3H2 [ 0.016 0.0102 0.003 -0.0059 0.0141 0.0221] L3H3 [ 0.0104 0.0049 0.0012 -0.0018 0.0084 0.0144] L3H4 [ 0.0196 0.0205 -0.0048 -0.0013 0.0227 0.0282] L3H5 [-0.005 -0.0047 0.0025 -0.004 -0.0044 -0.003 ] Largest earlier-head trace compositions (signed normalized trace) L2H0 [(0.01831488054770177, 'L0H4'), (0.015886681026769052, 'L0H5'), (0.015608857494623197, 'L0H0'), (0.012367579270624375, 'L0H1'), (-0.004385075845198281, 'L1H1')] L2H1 [(0.011626730747948765, 'L0H5'), (0.010425051990288375, 'L0H4'), (0.009803403048760628, 'L0H0'), (0.006525725144364843, 'L0H1'), (0.0030746091424649824, 'L1H3')] L2H2 [(0.03548392181702849, 'L0H4'), (0.029485073254451613, 'L0H0'), (0.023778341440254248, 'L0H1'), (0.009965517460207657, 'L0H5'), (0.007742027119735287, 'L1H4')] L2H3 [(0.023009692499583036, 'L0H4'), (0.022655398470314534, 'L0H0'), (0.02159495944098225, 'L0H5'), (0.016205740027523712, 'L0H1'), (0.002867347983306011, 'L0H2')] L2H4 [(0.02986057570010887, 'L0H4'), (0.02925880246372659, 'L0H0'), (0.021926972741687572, 'L0H1'), (0.02098021741375207, 'L0H5'), (-0.002996000199168058, 'L1H3')] L2H5 [(0.022163566772676495, 'L0H0'), (0.02137927166317656, 'L0H4'), (0.01679521007828187, 'L0H5'), (0.014566356586938216, 'L0H1'), (0.00035304827650425267, 'L1H2')] L3H0 [(0.00703240691007632, 'L0H4'), (0.006910922145819949, 'L0H5'), (0.004752035137208596, 'L0H1'), (0.004570716642140448, 'L0H0'), (-0.002247146526228783, 'L0H2')] L3H1 [(0.09486606986058785, 'L0H2'), (0.0811171357829096, 'L2H1'), (0.056764532759469445, 'L1H3'), (0.05549793736058776, 'L1H2'), (0.03612116030288871, 'L1H1')] L3H2 [(0.02205580262173361, 'L0H5'), (0.016017133729234214, 'L0H0'), (0.014117851397887326, 'L0H4'), (0.010234078605252283, 'L0H1'), (0.0029527680839085435, 'L0H2')] L3H3 [(0.014426395753234358, 'L0H5'), (0.010428059825454958, 'L0H0'), (0.008445143231364794, 'L0H4'), (0.004921340717805306, 'L0H1'), (0.0011732199754640239, 'L0H2')] L3H4 [(0.028195028938726028, 'L0H5'), (0.022746228327328106, 'L0H4'), (0.020535542760585734, 'L0H1'), (0.019626760961869532, 'L0H0'), (0.0007146672776497601, 'L2H0')] L3H5 [(0.01675224101706262, 'L2H3'), (0.01151727112046607, 'L2H1'), (0.0111474435163409, 'L1H0'), (0.011067087616306114, 'L2H4'), (0.01010903817591309, 'L2H2')] L0H2 effective QK spectrum singular values [11.3199 10.1995 9.7194 9.6486 9.272 8.7533 8.6907 8.1753 7.6461 7.3268 6.89 6.2532 5.8234 5.7713 4.1168 3.9356 3.4309 3.0088 2.5751 2.3197 1.5122 1.3247 1.0256 0.9255 0.8764 0.7851 0.601 0.4484 0.394 0.3479 0.2969 0.2686 0.2412 0.2179 0.1933 0.1571 0.156 0.1255 0.1161 0.1043] largest eigenvalues [ 5.0575+6.5134j 5.0575-6.5134j -1.8401+7.4813j -1.8401-7.4813j 2.6658+7.0019j 2.6658-7.0019j -6.2304+2.6725j -6.2304-2.6725j -6.5315+0.j 5.2072-3.2762j 5.2072+3.2762j 2.7023-4.0824j 2.7023+4.0824j 4.0955-0.1975j 4.0955+0.1975j 3.3297+0.j -1.8666+1.9902j -1.8666-1.9902j -2.3481+1.0233j -2.3481-1.0233j 0.1685+1.0747j 0.1685-1.0747j -0.946 +0.j -0.7171-0.0125j] left/right top-20 subspace overlap [0.9877 0.9845 0.9797 0.975 0.9689 0.9672 0.9649 0.9539 0.9482 0.9435 0.9412 0.9304 0.9103 0.8837 0.8478 0.8088 0.7843 0.6577 0.541 0.3834] L0 query/key and value subspace overlap with L0H2 dominant QK subspace L0H0 [0.6828 0.3702 0.8338] L0H1 [0.4909 0.3968 0.8247] L0H2 [0.9601 0.971 0.3177] L0H3 [0.6627 0.7298 0.3891] L0H4 [0.5381 0.4254 0.7063] L0H5 [0.6866 0.4371 0.8073] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import scipy.linalg as la
from threadpoolctl import threadpool_limits
threadpool_limits(4)
for l in range(4):
print("\nLAYER",l)
for name in ["W_fc","W_proj"]:
m = W[f"{name}.{l}"].astype(float)
s = la.svdvals(m)
print(name,"first 40 SV:",np.round(s[:40],3))
print("SV indices 40,60,80,100,150,200,300,383:",
np.round(s[[40,60,80,100,150,200,300,383]],3))
def head(l,j):
sl=slice(j*64,(j+1)*64)
w=W[f"ln1_w.{l}"].astype(float)
q=W[f"W_q.{l}"][sl].astype(float)*w
k=W[f"W_k.{l}"][sl].astype(float)*w
v=W[f"W_v.{l}"][sl].astype(float)*w
o=W[f"W_o.{l}"][:,sl].astype(float)
return q.T@k,o@v
aa=[head(0,j)[0] for j in range(6)]
bb=[head(0,j)[1] for j in range(6)]
print("\nL0 OV matrix cosine similarities:")
print(np.round([[np.sum(x*y)/(la.norm(x)*la.norm(y)) for y in bb] for x in bb],3))
print("L0 QK matrix cosine similarities:")
print(np.round([[np.sum(x*y)/(la.norm(x)*la.norm(y)) for y in aa] for x in aa],3))
print("\nL0H2 unweighted QK singular values:")
q=W["W_q.0"][128:192].astype(float)
k=W["W_k.0"][128:192].astype(float)
u,s,vt=la.svd(q.T@k,full_matrices=False)
print(np.round(s[:48],3))
p=la.orth(np.column_stack((u[:,:20],vt[:20].T)))
print("L0 raw Q/K/V energy in this subspace:")
for j in range(6):
print(j, np.round([la.norm(W[f"{name}.0"][64*j:64*(j+1)]@p)**2/
la.norm(W[f"{name}.0"][64*j:64*(j+1)])**2
for name in ["W_q","W_k","W_v"]],4))
print("\nL1 key-bias weight directions and L0 OV outputs")
for j in range(6):
sl=slice(j*64,(j+1)*64)
q=W["W_q.1"][sl].astype(float)
k=W["W_k.1"][sl].astype(float)
r=(k*W["ln1_w.1"]).T@(q@W["ln1_b.1"])
vals=[]
for b in bb:
vals.append(la.norm(r@b)/(la.norm(r)*la.norm(b)))
print(j,np.round(vals,4))
print("\nLinear-algebra-only MLP products: normalized trace with later QK")
# These are matrix products, not MLP evaluations: there is no ReLU or input.
mats={}
for l in range(3):
f=W[f"W_fc.{l}"].astype(float)*W[f"ln2_w.{l}"]
mats[l]=W[f"W_proj.{l}"].astype(float)@f
for l in range(1,4):
for j in range(6):
a,b=head(l,j)
print(f"L{l}H{j}",
np.round([np.trace(a@mats[m])/(la.norm(a)*la.norm(mats[m]))
for m in range(l)],4))
```
{
"t": 1792713326.4988477,
"s": 3.84,
"code": "import numpy as np\nimport scipy.linalg as la\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\nfor l in range(4):\n print(\"\\nLAYER\",l)\n for name in [\"W_fc\",\"W_proj\"]:\n m = W[f\"{name}.{l}\"].astype(float)\n s = la.svdvals(m)\n print(name,\"first 40 SV:\",np.round(s[:40],3))\n print(\"SV indices 40,60,80,100,150,200,300,383:\",\n np.round(s[[40,60,80,100,150,200,300,383]],3))\n\ndef head(l,j):\n sl=slice(j*64,(j+1)*64)\n w=W[f\"ln1_w.{l}\"].astype(float)\n q=W[f\"W_q.{l}\"][sl].astype(float)*w\n k=W[f\"W_k.{l}\"][sl].astype(float)*w\n v=W[f\"W_v.{l}\"][sl].astype(float)*w\n o=W[f\"W_o.{l}\"][:,sl].astype(float)\n return q.T@k,o@v\n\naa=[head(0,j)[0] for j in range(6)]\nbb=[head(0,j)[1] for j in range(6)]\nprint(\"\\nL0 OV matrix cosine similarities:\")\nprint(np.round([[np.sum(x*y)/(la.norm(x)*la.norm(y)) for y in bb] for x in bb],3))\nprint(\"L0 QK matrix cosine similarities:\")\nprint(np.round([[np.sum(x*y)/(la.norm(x)*la.norm(y)) for y in aa] for x in aa],3))\n\nprint(\"\\nL0H2 unweighted QK singular values:\")\nq=W[\"W_q.0\"][128:192].astype(float)\nk=W[\"W_k.0\"][128:192].astype(float)\nu,s,vt=la.svd(q.T@k,full_matrices=False)\nprint(np.round(s[:48],3))\np=la.orth(np.column_stack((u[:,:20],vt[:20].T)))\nprint(\"L0 raw Q/K/V energy in this subspace:\")\nfor j in range(6):\n print(j, np.round([la.norm(W[f\"{name}.0\"][64*j:64*(j+1)]@p)**2/\n la.norm(W[f\"{name}.0\"][64*j:64*(j+1)])**2\n for name in [\"W_q\",\"W_k\",\"W_v\"]],4))\n\nprint(\"\\nL1 key-bias weight directions and L0 OV outputs\")\nfor j in range(6):\n sl=slice(j*64,(j+1)*64)\n q=W[\"W_q.1\"][sl].astype(float)\n k=W[\"W_k.1\"][sl].astype(float)\n r=(k*W[\"ln1_w.1\"]).T@(q@W[\"ln1_b.1\"])\n vals=[]\n for b in bb:\n vals.append(la.norm(r@b)/(la.norm(r)*la.norm(b)))\n print(j,np.round(vals,4))\n\nprint(\"\\nLinear-algebra-only MLP products: normalized trace with later QK\")\n# These are matrix products, not MLP evaluations: there is no ReLU or input.\nmats={}\nfor l in range(3):\n f=W[f\"W_fc.{l}\"].astype(float)*W[f\"ln2_w.{l}\"]\n mats[l]=W[f\"W_proj.{l}\"].astype(float)@f\nfor l in range(1,4):\n for j in range(6):\n a,b=head(l,j)\n print(f\"L{l}H{j}\",\n np.round([np.trace(a@mats[m])/(la.norm(a)*la.norm(mats[m]))\n for m in range(l)],4))\n",
"out": "\nLAYER 0\nW_fc first 40 SV: [18.176 11.018 8.737 7.452 7.212 7.062 7.011 6.62 6.57 6.42 6.343 6.29 6.137 6.043 5.974 5.918 5.807 5.779 5.583 5.513 5.5\n 5.36 5.16 5.12 5.086 5.032 4.899 4.879 4.819 4.732 4.601 4.533 4.479 4.441 4.268 4.198 4.031 3.974 3.869 3.824]\nSV indices 40,60,80,100,150,200,300,383: [3.72 2.377 1.616 1.25 0.963 0.814 0.596 0.4 ]\nW_proj first 40 SV: [9.826 7.351 6.265 5.869 5.797 5.657 5.516 5.45 5.333 5.242 5.197 5.094 4.98 4.904 4.863 4.819 4.793 4.689 4.627 4.595 4.557 4.523 4.411 4.378\n 4.283 4.216 4.143 4.125 4.048 4.025 3.894 3.847 3.808 3.792 3.764 3.711 3.685 3.636 3.595 3.525]\nSV indices 40,60,80,100,150,200,300,383: [3.499 2.962 2.34 1.767 0.996 0.607 0.343 0.213]\n\nLAYER 1\nW_fc first 40 SV: [21.488 13.135 12.804 12.625 12.328 12.118 11.85 11.691 11.43 11.269 11.049 10.846 10.646 10.62 10.324 9.928 9.758 9.699 9.592 9.353 9.219\n 9.169 9.074 8.921 8.816 8.729 8.516 8.442 8.433 8.247 8.106 7.853 7.745 7.696 7.617 7.54 7.422 7.349 7.124 7.084]\nSV indices 40,60,80,100,150,200,300,383: [6.851 5.28 4.27 3.346 1.842 1.274 0.846 0.411]\nW_proj first 40 SV: [8.193 7.894 7.199 6.997 6.671 6.433 6.274 6.133 6.099 6.021 5.942 5.865 5.829 5.789 5.698 5.675 5.646 5.629 5.581 5.564 5.553 5.538 5.528 5.477\n 5.456 5.447 5.438 5.403 5.399 5.38 5.367 5.351 5.332 5.327 5.314 5.296 5.252 5.235 5.219 5.208]\nSV indices 40,60,80,100,150,200,300,383: [5.197 4.925 4.704 4.479 3.985 3.509 2.629 0.319]\n\nLAYER 2\nW_fc first 40 SV: [15.363 8.021 7.895 7.725 7.597 7.49 7.257 7.216 7.141 7.053 6.995 6.974 6.881 6.828 6.8 6.773 6.692 6.634 6.589 6.554 6.512\n 6.466 6.411 6.364 6.328 6.312 6.293 6.225 6.17 6.154 6.11 6.101 6.049 6.032 5.963 5.934 5.917 5.91 5.902 5.854]\nSV indices 40,60,80,100,150,200,300,383: [5.822 5.301 4.876 4.56 3.917 3.354 2.38 0.645]\nW_proj first 40 SV: [9.404 9.187 8.723 8.47 8.04 7.795 7.614 7.45 7.392 7.253 7.223 7.148 7.082 6.972 6.829 6.733 6.637 6.547 6.46 6.404 6.374 6.307 6.209 6.181\n 6.12 6.091 5.989 5.9 5.88 5.833 5.758 5.703 5.685 5.645 5.588 5.559 5.491 5.478 5.467 5.429]\nSV indices 40,60,80,100,150,200,300,383: [5.406 4.97 4.657 4.389 3.809 3.329 2.449 0.362]\n\nLAYER 3\nW_fc first 40 SV: [14.162 8.125 7.915 7.326 7.148 6.856 6.763 6.691 6.489 6.436 6.387 6.31 6.26 6.177 6.144 6.088 6.066 6.017 5.962 5.934 5.883\n 5.855 5.843 5.781 5.752 5.72 5.697 5.657 5.632 5.617 5.593 5.568 5.519 5.517 5.495 5.45 5.431 5.389 5.379 5.361]\nSV indices 40,60,80,100,150,200,300,383: [5.341 4.986 4.708 4.465 3.94 3.5 2.678 0.683]\nW_proj first 40 SV: [26.309 20.685 19.699 19.261 18.594 18.197 17.862 17.667 17.044 16.749 16.039 15.151 14.756 14.326 14.05 13.5 13.293 12.531 12.401 11.958 11.74\n 11.594 10.367 8.351 8.064 7.914 7.461 7.166 5.833 4.072 3.221 3.139 2.842 2.8 2.687 2.571 2.403 2.328 2.244 2.185]\nSV indices 40,60,80,100,150,200,300,383: [2.181 1.645 1.422 1.244 0.961 0.765 0.509 0.318]\n\nL0 OV matrix cosine similarities:\n[[1. 0.885 0.008 0.163 0.903 0.724]\n [0.885 1. 0.006 0.195 0.889 0.802]\n [0.008 0.006 1. 0.012 0.006 0.017]\n [0.163 0.195 0.012 1. 0.195 0.156]\n [0.903 0.889 0.006 0.195 1. 0.728]\n [0.724 0.802 0.017 0.156 0.728 1. ]]\nL0 QK matrix cosine similarities:\n[[ 1. 0.209 -0.016 -0.014 0.115 0.018]\n [ 0.209 1. -0.004 -0.036 0.209 -0.008]\n [-0.016 -0.004 1. 0.239 -0.006 -0.026]\n [-0.014 -0.036 0.239 1. -0.054 -0.058]\n [ 0.115 0.209 -0.006 -0.054 1. 0.172]\n [ 0.018 -0.008 -0.026 -0.058 0.172 1. ]]\n\nL0H2 unweighted QK singular values:\n[59.879 57.279 55.581 54.347 51.234 49.452 47.184 44.678 43.21 41.651 39.871 35.376 31.577 26.606 23.538 21.633 19.958 18.271 16.837 14.93 13.224\n 10.856 9.22 8.099 8.066 6.534 4.851 4.339 3.972 3.207 3.015 2.925 2.44 2.351 1.993 1.798 1.513 1.451 1.349 1.297 1.132 1.074\n 0.976 0.837 0.819 0.709 0.665"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 3 of at most 100: ``` LAYER 0 W_fc first 40 SV: [18.176 11.018 8.737 7.452 7.212 7.062 7.011 6.62 6.57 6.42 6.343 6.29 6.137 6.043 5.974 5.918 5.807 5.779 5.583 5.513 5.5 5.36 5.16 5.12 5.086 5.032 4.899 4.879 4.819 4.732 4.601 4.533 4.479 4.441 4.268 4.198 4.031 3.974 3.869 3.824] SV indices 40,60,80,100,150,200,300,383: [3.72 2.377 1.616 1.25 0.963 0.814 0.596 0.4 ] W_proj first 40 SV: [9.826 7.351 6.265 5.869 5.797 5.657 5.516 5.45 5.333 5.242 5.197 5.094 4.98 4.904 4.863 4.819 4.793 4.689 4.627 4.595 4.557 4.523 4.411 4.378 4.283 4.216 4.143 4.125 4.048 4.025 3.894 3.847 3.808 3.792 3.764 3.711 3.685 3.636 3.595 3.525] SV indices 40,60,80,100,150,200,300,383: [3.499 2.962 2.34 1.767 0.996 0.607 0.343 0.213] LAYER 1 W_fc first 40 SV: [21.488 13.135 12.804 12.625 12.328 12.118 11.85 11.691 11.43 11.269 11.049 10.846 10.646 10.62 10.324 9.928 9.758 9.699 9.592 9.353 9.219 9.169 9.074 8.921 8.816 8.729 8.516 8.442 8.433 8.247 8.106 7.853 7.745 7.696 7.617 7.54 7.422 7.349 7.124 7.084] SV indices 40,60,80,100,150,200,300,383: [6.851 5.28 4.27 3.346 1.842 1.274 0.846 0.411] W_proj first 40 SV: [8.193 7.894 7.199 6.997 6.671 6.433 6.274 6.133 6.099 6.021 5.942 5.865 5.829 5.789 5.698 5.675 5.646 5.629 5.581 5.564 5.553 5.538 5.528 5.477 5.456 5.447 5.438 5.403 5.399 5.38 5.367 5.351 5.332 5.327 5.314 5.296 5.252 5.235 5.219 5.208] SV indices 40,60,80,100,150,200,300,383: [5.197 4.925 4.704 4.479 3.985 3.509 2.629 0.319] LAYER 2 W_fc first 40 SV: [15.363 8.021 7.895 7.725 7.597 7.49 7.257 7.216 7.141 7.053 6.995 6.974 6.881 6.828 6.8 6.773 6.692 6.634 6.589 6.554 6.512 6.466 6.411 6.364 6.328 6.312 6.293 6.225 6.17 6.154 6.11 6.101 6.049 6.032 5.963 5.934 5.917 5.91 5.902 5.854] SV indices 40,60,80,100,150,200,300,383: [5.822 5.301 4.876 4.56 3.917 3.354 2.38 0.645] W_proj first 40 SV: [9.404 9.187 8.723 8.47 8.04 7.795 7.614 7.45 7.392 7.253 7.223 7.148 7.082 6.972 6.829 6.733 6.637 6.547 6.46 6.404 6.374 6.307 6.209 6.181 6.12 6.091 5.989 5.9 5.88 5.833 5.758 5.703 5.685 5.645 5.588 5.559 5.491 5.478 5.467 5.429] SV indices 40,60,80,100,150,200,300,383: [5.406 4.97 4.657 4.389 3.809 3.329 2.449 0.362] LAYER 3 W_fc first 40 SV: [14.162 8.125 7.915 7.326 7.148 6.856 6.763 6.691 6.489 6.436 6.387 6.31 6.26 6.177 6.144 6.088 6.066 6.017 5.962 5.934 5.883 5.855 5.843 5.781 5.752 5.72 5.697 5.657 5.632 5.617 5.593 5.568 5.519 5.517 5.495 5.45 5.431 5.389 5.379 5.361] SV indices 40,60,80,100,150,200,300,383: [5.341 4.986 4.708 4.465 3.94 3.5 2.678 0.683] W_proj first 40 SV: [26.309 20.685 19.699 19.261 18.594 18.197 17.862 17.667 17.044 16.749 16.039 15.151 14.756 14.326 14.05 13.5 13.293 12.531 12.401 11.958 11.74 11.594 10.367 8.351 8.064 7.914 7.461 7.166 5.833 4.072 3.221 3.139 2.842 2.8 2.687 2.571 2.403 2.328 2.244 2.185] SV indices 40,60,80,100,150,200,300,383: [2.181 1.645 1.422 1.244 0.961 0.765 0.509 0.318] L0 OV matrix cosine similarities: [[1. 0.885 0.008 0.163 0.903 0.724] [0.885 1. 0.006 0.195 0.889 0.802] [0.008 0.006 1. 0.012 0.006 0.017] [0.163 0.195 0.012 1. 0.195 0.156] [0.903 0.889 0.006 0.195 1. 0.728] [0.724 0.802 0.017 0.156 0.728 1. ]] L0 QK matrix cosine similarities: [[ 1. 0.209 -0.016 -0.014 0.115 0.018] [ 0.209 1. -0.004 -0.036 0.209 -0.008] [-0.016 -0.004 1. 0.239 -0.006 -0.026] [-0.014 -0.036 0.239 1. -0.054 -0.058] [ 0.115 0.209 -0.006 -0.054 1. 0.172] [ 0.018 -0.008 -0.026 -0.058 0.172 1. ]] L0H2 unweighted QK singular values: [59.879 57.279 55.581 54.347 51.234 49.452 47.184 44.678 43.21 41.651 39.871 35.376 31.577 26.606 23.538 21.633 19.958 18.271 16.837 14.93 13.224 10.856 9.22 8.099 8.066 6.534 4.851 4.339 3.972 3.207 3.015 2.925 2.44 2.351 1.993 1.798 1.513 1.451 1.349 1.297 1.132 1.074 0.976 0.837 0.819 0.709 0.665 0.625] L0 raw Q/K/V energy in this subspace: 0 [0.2455 0.1965 0.368 ] 1 [0.1627 0.197 0.3512] 2 [0.9298 0.9329 0.1354] 3 [0.3366 0.3524 0.1739] 4 [0.1829 0.1872 0.2346] 5 [0.2132 0.2109 0.3322] L1 key-bias weight directions and L0 OV outputs 0 [0.4286 0.5531 0.0093 0.0966 0.4738 0.7892] 1 [0.4773 0.5822 0.0093 0.0781 0.5123 0.8106] 2 [0.4795 0.582 0.0179 0.0758 0.5122 0.8098] 3 [0.4589 0.5676 0.0232 0.0876 0.4939 0.799 ] 4 [0.5101 0.5937 0.0332 0.0676 0.5308 0.8157] 5 [0.4521 0.5668 0.0285 0.0787 0.4904 0.7964] Linear-algebra-only MLP products: normalized trace with later QK L1H0 [-0.0073] L1H1 [-0.041] L1H2 [-0.0388] L1H3 [-0.0265] L1H4 [-0.0369] L1H5 [-0.0265] L2H0 [0.0028 0.0035] L2H1 [-0.0083 0.0235] L2H2 [0.0014 0.0149] L2H3 [-0.0047 0.0137] L2H4 [-0.0058 0.0046] L2H5 [ 0.001 -0.0012] L3H0 [-0.005 -0.0103 0.0057] L3H1 [-0.0464 -0.0133 -0.0229] L3H2 [-0.0034 0.0021 0.0054] L3H3 [-0.0012 -0.0023 0.012 ] L3H4 [-0.0011 0.0005 0.0093] L3H5 [-0.0067 -0.0014 0.0039] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import scipy.linalg as la
from threadpoolctl import threadpool_limits
threadpool_limits(4)
A,B,Rkey = {},{},{}
for l in range(4):
d=W[f"ln1_w.{l}"].astype(float)
z=W[f"ln1_b.{l}"].astype(float)
for j in range(6):
sl=slice(64*j,64*(j+1))
q=W[f"W_q.{l}"][sl].astype(float)
k=W[f"W_k.{l}"][sl].astype(float)
v=W[f"W_v.{l}"][sl].astype(float)
o=W[f"W_o.{l}"][:,sl].astype(float)
A[l,j]=(q*d).T@(k*d)
B[l,j]=o@(v*d)
Rkey[l,j]=(k*d).T@(q@z)
# Common write subspace of the four similar first-layer OV matrices.
common=np.concatenate([B[0,j] for j in [0,1,4,5]],axis=1)
u,s,vt=la.svd(common,full_matrices=False)
print("Shared L0H0/H1/H4/H5 OV singular values:",np.round(s[:20],4))
clock=u[:,:2]
print("Fraction of each L0 OV's squared norm in shared top-2 write subspace:")
for j in range(6):
print(j,round(la.norm(clock.T@B[0,j])**2/la.norm(B[0,j])**2,4))
print("\nLater QK energy on shared top-2 write subspace and bias-key alignment")
print("head: query_fraction key_fraction key_bias_fraction key_bias_coordinates")
for l in range(1,4):
for j in range(6):
a=A[l,j]
r=Rkey[l,j]
print(f"L{l}H{j}",
np.round([la.norm(clock.T@a)**2/la.norm(a)**2,
la.norm(a@clock)**2/la.norm(a)**2,
la.norm(clock.T@r)**2/la.norm(r)**2],4),
np.round(clock.T@r,3))
# A character-channel candidate: right singular subspace of L0H2 OV.
u2,s2,vt2=la.svd(B[0,2],full_matrices=False)
print("\nL0H2 gain-weighted OV singular values:",np.round(s2[:40],4))
T=vt2[:27].T
print("Projected compositions on L0H2's top-27 read subspace:")
print("head: trace(A Bprev) projected_trace symmetric_fraction positive/negative_sym_eigenmass")
for l in [1,2,3]:
for j in range(6):
a=A[l,j]
m=T.T@a@B[0,2]@T
sym=(m+m.T)/2
es=la.eigvalsh(sym)
print(f"L{l}H{j}",
np.round([np.trace(a@B[0,2]),np.trace(m),
la.norm(sym)/la.norm(m),
es[es>0].sum(),-es[es<0].sum()],4))
print("\nUnweighted L0H2 V singular values:")
print(np.round(la.svdvals(W["W_v.0"][128:192].astype(float)),4))
```
{
"t": 1792713446.7298477,
"s": 2.7,
"code": "import numpy as np\nimport scipy.linalg as la\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\nA,B,Rkey = {},{},{}\nfor l in range(4):\n d=W[f\"ln1_w.{l}\"].astype(float)\n z=W[f\"ln1_b.{l}\"].astype(float)\n for j in range(6):\n sl=slice(64*j,64*(j+1))\n q=W[f\"W_q.{l}\"][sl].astype(float)\n k=W[f\"W_k.{l}\"][sl].astype(float)\n v=W[f\"W_v.{l}\"][sl].astype(float)\n o=W[f\"W_o.{l}\"][:,sl].astype(float)\n A[l,j]=(q*d).T@(k*d)\n B[l,j]=o@(v*d)\n Rkey[l,j]=(k*d).T@(q@z)\n\n# Common write subspace of the four similar first-layer OV matrices.\ncommon=np.concatenate([B[0,j] for j in [0,1,4,5]],axis=1)\nu,s,vt=la.svd(common,full_matrices=False)\nprint(\"Shared L0H0/H1/H4/H5 OV singular values:\",np.round(s[:20],4))\nclock=u[:,:2]\nprint(\"Fraction of each L0 OV's squared norm in shared top-2 write subspace:\")\nfor j in range(6):\n print(j,round(la.norm(clock.T@B[0,j])**2/la.norm(B[0,j])**2,4))\n\nprint(\"\\nLater QK energy on shared top-2 write subspace and bias-key alignment\")\nprint(\"head: query_fraction key_fraction key_bias_fraction key_bias_coordinates\")\nfor l in range(1,4):\n for j in range(6):\n a=A[l,j]\n r=Rkey[l,j]\n print(f\"L{l}H{j}\",\n np.round([la.norm(clock.T@a)**2/la.norm(a)**2,\n la.norm(a@clock)**2/la.norm(a)**2,\n la.norm(clock.T@r)**2/la.norm(r)**2],4),\n np.round(clock.T@r,3))\n\n# A character-channel candidate: right singular subspace of L0H2 OV.\nu2,s2,vt2=la.svd(B[0,2],full_matrices=False)\nprint(\"\\nL0H2 gain-weighted OV singular values:\",np.round(s2[:40],4))\nT=vt2[:27].T\nprint(\"Projected compositions on L0H2's top-27 read subspace:\")\nprint(\"head: trace(A Bprev) projected_trace symmetric_fraction positive/negative_sym_eigenmass\")\nfor l in [1,2,3]:\n for j in range(6):\n a=A[l,j]\n m=T.T@a@B[0,2]@T\n sym=(m+m.T)/2\n es=la.eigvalsh(sym)\n print(f\"L{l}H{j}\",\n np.round([np.trace(a@B[0,2]),np.trace(m),\n la.norm(sym)/la.norm(m),\n es[es>0].sum(),-es[es<0].sum()],4))\n\nprint(\"\\nUnweighted L0H2 V singular values:\")\nprint(np.round(la.svdvals(W[\"W_v.0\"][128:192].astype(float)),4))\n",
"out": "Shared L0H0/H1/H4/H5 OV singular values: [11.996 8.4336 3.3624 1.312 0.9992 0.8615 0.7301 0.6058 0.5661 0.5402 0.4827 0.436 0.3932 0.3667 0.347 0.3209 0.306 0.2954\n 0.2622 0.246 ]\nFraction of each L0 OV's squared norm in shared top-2 write subspace:\n0 0.9123\n1 0.9385\n2 0.0028\n3 0.2485\n4 0.8528\n5 0.9345\n\nLater QK energy on shared top-2 write subspace and bias-key alignment\nhead: query_fraction key_fraction key_bias_fraction key_bias_coordinates\nL1H0 [0.0614 0.5748 0.7729] [7.137 6.341]\nL1H1 [0.0648 0.5364 0.8146] [13.096 10.032]\nL1H2 [0.0606 0.5491 0.8131] [14.859 11.302]\nL1H3 [0.0591 0.5547 0.7933] [12.977 10.594]\nL1H4 [0.0569 0.6026 0.8268] [13.741 9.42 ]\nL1H5 [0.0677 0.5488 0.7836] [9.62 7.843]\nL2H0 [0.0275 0.0919 0.8536] [10.353 7.109]\nL2H1 [0.0082 0.0444 0.7929] [4.547 2.734]\nL2H2 [0.0203 0.1766 0.8019] [4.247 3.356]\nL2H3 [0.0381 0.092 0.8678] [9.315 5.662]\nL2H4 [0.0304 0.0922 0.8624] [10.055 6.023]\nL2H5 [0.0306 0.0882 0.8535] [9.473 5.885]\nL3H0 [0.0014 0.0127 0.1481] [0.676 0.418]\nL3H1 [0.001 0.0023 0.0316] [-0.327 0.106]\nL3H2 [0.0118 0.0693 0.5917] [1.495 0.97 ]\nL3H3 [0.0162 0.0655 0.666 ] [2.074 1.111]\nL3H4 [0.0054 0.044 0.5194] [1.545 0.865]\nL3H5 [0.0029 0.0084 0.0312] [ 0.232 -0.016]\n\nL0H2 gain-weighted OV singular values: [0.7873 0.718 0.6827 0.6713 0.6378 0.6013 0.5692 0.5583 0.5504 0.5222 0.5151 0.4872 0.4805 0.4696 0.4391 0.4211 0.4006 0.3931 0.3471 0.3044 0.2772\n 0.2754 0.2565 0.2286 0.2181 0.205 0.1814 0.1707 0.1628 0.1593 0.1314 0.1184 0.1108 0.1014 0.0949 0.0807 0.0726 0.0649 0.0508 0.0469]\nProjected compositions on L0H2's top-27 read subspace:\nhead: trace(A Bprev) projected_trace symmetric_fraction positive/negative_sym_eigenmass\nL1H0 [-0.0978 -0.109 0.7328 0.2629 0.372 ]\nL1H1 [-0.0033 -0.0118 0.7879 0.7768 0.7886]\nL1H2 [0.458 0.4426 0.7303 1.138 0.6953]\nL1H3 [0.4099 0.4019 0.7498 0.9661 0.5643]\nL1H4 [0.3922 0.3805 0.7224 0.9042 0.5237]\nL1H5 [0.1614 0.1521 0.7166 0.9226 0.7705]\nL2H0 [-0.248 -0.2914 0.707 1.0645 1.3559]\nL2H1 [-0.2372 -0.2557 0.7393 0.8213 1.077 ]\nL2H2 [0.0327 0.0371 0.7256 0.635 0.5978]\nL2H3 [0.1216 0.1512 0.7026 0.9162 0.765 ]\nL2H4 [-0.3803 -0.3931 0.705 0.8113 1.2044]\nL2H5 [0.0158 0.0345 0.7101 1.0638 1.0293]\nL3H0 [-0.1019 -0.0942 0.7048 0.9675 1.0617]\nL3H1 [3.5295 3.5096 0.9062 3.7617 0.252 ]\nL3H2 [0.0903 0.0916 0.7098 0.6001 0.5085]\nL3H3 [0.0381 0.0281 0.6743 0.5445 0.5164]\nL3H4 [-0.1812 -0.1594 0.7495 0.6723 0.8317]\nL3H5 [0.0419 0.0467 0.7255 0.3519 0.3052]\n\nUnweighted L0H2 V singular values:\n[1.9539 1.8791 1.8467 1.8044 1.7716 1.7334 1.7213 1.6695 1.6478 1.5709 1.5539 1.5198 1.5167 1.4806 1.4422 1.331 1.2633 1.2307 1.0507 0.9934 0.9376\n 0.9084 0.843 0.8111 0.7783 0.7104 0.6947 0.6454 0.6367 0.599 0.5752 0.5655 0.5477 0.5344 0.5259 0.5052 0.4878 0.4767 0.4555 0.4349 0.427 0.4089\n 0.3946 0.3823 0.3712 0.3616 0.3501 0.3437 0.3341 0.323 0.3187 0.3139 0.302 0.2976 0.2964 0.2836 0.281 0.2722 0.2652 0.2598 0.2594 0.2497 0.2364\n 0.2305]\n"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 4 of at most 100: ``` Shared L0H0/H1/H4/H5 OV singular values: [11.996 8.4336 3.3624 1.312 0.9992 0.8615 0.7301 0.6058 0.5661 0.5402 0.4827 0.436 0.3932 0.3667 0.347 0.3209 0.306 0.2954 0.2622 0.246 ] Fraction of each L0 OV's squared norm in shared top-2 write subspace: 0 0.9123 1 0.9385 2 0.0028 3 0.2485 4 0.8528 5 0.9345 Later QK energy on shared top-2 write subspace and bias-key alignment head: query_fraction key_fraction key_bias_fraction key_bias_coordinates L1H0 [0.0614 0.5748 0.7729] [7.137 6.341] L1H1 [0.0648 0.5364 0.8146] [13.096 10.032] L1H2 [0.0606 0.5491 0.8131] [14.859 11.302] L1H3 [0.0591 0.5547 0.7933] [12.977 10.594] L1H4 [0.0569 0.6026 0.8268] [13.741 9.42 ] L1H5 [0.0677 0.5488 0.7836] [9.62 7.843] L2H0 [0.0275 0.0919 0.8536] [10.353 7.109] L2H1 [0.0082 0.0444 0.7929] [4.547 2.734] L2H2 [0.0203 0.1766 0.8019] [4.247 3.356] L2H3 [0.0381 0.092 0.8678] [9.315 5.662] L2H4 [0.0304 0.0922 0.8624] [10.055 6.023] L2H5 [0.0306 0.0882 0.8535] [9.473 5.885] L3H0 [0.0014 0.0127 0.1481] [0.676 0.418] L3H1 [0.001 0.0023 0.0316] [-0.327 0.106] L3H2 [0.0118 0.0693 0.5917] [1.495 0.97 ] L3H3 [0.0162 0.0655 0.666 ] [2.074 1.111] L3H4 [0.0054 0.044 0.5194] [1.545 0.865] L3H5 [0.0029 0.0084 0.0312] [ 0.232 -0.016] L0H2 gain-weighted OV singular values: [0.7873 0.718 0.6827 0.6713 0.6378 0.6013 0.5692 0.5583 0.5504 0.5222 0.5151 0.4872 0.4805 0.4696 0.4391 0.4211 0.4006 0.3931 0.3471 0.3044 0.2772 0.2754 0.2565 0.2286 0.2181 0.205 0.1814 0.1707 0.1628 0.1593 0.1314 0.1184 0.1108 0.1014 0.0949 0.0807 0.0726 0.0649 0.0508 0.0469] Projected compositions on L0H2's top-27 read subspace: head: trace(A Bprev) projected_trace symmetric_fraction positive/negative_sym_eigenmass L1H0 [-0.0978 -0.109 0.7328 0.2629 0.372 ] L1H1 [-0.0033 -0.0118 0.7879 0.7768 0.7886] L1H2 [0.458 0.4426 0.7303 1.138 0.6953] L1H3 [0.4099 0.4019 0.7498 0.9661 0.5643] L1H4 [0.3922 0.3805 0.7224 0.9042 0.5237] L1H5 [0.1614 0.1521 0.7166 0.9226 0.7705] L2H0 [-0.248 -0.2914 0.707 1.0645 1.3559] L2H1 [-0.2372 -0.2557 0.7393 0.8213 1.077 ] L2H2 [0.0327 0.0371 0.7256 0.635 0.5978] L2H3 [0.1216 0.1512 0.7026 0.9162 0.765 ] L2H4 [-0.3803 -0.3931 0.705 0.8113 1.2044] L2H5 [0.0158 0.0345 0.7101 1.0638 1.0293] L3H0 [-0.1019 -0.0942 0.7048 0.9675 1.0617] L3H1 [3.5295 3.5096 0.9062 3.7617 0.252 ] L3H2 [0.0903 0.0916 0.7098 0.6001 0.5085] L3H3 [0.0381 0.0281 0.6743 0.5445 0.5164] L3H4 [-0.1812 -0.1594 0.7495 0.6723 0.8317] L3H5 [0.0419 0.0467 0.7255 0.3519 0.3052] Unweighted L0H2 V singular values: [1.9539 1.8791 1.8467 1.8044 1.7716 1.7334 1.7213 1.6695 1.6478 1.5709 1.5539 1.5198 1.5167 1.4806 1.4422 1.331 1.2633 1.2307 1.0507 0.9934 0.9376 0.9084 0.843 0.8111 0.7783 0.7104 0.6947 0.6454 0.6367 0.599 0.5752 0.5655 0.5477 0.5344 0.5259 0.5052 0.4878 0.4767 0.4555 0.4349 0.427 0.4089 0.3946 0.3823 0.3712 0.3616 0.3501 0.3437 0.3341 0.323 0.3187 0.3139 0.302 0.2976 0.2964 0.2836 0.281 0.2722 0.2652 0.2598 0.2594 0.2497 0.2364 0.2305] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import scipy.linalg as la
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from threadpoolctl import threadpool_limits
threadpool_limits(4)
d=W["ln1_w.0"].astype(float)
v=W["W_v.0"][128:192].astype(float)*d
o=W["W_o.0"][:,128:192].astype(float)
B=o@v
F=W["W_fc.0"].astype(float)*W["ln2_w.0"]
G=F@B
u,s,vt=la.svd(G-G.mean(axis=0),full_matrices=False)
print("Singular values of centered W_fc.0 D_ln2.0 B_L0H2:")
print(np.round(s[:50],4))
# Clustering weight rows, not activations or model inputs.
for r in [20,24,27,32]:
X=(G-G.mean(axis=0))@vt[:r].T
X/=np.sqrt(s[:r])[None,:]
X/=np.maximum(la.norm(X,axis=1,keepdims=True),1e-12)
km=KMeans(n_clusters=r+1,n_init=10,random_state=0).fit(X)
sizes=np.bincount(km.labels_,minlength=r+1)
coh=np.array([la.norm(X[km.labels_==i].mean(axis=0)) for i in range(r+1)])
sil=silhouette_score(X,km.labels_,sample_size=1000,random_state=0)
print("rank",r,"clusters",r+1,"silhouette",round(sil,4),
"mean_cohesion",round(np.average(coh,weights=sizes),4))
print("sizes descending",np.sort(sizes)[::-1])
print("\nEffective QK symmetric eigenvalue masses, no input evaluation:")
for l in range(4):
d=W[f"ln1_w.{l}"].astype(float)
for j in range(6):
sl=slice(64*j,64*(j+1))
q=W[f"W_q.{l}"][sl].astype(float)*d
k=W[f"W_k.{l}"][sl].astype(float)*d
a=q.T@k
ev=la.eigvalsh((a+a.T)/2)
pos=ev[ev>0].sum()
neg=-ev[ev<0].sum()
print(f"L{l}H{j}", "positive",round(pos,3),"negative",round(neg,3),
"signed_mass_fraction",round((pos-neg)/(pos+neg),4))
```
{
"t": 1792713550.651643,
"s": 10.79,
"code": "import numpy as np\nimport scipy.linalg as la\nfrom sklearn.cluster import KMeans\nfrom sklearn.metrics import silhouette_score\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\nd=W[\"ln1_w.0\"].astype(float)\nv=W[\"W_v.0\"][128:192].astype(float)*d\no=W[\"W_o.0\"][:,128:192].astype(float)\nB=o@v\nF=W[\"W_fc.0\"].astype(float)*W[\"ln2_w.0\"]\nG=F@B\nu,s,vt=la.svd(G-G.mean(axis=0),full_matrices=False)\nprint(\"Singular values of centered W_fc.0 D_ln2.0 B_L0H2:\")\nprint(np.round(s[:50],4))\n\n# Clustering weight rows, not activations or model inputs.\nfor r in [20,24,27,32]:\n X=(G-G.mean(axis=0))@vt[:r].T\n X/=np.sqrt(s[:r])[None,:]\n X/=np.maximum(la.norm(X,axis=1,keepdims=True),1e-12)\n km=KMeans(n_clusters=r+1,n_init=10,random_state=0).fit(X)\n sizes=np.bincount(km.labels_,minlength=r+1)\n coh=np.array([la.norm(X[km.labels_==i].mean(axis=0)) for i in range(r+1)])\n sil=silhouette_score(X,km.labels_,sample_size=1000,random_state=0)\n print(\"rank\",r,\"clusters\",r+1,\"silhouette\",round(sil,4),\n \"mean_cohesion\",round(np.average(coh,weights=sizes),4))\n print(\"sizes descending\",np.sort(sizes)[::-1])\n\nprint(\"\\nEffective QK symmetric eigenvalue masses, no input evaluation:\")\nfor l in range(4):\n d=W[f\"ln1_w.{l}\"].astype(float)\n for j in range(6):\n sl=slice(64*j,64*(j+1))\n q=W[f\"W_q.{l}\"][sl].astype(float)*d\n k=W[f\"W_k.{l}\"][sl].astype(float)*d\n a=q.T@k\n ev=la.eigvalsh((a+a.T)/2)\n pos=ev[ev>0].sum()\n neg=-ev[ev<0].sum()\n print(f\"L{l}H{j}\", \"positive\",round(pos,3),\"negative\",round(neg,3),\n \"signed_mass_fraction\",round((pos-neg)/(pos+neg),4))\n",
"out": "Singular values of centered W_fc.0 D_ln2.0 B_L0H2:\n[3.6485 2.7127 2.5966 2.3846 2.3094 1.9764 1.9523 1.9044 1.7751 1.6206 1.6077 1.4051 1.3241 1.2869 1.1878 1.1569 1.0534 0.8885 0.6879 0.4863 0.406\n 0.3707 0.3187 0.2846 0.2552 0.2371 0.2254 0.1925 0.1892 0.1811 0.1509 0.1421 0.1287 0.113 0.1058 0.0921 0.0829 0.0712 0.0631 0.0553 0.0482 0.0476\n 0.04 0.034 0.0309 0.0276 0.0267 0.0212 0.0188 0.0177]\nrank 20 clusters 21 silhouette 0.0421 mean_cohesion 0.5708\nsizes descending [108 92 91 89 88 87 83 80 79 78 77 75 74 73 71 70 59 50 42 38 32]\nrank 24 clusters 25 silhouette 0.0385 mean_cohesion 0.5578\nsizes descending [87 82 82 79 79 77 76 69 67 67 66 63 59 58 56 55 55 53 50 48 46 45 45 37 35]\nrank 27 clusters 28 silhouette 0.038 mean_cohesion 0.5501\nsizes descending [82 80 77 77 75 69 68 68 67 60 60 58 58 57 55 53 53 50 44 42 42 41 39 39 39 35 24 24]\nrank 32 clusters 33 silhouette 0.0362 mean_cohesion 0.5417\nsizes descending [69 63 62 60 59 59 58 58 54 53 51 50 50 48 48 47 46 46 46 45 45 42 41 40 38 38 37 37 33 31 30 27 25]\n\nEffective QK symmetric eigenvalue masses, no input evaluation:\nL0H0 positive 0.526 negative 0.563 signed_mass_fraction -0.0348\nL0H1 positive 0.35 negative 0.462 signed_mass_fraction -0.1372\nL0H2 positive 57.4 negative 47.8 signed_mass_fraction 0.0913\nL0H3 positive 4.184 negative 2.609 signed_mass_fraction 0.2319\nL0H4 positive 0.591 negative 1.158 signed_mass_fraction -0.3243\nL0H5 positive 0.456 negative 0.733 signed_mass_fraction -0.2334\nL1H0 positive 8.996 negative 14.15 signed_mass_fraction -0.2227\nL1H1 positive 12.839 negative 22.133 signed_mass_fraction -0.2658\nL1H2 positive 14.061 negative 27.016 signed_mass_fraction -0.3154\nL1H3 positive 13.096 negative 22.354 signed_mass_fraction -0.2612\nL1H4 positive 13.29 negative 23.248 signed_mass_fraction -0.2725\nL1H5 positive 13.028 negative 21.144 signed_mass_fraction -0.2375\nL2H0 positive 62.04 negative 62.213 signed_mass_fraction -0.0014\nL2H1 positive 58.967 negative 54.73 signed_mass_fraction 0.0373\nL2H2 positive 38.345 negative 47.188 signed_mass_fraction -0.1034\nL2H3 positive 53.32 negative 59.226 signed_mass_fraction -0.0525\nL2H4 positive 60.195 negative 64.0 signed_mass_fraction -0.0306\nL2H5 positive 56.408 negative 61.986 signed_mass_fraction -0.0471\nL3H0 positive 78.679 negative 51.383 signed_mass_fraction 0.2099\nL3H1 positive 78.364 negative 27.915 signed_mass_fraction 0.4747\nL3H2 positive 42.154 negative 40.39 signed_mass_fraction 0.0214\nL3H3 positive 45.205 negative 44.312 signed_mass_fraction 0.01\nL3H4 positive 56.44 negative 49.211 signed_mass_fraction 0.0684\nL3H5 positive 22.97 negative 22.177 signed_mass_fraction 0.0176\n"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 5 of at most 100: ``` Singular values of centered W_fc.0 D_ln2.0 B_L0H2: [3.6485 2.7127 2.5966 2.3846 2.3094 1.9764 1.9523 1.9044 1.7751 1.6206 1.6077 1.4051 1.3241 1.2869 1.1878 1.1569 1.0534 0.8885 0.6879 0.4863 0.406 0.3707 0.3187 0.2846 0.2552 0.2371 0.2254 0.1925 0.1892 0.1811 0.1509 0.1421 0.1287 0.113 0.1058 0.0921 0.0829 0.0712 0.0631 0.0553 0.0482 0.0476 0.04 0.034 0.0309 0.0276 0.0267 0.0212 0.0188 0.0177] rank 20 clusters 21 silhouette 0.0421 mean_cohesion 0.5708 sizes descending [108 92 91 89 88 87 83 80 79 78 77 75 74 73 71 70 59 50 42 38 32] rank 24 clusters 25 silhouette 0.0385 mean_cohesion 0.5578 sizes descending [87 82 82 79 79 77 76 69 67 67 66 63 59 58 56 55 55 53 50 48 46 45 45 37 35] rank 27 clusters 28 silhouette 0.038 mean_cohesion 0.5501 sizes descending [82 80 77 77 75 69 68 68 67 60 60 58 58 57 55 53 53 50 44 42 42 41 39 39 39 35 24 24] rank 32 clusters 33 silhouette 0.0362 mean_cohesion 0.5417 sizes descending [69 63 62 60 59 59 58 58 54 53 51 50 50 48 48 47 46 46 46 45 45 42 41 40 38 38 37 37 33 31 30 27 25] Effective QK symmetric eigenvalue masses, no input evaluation: L0H0 positive 0.526 negative 0.563 signed_mass_fraction -0.0348 L0H1 positive 0.35 negative 0.462 signed_mass_fraction -0.1372 L0H2 positive 57.4 negative 47.8 signed_mass_fraction 0.0913 L0H3 positive 4.184 negative 2.609 signed_mass_fraction 0.2319 L0H4 positive 0.591 negative 1.158 signed_mass_fraction -0.3243 L0H5 positive 0.456 negative 0.733 signed_mass_fraction -0.2334 L1H0 positive 8.996 negative 14.15 signed_mass_fraction -0.2227 L1H1 positive 12.839 negative 22.133 signed_mass_fraction -0.2658 L1H2 positive 14.061 negative 27.016 signed_mass_fraction -0.3154 L1H3 positive 13.096 negative 22.354 signed_mass_fraction -0.2612 L1H4 positive 13.29 negative 23.248 signed_mass_fraction -0.2725 L1H5 positive 13.028 negative 21.144 signed_mass_fraction -0.2375 L2H0 positive 62.04 negative 62.213 signed_mass_fraction -0.0014 L2H1 positive 58.967 negative 54.73 signed_mass_fraction 0.0373 L2H2 positive 38.345 negative 47.188 signed_mass_fraction -0.1034 L2H3 positive 53.32 negative 59.226 signed_mass_fraction -0.0525 L2H4 positive 60.195 negative 64.0 signed_mass_fraction -0.0306 L2H5 positive 56.408 negative 61.986 signed_mass_fraction -0.0471 L3H0 positive 78.679 negative 51.383 signed_mass_fraction 0.2099 L3H1 positive 78.364 negative 27.915 signed_mass_fraction 0.4747 L3H2 positive 42.154 negative 40.39 signed_mass_fraction 0.0214 L3H3 positive 45.205 negative 44.312 signed_mass_fraction 0.01 L3H4 positive 56.44 negative 49.211 signed_mass_fraction 0.0684 L3H5 positive 22.97 negative 22.177 signed_mass_fraction 0.0176 ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import scipy.linalg as la
from threadpoolctl import threadpool_limits
threadpool_limits(4)
d=W["ln1_w.0"].astype(float)
A,B={},{}
for j in range(6):
sl=slice(64*j,64*(j+1))
q=W["W_q.0"][sl].astype(float)*d
k=W["W_k.0"][sl].astype(float)*d
v=W["W_v.0"][sl].astype(float)*d
o=W["W_o.0"][:,sl].astype(float)
A[j]=q.T@k
B[j]=o@v
# Compare weight matrices with powers of the orthogonal polar factor
# of L0H2's QK matrix. These are matrix alignments, not attention scores.
q=W["W_q.0"][128:192].astype(float)*d
k=W["W_k.0"][128:192].astype(float)*d
c=q.T@q+k.T@k
ev,U=la.eigh(c)
ix=np.argsort(ev)[::-1]
ev,U=ev[ix],U[:,ix]
for r in [16,20,24,32]:
Z=U[:,:r]/np.sqrt(ev[:r])[None,:]
t=Z.T@A[2]@Z
rot,_=la.polar(t)
print("\nrank",r,"whitened L0H2 singular values",np.round(la.svdvals(t),3))
print("Alignment with polar-factor powers, columns -4..4:")
for j in [0,1,3,4,5]:
m=Z.T@A[j]@Z
vals=[np.sum(m*np.linalg.matrix_power(rot,p))/
(la.norm(m)*np.sqrt(r)) for p in range(-4,5)]
print(f"L0H{j}",np.round(vals,4))
print("\nShared first-layer read subspace and positional-circuit overlap")
stack=np.concatenate([B[j] for j in [0,1,4,5]],axis=0)
_,s,vt=la.svd(stack,full_matrices=False)
print("Shared-read singular values",np.round(s[:10],4))
a_u,a_s,a_vt=la.svd(A[2],full_matrices=False)
P=la.orth(np.column_stack([a_u[:,:20],a_vt[:20].T]))
for i in range(4):
v=vt[i]
print("read direction",i,"energy in L0H2 QK subspace",
round(la.norm(P.T@v)**2,4))
# Check the induction-like composition without choosing one particular rank.
a=(W["W_q.3"][64:128].astype(float)*W["ln1_w.3"]).T @ (
W["W_k.3"][64:128].astype(float)*W["ln1_w.3"])
_,_,vt=la.svd(B[2],full_matrices=False)
print("\nL3H1 x L0H2 induction-like composition, subspace-rank sensitivity")
for r in [16,20,24,27,32,40,64]:
T=vt[:r].T
m=T.T@a@B[2]@T
e=la.eigvalsh((m+m.T)/2)
print(r,"trace",round(np.trace(m),4),
"positive_mass",round(e[e>0].sum(),4),
"negative_mass",round(-e[e<0].sum(),4),
"symmetry_fraction",round(la.norm((m+m.T)/2)/la.norm(m),4))
```
{
"t": 1792713682.5713809,
"s": 3.14,
"code": "import numpy as np\nimport scipy.linalg as la\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\nd=W[\"ln1_w.0\"].astype(float)\nA,B={},{}\nfor j in range(6):\n sl=slice(64*j,64*(j+1))\n q=W[\"W_q.0\"][sl].astype(float)*d\n k=W[\"W_k.0\"][sl].astype(float)*d\n v=W[\"W_v.0\"][sl].astype(float)*d\n o=W[\"W_o.0\"][:,sl].astype(float)\n A[j]=q.T@k\n B[j]=o@v\n\n# Compare weight matrices with powers of the orthogonal polar factor\n# of L0H2's QK matrix. These are matrix alignments, not attention scores.\nq=W[\"W_q.0\"][128:192].astype(float)*d\nk=W[\"W_k.0\"][128:192].astype(float)*d\nc=q.T@q+k.T@k\nev,U=la.eigh(c)\nix=np.argsort(ev)[::-1]\nev,U=ev[ix],U[:,ix]\nfor r in [16,20,24,32]:\n Z=U[:,:r]/np.sqrt(ev[:r])[None,:]\n t=Z.T@A[2]@Z\n rot,_=la.polar(t)\n print(\"\\nrank\",r,\"whitened L0H2 singular values\",np.round(la.svdvals(t),3))\n print(\"Alignment with polar-factor powers, columns -4..4:\")\n for j in [0,1,3,4,5]:\n m=Z.T@A[j]@Z\n vals=[np.sum(m*np.linalg.matrix_power(rot,p))/\n (la.norm(m)*np.sqrt(r)) for p in range(-4,5)]\n print(f\"L0H{j}\",np.round(vals,4))\n\nprint(\"\\nShared first-layer read subspace and positional-circuit overlap\")\nstack=np.concatenate([B[j] for j in [0,1,4,5]],axis=0)\n_,s,vt=la.svd(stack,full_matrices=False)\nprint(\"Shared-read singular values\",np.round(s[:10],4))\na_u,a_s,a_vt=la.svd(A[2],full_matrices=False)\nP=la.orth(np.column_stack([a_u[:,:20],a_vt[:20].T]))\nfor i in range(4):\n v=vt[i]\n print(\"read direction\",i,\"energy in L0H2 QK subspace\",\n round(la.norm(P.T@v)**2,4))\n\n# Check the induction-like composition without choosing one particular rank.\na=(W[\"W_q.3\"][64:128].astype(float)*W[\"ln1_w.3\"]).T @ (\n W[\"W_k.3\"][64:128].astype(float)*W[\"ln1_w.3\"])\n_,_,vt=la.svd(B[2],full_matrices=False)\nprint(\"\\nL3H1 x L0H2 induction-like composition, subspace-rank sensitivity\")\nfor r in [16,20,24,27,32,40,64]:\n T=vt[:r].T\n m=T.T@a@B[2]@T\n e=la.eigvalsh((m+m.T)/2)\n print(r,\"trace\",round(np.trace(m),4),\n \"positive_mass\",round(e[e>0].sum(),4),\n \"negative_mass\",round(-e[e<0].sum(),4),\n \"symmetry_fraction\",round(la.norm((m+m.T)/2)/la.norm(m),4))\n",
"out": "\nrank 16 whitened L0H2 singular values [0.617 0.575 0.566 0.53 0.521 0.507 0.493 0.472 0.468 0.443 0.419 0.405 0.391 0.37 0.083 0.017]\nAlignment with polar-factor powers, columns -4..4:\nL0H0 [-0.026 0.076 0.1024 -0.018 -0.0802 -0.1375 -0.0964 0.0035 0.1072]\nL0H1 [ 0.042 0.0077 0.007 -0.0733 -0.0518 -0.0235 0.0194 -0.0014 0.0007]\nL0H3 [-0.2319 -0.2995 -0.1674 0.1027 0.2253 0.2039 -0.0107 -0.2293 -0.2839]\nL0H4 [ 0.1195 0.0721 -0.047 -0.1282 -0.1541 0.0002 0.1201 0.0769 -0.0293]\nL0H5 [ 0.1541 0.1988 0.0798 -0.0938 -0.2027 -0.1472 0.0305 0.1617 0.1551]\n\nrank 20 whitened L0H2 singular values [0.646 0.605 0.596 0.588 0.558 0.541 0.527 0.525 0.51 0.501 0.461 0.436 0.429 0.427 0.414 0.394 0.38 0.291 0.243 0.235]\nAlignment with polar-factor powers, columns -4..4:\nL0H0 [-0.0006 0.0326 0.0802 0.0299 -0.0569 -0.0544 -0.0776 -0.0236 0.0441]\nL0H1 [ 0.0427 0.0052 -0.0097 -0.0298 -0.0768 0.0005 0.0062 0.005 -0.0008]\nL0H3 [-0.1689 -0.2718 -0.164 0.0997 0.1326 0.199 -0.0295 -0.2152 -0.2294]\nL0H4 [ 0.0984 0.0222 -0.0847 -0.1366 -0.1324 -0.0288 0.0773 0.0386 -0.0272]\nL0H5 [ 0.1218 0.1262 0.0674 -0.0784 -0.1501 -0.1069 0.0388 0.1069 0.1294]\n\nrank 24 whitened L0H2 singular values [0.695 0.655 0.624 0.617 0.606 0.591 0.571 0.565 0.543 0.54 0.521 0.504 0.495 0.488 0.454 0.428 0.412 0.39 0.351 0.338 0.283 0.201 0.095 0.025]\nAlignment with polar-factor powers, columns -4..4:\nL0H0 [-0.009 0.0091 0.0543 0.0603 0.051 -0.012 0.0038 -0.035 0.0292]\nL0H1 [ 0.0074 0.0155 0.0067 0.0354 -0.0988 0.0211 0.0051 -0.0227 -0.0332]\nL0H3 [-0.1302 -0.1833 -0.077 0.0664 0.0764 0.1546 0.0565 -0.0982 -0.1215]\nL0H4 [ 0.0085 0.0169 -0.0877 0.059 -0.1798 0.055 -0.082 0.0621 -0.077 ]\nL0H5 [ 0.027 0.0892 0.0264 0.0462 -0.1206 0.0031 -0.0609 0.0293 0.0147]\n\nrank 32 whitened L0H2 singular values [0.89 0.764 0.721 0.711 0.684 0.67 0.651 0.634 0.623 0.612 0.593 0.586 0.57 0.562 0.539 0.526 0.514 0.474 0.457 0.417 0.385 0.356 0.345 0.311\n 0.282 0.256 0.188 0.121 0.049 0.03 0.013 0. ]\nAlignment with polar-factor powers, columns -4..4:\nL0H0 [-0.0058 -0.0268 -0.0259 0.0236 0.0453 -0.0054 -0.0143 0.0132 0.0138]\nL0H1 [-0.0239 -0.0035 -0.0019 -0.0149 -0.0328 -0.023 0.0047 0.0314 0.0027]\nL0H3 [-0.0339 0.0386 0.0535 -0.0087 -0.0414 0.1335 0.013 -0.0334 0.0008]\nL0H4 [-0.0246 -0.0487 0.047 0.0855 -0.1421 -0.0777 0.0692 -0.0124 -0.0669]\nL0H5 [-0.0346 -0.0258 0.0829 0.0006 -0.1251 0.0222 0.092 -0.0212 -0.0558]\n\nShared first-layer read subspace and positional-circuit overlap\nShared-read singular values [12.4276 7.5882 3.5107 1.4158 1.1895 0.935 0.8361 0.731 0.6597 0.558 ]\nread direction 0 energy in L0H2 QK subspace 0.9829\nread direction 1 energy in L0H2 QK subspace 0.9488\nread direction 2 energy in L0H2 QK subspace 0.8122\nread direction 3 energy in L0H2 QK subspace 0.4707\n\nL3H1 x L0H2 induction-like composition, subspace-rank sensitivity\n16 trace 3.2581 positive_mass 3.2621 negative_mass 0.0039 symmetry_fraction 0.9338\n20 trace 3.4768 positive_mass 3.5445 negative_mass 0.0677 symmetry_fraction 0.9243\n24 trace 3.4982 positive_mass 3.6684 negative_mass 0.1702 symmetry_fraction 0.9143\n27 trace 3.5096 positive_mass 3.7617 negative_mass 0.252 symmetry_fraction 0.9062\n32 trace 3.5278 positive_mass 3.8829 negative_mass 0.3551 symmetry_fraction 0.8961\n40 trace 3.5302 positive_mass 4.081 negative_mass 0.5508 symmetry_fraction 0.8772\n64 trace 3.5295 positive_mass 4.643 negative_mass 1.1134 symmetry_fraction 0.8402\n"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 6 of at most 100: ``` rank 16 whitened L0H2 singular values [0.617 0.575 0.566 0.53 0.521 0.507 0.493 0.472 0.468 0.443 0.419 0.405 0.391 0.37 0.083 0.017] Alignment with polar-factor powers, columns -4..4: L0H0 [-0.026 0.076 0.1024 -0.018 -0.0802 -0.1375 -0.0964 0.0035 0.1072] L0H1 [ 0.042 0.0077 0.007 -0.0733 -0.0518 -0.0235 0.0194 -0.0014 0.0007] L0H3 [-0.2319 -0.2995 -0.1674 0.1027 0.2253 0.2039 -0.0107 -0.2293 -0.2839] L0H4 [ 0.1195 0.0721 -0.047 -0.1282 -0.1541 0.0002 0.1201 0.0769 -0.0293] L0H5 [ 0.1541 0.1988 0.0798 -0.0938 -0.2027 -0.1472 0.0305 0.1617 0.1551] rank 20 whitened L0H2 singular values [0.646 0.605 0.596 0.588 0.558 0.541 0.527 0.525 0.51 0.501 0.461 0.436 0.429 0.427 0.414 0.394 0.38 0.291 0.243 0.235] Alignment with polar-factor powers, columns -4..4: L0H0 [-0.0006 0.0326 0.0802 0.0299 -0.0569 -0.0544 -0.0776 -0.0236 0.0441] L0H1 [ 0.0427 0.0052 -0.0097 -0.0298 -0.0768 0.0005 0.0062 0.005 -0.0008] L0H3 [-0.1689 -0.2718 -0.164 0.0997 0.1326 0.199 -0.0295 -0.2152 -0.2294] L0H4 [ 0.0984 0.0222 -0.0847 -0.1366 -0.1324 -0.0288 0.0773 0.0386 -0.0272] L0H5 [ 0.1218 0.1262 0.0674 -0.0784 -0.1501 -0.1069 0.0388 0.1069 0.1294] rank 24 whitened L0H2 singular values [0.695 0.655 0.624 0.617 0.606 0.591 0.571 0.565 0.543 0.54 0.521 0.504 0.495 0.488 0.454 0.428 0.412 0.39 0.351 0.338 0.283 0.201 0.095 0.025] Alignment with polar-factor powers, columns -4..4: L0H0 [-0.009 0.0091 0.0543 0.0603 0.051 -0.012 0.0038 -0.035 0.0292] L0H1 [ 0.0074 0.0155 0.0067 0.0354 -0.0988 0.0211 0.0051 -0.0227 -0.0332] L0H3 [-0.1302 -0.1833 -0.077 0.0664 0.0764 0.1546 0.0565 -0.0982 -0.1215] L0H4 [ 0.0085 0.0169 -0.0877 0.059 -0.1798 0.055 -0.082 0.0621 -0.077 ] L0H5 [ 0.027 0.0892 0.0264 0.0462 -0.1206 0.0031 -0.0609 0.0293 0.0147] rank 32 whitened L0H2 singular values [0.89 0.764 0.721 0.711 0.684 0.67 0.651 0.634 0.623 0.612 0.593 0.586 0.57 0.562 0.539 0.526 0.514 0.474 0.457 0.417 0.385 0.356 0.345 0.311 0.282 0.256 0.188 0.121 0.049 0.03 0.013 0. ] Alignment with polar-factor powers, columns -4..4: L0H0 [-0.0058 -0.0268 -0.0259 0.0236 0.0453 -0.0054 -0.0143 0.0132 0.0138] L0H1 [-0.0239 -0.0035 -0.0019 -0.0149 -0.0328 -0.023 0.0047 0.0314 0.0027] L0H3 [-0.0339 0.0386 0.0535 -0.0087 -0.0414 0.1335 0.013 -0.0334 0.0008] L0H4 [-0.0246 -0.0487 0.047 0.0855 -0.1421 -0.0777 0.0692 -0.0124 -0.0669] L0H5 [-0.0346 -0.0258 0.0829 0.0006 -0.1251 0.0222 0.092 -0.0212 -0.0558] Shared first-layer read subspace and positional-circuit overlap Shared-read singular values [12.4276 7.5882 3.5107 1.4158 1.1895 0.935 0.8361 0.731 0.6597 0.558 ] read direction 0 energy in L0H2 QK subspace 0.9829 read direction 1 energy in L0H2 QK subspace 0.9488 read direction 2 energy in L0H2 QK subspace 0.8122 read direction 3 energy in L0H2 QK subspace 0.4707 L3H1 x L0H2 induction-like composition, subspace-rank sensitivity 16 trace 3.2581 positive_mass 3.2621 negative_mass 0.0039 symmetry_fraction 0.9338 20 trace 3.4768 positive_mass 3.5445 negative_mass 0.0677 symmetry_fraction 0.9243 24 trace 3.4982 positive_mass 3.6684 negative_mass 0.1702 symmetry_fraction 0.9143 27 trace 3.5096 positive_mass 3.7617 negative_mass 0.252 symmetry_fraction 0.9062 32 trace 3.5278 positive_mass 3.8829 negative_mass 0.3551 symmetry_fraction 0.8961 40 trace 3.5302 positive_mass 4.081 negative_mass 0.5508 symmetry_fraction 0.8772 64 trace 3.5295 positive_mass 4.643 negative_mass 1.1134 symmetry_fraction 0.8402 ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
## Observations
These are **weight-only inferences, not measured attention patterns**. Exact distances and character identities remain uncertain.
Define \(A_{lh}=D_lW_{q,lh}^{T}W_{k,lh}D_l\) and \(B_{lh}=W_{o,lh}W_{v,lh}D_l\), where \(D_l=\operatorname{diag}(\text{ln1_w.l})\).
- **One exceptional positional-selector candidate:** L0H2 has \(\|A\|_F=32.627\), versus 0.236–3.157 for the other first-layer heads. Its substantial complex eigenpairs, including \(5.058\pm6.513i\), suggest a shift-like positional circuit. “Previous character” is plausible, but the missing position embeddings prevent identifying its lag conclusively.
- **Four shared positional-summary heads:** L0H0/H1/H4/H5 put respectively **91.2%, 93.9%, 85.3%, 93.5%** of their OV energy into the same two-dimensional write subspace, called \(C\) below. Their two dominant shared read directions overlap the L0H2 QK subspace by **98.3% and 94.9%**. This suggests positional summaries subsequently used for routing.
- **A particularly strong induction-like connection:** L3H1 uniquely aligns with L0H2’s OV map: normalized \(\operatorname{tr}(A_{31}B_{02})=0.0949\). Restricted to the strongest 16 read directions, the symmetric composition has positive eigenvalue mass **3.2621**, versus negative mass **0.0039**. Conditional on L0H2 transporting the previous character, this is compelling successor-matching evidence.
- **Corpus identification is much weaker.** `W_proj.3` has singular values \(s_1=26.309,\ s_{23}=10.367,\ s_{28}=7.166\): substantial, diverse learned output directions. This does not recover letters or uniquely identify a language. Weight-row clustering also did not reveal clean character identities.
## Heads
Roles below are provisional. “Clock-key energy” means QK energy reading the shared subspace \(C\); bias alignment refers to the key-linear weight direction generated by `ln1_b`.
L0H0: **broad** — small QK norm 0.437; 91.2% shared positional-summary write energy.
L0H1: **broad** — smallest QK norm, 0.236; 93.9% shared write energy.
L0H2: **previous** — exceptional, shift-like QK structure and the downstream induction-like connection; exact lag unverified.
L0H3: **local** — QK matrix cosine 0.239 with L0H2; weaker, lower-rank positional selection.
L0H4: **broad** — QK norm 0.851; 85.3% shared write energy.
L0H5: **broad** — QK norm 0.621; 93.5% shared write energy.
L1H0: **local** — clock-key energy 57.5%; bias alignment 77.3%.
L1H1: **local** — clock-key energy 53.6%; bias alignment 81.5%.
L1H2: **local** — clock-key energy 54.9%; strong clock-aligned key bias, norm 20.70.
L1H3: **local** — clock-key energy 55.5%; bias alignment 79.3%.
L1H4: **local** — highest L1 clock-key energy, 60.3%; bias alignment 82.7%.
L1H5: **local** — clock-key energy 54.9%; bias alignment 78.4%.
L2H0: **local** — strong key bias, norm 13.59, with 85.4% clock alignment.
L2H1: **local** — weaker positional bias, norm 5.96; 79.3% clock alignment. Lower-confidence assignment.
L2H2: **local** — largest L2 clock-key energy, 17.7%; bias alignment 80.2%.
L2H3: **local** — key-bias norm 11.70; clock alignment 86.8%.
L2H4: **local** — key-bias norm 12.62; clock alignment 86.2%.
L2H5: **local** — key-bias norm 12.07; clock alignment 85.4%.
L3H0: **unclear** — positive QK trace 27.30 suggests content matching, but no clear successor-matching composition.
L3H1: **copy** — uniquely strong, predominantly positive composition with L0H2; strongest induction candidate.
L3H2: **unclear** — modest clock-key energy, 6.9%; neither distance nor induction is established.
L3H3: **unclear** — modest clock-key energy, 6.5%; weak successor-composition evidence.
L3H4: **unclear** — clock-key energy 4.4%; content-dependent routing likely, but its range is unresolved.
L3H5: **broad** — weakest final-layer QK norm, 6.679, and only 0.84% clock-key energy; strong OV map suggests aggregation.
## Reasoning
The strongest interpretation is **positional summaries → nearby-character composition → later content matching and copying**. Such an organized successor-matching circuit is substantially more useful for intact text than for independently shuffled lines.
**English is a weak corpus preference**, not a decoded identification. The diverse feature structure is compatible with English spelling and repeated words; Python remains a substantial alternative. The spectrum provides, at most, weak evidence against romaji’s more restricted orthography. The ordered-versus-shuffled conclusion is much stronger than the corpus choice.
corpus: english
order: as is
probabilities: python=0.35 romaji=0.17 english=0.48; as is=0.98 shuffled=0.02## Observations
These are **weight-only inferences, not measured attention patterns**. Exact distances and character identities remain uncertain.
Define \(A_{lh}=D_lW_{q,lh}^{T}W_{k,lh}D_l\) and \(B_{lh}=W_{o,lh}W_{v,lh}D_l\), where \(D_l=\operatorname{diag}(\text{ln1_w.l})\).
- **One exceptional positional-selector candidate:** L0H2 has \(\|A\|_F=32.627\), versus 0.236–3.157 for the other first-layer heads. Its substantial complex eigenpairs, including \(5.058\pm6.513i\), suggest a shift-like positional circuit. “Previous character” is plausible, but the missing position embeddings prevent identifying its lag conclusively.
- **Four shared positional-summary heads:** L0H0/H1/H4/H5 put respectively **91.2%, 93.9%, 85.3%, 93.5%** of their OV energy into the same two-dimensional write subspace, called \(C\) below. Their two dominant shared read directions overlap the L0H2 QK subspace by **98.3% and 94.9%**. This suggests positional summaries subsequently used for routing.
- **A particularly strong induction-like connection:** L3H1 uniquely aligns with L0H2’s OV map: normalized \(\operatorname{tr}(A_{31}B_{02})=0.0949\). Restricted to the strongest 16 read directions, the symmetric composition has positive eigenvalue mass **3.2621**, versus negative mass **0.0039**. Conditional on L0H2 transporting the previous character, this is compelling successor-matching evidence.
- **Corpus identification is much weaker.** `W_proj.3` has singular values \(s_1=26.309,\ s_{23}=10.367,\ s_{28}=7.166\): substantial, diverse learned output directions. This does not recover letters or uniquely identify a language. Weight-row clustering also did not reveal clean character identities.
## Heads
Roles below are provisional. “Clock-key energy” means QK energy reading the shared subspace \(C\); bias alignment refers to the key-linear weight direction generated by `ln1_b`.
L0H0: **broad** — small QK norm 0.437; 91.2% shared positional-summary write energy.
L0H1: **broad** — smallest QK norm, 0.236; 93.9% shared write energy.
L0H2: **previous** — exceptional, shift-like QK structure and the downstream induction-like connection; exact lag unverified.
L0H3: **local** — QK matrix cosine 0.239 with L0H2; weaker, lower-rank positional selection.
L0H4: **broad** — QK norm 0.851; 85.3% shared write energy.
L0H5: **broad** — QK norm 0.621; 93.5% shared write energy.
L1H0: **local** — clock-key energy 57.5%; bias alignment 77.3%.
L1H1: **local** — clock-key energy 53.6%; bias alignment 81.5%.
L1H2: **local** — clock-key energy 54.9%; strong clock-aligned key bias, norm 20.70.
L1H3: **local** — clock-key energy 55.5%; bias alignment 79.3%.
L1H4: **local** — highest L1 clock-key energy, 60.3%; bias alignment 82.7%.
L1H5: **local** — clock-key energy 54.9%; bias alignment 78.4%.
L2H0: **local** — strong key bias, norm 13.59, with 85.4% clock alignment.
L2H1: **local** — weaker positional bias, norm 5.96; 79.3% clock alignment. Lower-confidence assignment.
L2H2: **local** — largest L2 clock-key energy, 17.7%; bias alignment 80.2%.
L2H3: **local** — key-bias norm 11.70; clock alignment 86.8%.
L2H4: **local** — key-bias norm 12.62; clock alignment 86.2%.
L2H5: **local** — key-bias norm 12.07; clock alignment 85.4%.
L3H0: **unclear** — positive QK trace 27.30 suggests content matching, but no clear successor-matching composition.
L3H1: **copy** — uniquely strong, predominantly positive composition with L0H2; strongest induction candidate.
L3H2: **unclear** — modest clock-key energy, 6.9%; neither distance nor induction is established.
L3H3: **unclear** — modest clock-key energy, 6.5%; weak successor-composition evidence.
L3H4: **unclear** — clock-key energy 4.4%; content-dependent routing likely, but its range is unresolved.
L3H5: **broad** — weakest final-layer QK norm, 6.679, and only 0.84% clock-key energy; strong OV map suggests aggregation.
## Reasoning
The strongest interpretation is **positional summaries → nearby-character composition → later content matching and copying**. Such an organized successor-matching circuit is substantially more useful for intact text than for independently shuffled lines.
**English is a weak corpus preference**, not a decoded identification. The diverse feature structure is compatible with English spelling and repeated words; Python remains a substantial alternative. The spectrum provides, at most, weak evidence against romaji’s more restricted orthography. The ordered-versus-shuffled conclusion is much stronger than the corpus choice.
corpus: english
order: as is
probabilities: python=0.35 romaji=0.17 english=0.48; as is=0.98 shuffled=0.02