Built-in Codex tools were disabled. Harness execution is separately permitted only in Experiments 3 and 6. Text below is taken from saved records; it is not a new model run. Reasoning blocks are the summaries actually emitted by the CLI.
Answer the user's request.
Exact prompt · Raw CLI events · Settings and audit
Below is a small character-level language model: a 4-layer transformer with 6 attention heads per layer, which reads up to 128 characters and outputs scores (logits) for the next character. This is exactly how it computes (numpy):
```python
import numpy as np
VOCAB = "\n " + "abcdefghijklmnopqrstuvwxyz" # index 0 = newline, 1 = space, 2..27 = a..z
HEADS, LAYERS = 6, 4
def layer_norm(x, w, b): # over the last axis, eps 1e-5
mu = x.mean(-1, keepdims=True)
var = ((x - mu) ** 2).mean(-1, keepdims=True)
return (x - mu) / np.sqrt(var + 1e-5) * w + b
def next_char_logits(text, p):
"""text: up to 128 characters (only the last 128 are used); p: dict of the weight arrays.
Returns the 28 logits for the character that comes next."""
ids = [VOCAB.index(c) for c in text[-128:]]
n, d = len(ids), p["tok_emb"].shape[1]
hd = d // HEADS
x = p["tok_emb"][ids] + p["pos_emb"][:n] # (n, d); positions 0..n-1 from the start of the text
for l in range(LAYERS): # layer l uses the tensors named <name>.<l>
h = layer_norm(x, p[f"ln1_w.{l}"], p[f"ln1_b.{l}"])
q, k, v = h @ p[f"W_q.{l}"].T, h @ p[f"W_k.{l}"].T, h @ p[f"W_v.{l}"].T
out = np.zeros((n, d))
for j in range(HEADS): # head j uses dimensions j*hd .. (j+1)*hd - 1
s = slice(j * hd, (j + 1) * hd)
att = q[:, s] @ k[:, s].T / np.sqrt(hd) # (n, n)
att = att + np.triu(np.full((n, n), -np.inf), 1) # causal: position i sees positions 0..i
att = np.exp(att - att.max(-1, keepdims=True))
att = att / att.sum(-1, keepdims=True)
out[:, s] = att @ v[:, s]
x = x + out @ p[f"W_o.{l}"].T
h = layer_norm(x, p[f"ln2_w.{l}"], p[f"ln2_b.{l}"])
x = x + np.maximum(0.0, h @ p[f"W_fc.{l}"].T) @ p[f"W_proj.{l}"].T # ReLU MLP, hidden 4 d
h = layer_norm(x, p["lnf_w"], p["lnf_b"])
return h[-1] @ p["W_out"].T + p["b_out"] # logits of the next character (28)
```
Training: from random initialization (every matrix entry drawn from a normal distribution with standard deviation 0.02, or 0.007071 for W_o and W_proj; LayerNorm weights 1 and biases 0; b_out 0), next-character prediction with AdamW (30,517 steps of 256 random 128-character windows, 1,000,000,000 characters in total; learning-rate warm-up then cosine decay; weight decay 0.01 on the matrices).
The training text was ONE of these three corpora:
- python: Python source code (about 36,600 files of Python libraries, docstrings and comments removed; 320 million characters)
- romaji: Japanese literature (the Aozora Bunko collection, about 17,000 works) written in romaji (Latin letters, Hepburn-style romanization; 564 million characters)
- english: English text (English Wikipedia articles; 500 million characters)
and it was used EITHER as is OR with the characters of every line randomly shuffled (each line keeps its letters and its length, the order of the characters inside the line is random, the order of the lines is kept).
Normalization (the same for all three corpora, before anything else): lowercased; in each line the leading spaces (indentation) were kept, every other character that is not a letter a-z became a space, runs of spaces were collapsed to one and trailing spaces removed. So the text contains only newlines, spaces and the letters a-z (the 28 characters of VOCAB).
WHAT YOU CAN USE. The weights inside the four transformer blocks, exactly as trained (float32): for layer l = 0, 1, 2 and 3, ln1_w.l, ln1_b.l, W_q.l, W_k.l, W_v.l, W_o.l (attention) and ln2_w.l, ln2_b.l, W_fc.l, W_proj.l (MLP). The input layer (tok_emb, the 28 x d character embeddings, and pos_emb, the 128 x d position embeddings) and the output layer (lnf_w, lnf_b, W_out, b_out) are NOT available. The d = 384 dimensions of the residual stream carry no labels. Head j of layer l uses rows 64j .. 64j+63 of W_q.l, W_k.l and W_v.l and columns 64j .. 64j+63 of W_o.l.
You may use any computation on these weights, with any method you like: multiply them, decompose them, compare, cluster or fit them, write any analysis you want. Two things are NOT allowed:
1. A forward pass: do not run the network or any of its layers (LayerNorm, attention with its softmax, the MLP with its ReLU) on any input, whether real, random or constructed, in code, in your head or on paper. Treating the weight matrices as linear-algebra objects (products, eigenvalues, singular vectors, applying one weight matrix to directions taken from other weights) is fine; simulating what the network does to an input is not.
2. Search: do not look anything up (there is no internet access in any case).
HOW YOU COMPUTE. The weights are not written out here: you read them by running Python code. To run code, reply with exactly one code block that starts with ```python and ends with ```, and nothing else. We run it and reply with its printed output. Available: numpy, scipy, scikit-learn, PyTorch (CPU), pandas, sympy, networkx. Before your code runs, numpy is imported as np and W is a dict of the weight arrays by name: W['W_q.0'] is 384 x 384, W['W_fc.0'] is 1536 x 384, W['W_proj.0'] is 384 x 1536, W['ln1_w.0'] has 384 entries, and so on; matrices are (out, in), as used in the code above. There is nothing else to read (no other files, no network). Each run is a fresh process (nothing is kept between runs), may take up to 600 seconds, and its output is cut at 20,000 characters. You may run code up to 100 times. When you are done, reply WITHOUT a code block, with your final answer in the order given in the question.
QUESTION. What was this network trained on, and how does it use the context? Answer in this order:
Observations: <what you see in the shown weights and what it suggests; quote the weights you rely on>
Heads: <one line per head, L0H0 .. L0H5, L1H0 .. L1H5, L2H0 .. L2H5, L3H0 .. L3H5: what it most likely does, as one of: 'previous' (attends to the previous character), 'local' (a few characters back), 'broad' (spread far back), 'copy' (induction: attends to the character right after an earlier occurrence of the current character or context, so the network can repeat earlier text), or 'unclear'; then your evidence>
Reasoning: <which training text fits these observations, and why>
corpus: <python | romaji | english>
order: <as is | shuffled>
probabilities: python=<p> romaji=<p> english=<p>; as is=<p> shuffled=<p>
Equal probabilities are not allowed. Even if the evidence is weak, give your best judgement.Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(4)
def summary(a):
return "norm=%.3f mean=%.5f std=%.5f min=%.4f max=%.4f" % (
np.linalg.norm(a), a.mean(), a.std(), a.min(), a.max())
for l in range(4):
print("\nLAYER", l)
for name in ["ln1_w", "ln1_b", "ln2_w", "ln2_b", "W_q", "W_k", "W_v", "W_o", "W_fc", "W_proj"]:
print(name, summary(W[f"{name}.{l}"]))
print("head: Qnorm Knorm Vnorm Onorm | QK sv(top6), top1_energy, symmetry, trace/norm | OV sv(top4), trace/norm")
for j in range(6):
s = slice(j*64, (j+1)*64)
q = W[f"W_q.{l}"][s].astype(float)
k = W[f"W_k.{l}"][s].astype(float)
v = W[f"W_v.{l}"][s].astype(float)
o = W[f"W_o.{l}"][:, s].astype(float)
g = W[f"ln1_w.{l}"]
A = (q*g).T @ (k*g) / 8
B = o @ (v*g)
sa = np.linalg.svd(A, compute_uv=False)[:64]
sb = np.linalg.svd(B, compute_uv=False)[:64]
print("L%dH%d" % (l,j),
np.round([np.linalg.norm(z) for z in [q,k,v,o]], 3),
"|", np.round(sa[:6],3), "E1=%.3f sym=%.3f tr=%.3f" %
(sa[0]**2/(sa@sa), np.sum(A*A.T)/np.sum(A*A), np.trace(A)/np.linalg.norm(A)),
"|", np.round(sb[:4],3), "tr=%.3f" % (np.trace(B)/np.linalg.norm(B)))
```
{
"t": 1792713176.6073024,
"s": 2.32,
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\ndef summary(a):\n return \"norm=%.3f mean=%.5f std=%.5f min=%.4f max=%.4f\" % (\n np.linalg.norm(a), a.mean(), a.std(), a.min(), a.max())\n\nfor l in range(4):\n print(\"\\nLAYER\", l)\n for name in [\"ln1_w\", \"ln1_b\", \"ln2_w\", \"ln2_b\", \"W_q\", \"W_k\", \"W_v\", \"W_o\", \"W_fc\", \"W_proj\"]:\n print(name, summary(W[f\"{name}.{l}\"]))\n print(\"head: Qnorm Knorm Vnorm Onorm | QK sv(top6), top1_energy, symmetry, trace/norm | OV sv(top4), trace/norm\")\n for j in range(6):\n s = slice(j*64, (j+1)*64)\n q = W[f\"W_q.{l}\"][s].astype(float)\n k = W[f\"W_k.{l}\"][s].astype(float)\n v = W[f\"W_v.{l}\"][s].astype(float)\n o = W[f\"W_o.{l}\"][:, s].astype(float)\n g = W[f\"ln1_w.{l}\"]\n A = (q*g).T @ (k*g) / 8\n B = o @ (v*g)\n sa = np.linalg.svd(A, compute_uv=False)[:64]\n sb = np.linalg.svd(B, compute_uv=False)[:64]\n print(\"L%dH%d\" % (l,j),\n np.round([np.linalg.norm(z) for z in [q,k,v,o]], 3),\n \"|\", np.round(sa[:6],3), \"E1=%.3f sym=%.3f tr=%.3f\" %\n (sa[0]**2/(sa@sa), np.sum(A*A.T)/np.sum(A*A), np.trace(A)/np.linalg.norm(A)),\n \"|\", np.round(sb[:4],3), \"tr=%.3f\" % (np.trace(B)/np.linalg.norm(B)))\n",
"out": "\nLAYER 0\nln1_w norm=6.884 mean=0.29650 std=0.18837 min=0.0352 max=0.9785\nln1_b norm=0.610 mean=0.00039 std=0.03113 min=-0.0954 max=0.0906\nln2_w norm=18.323 mean=0.87176 std=0.33816 min=0.0420 max=1.7750\nln2_b norm=1.380 mean=0.01340 std=0.06913 min=-0.6236 max=0.4636\nW_q norm=17.669 mean=0.00005 std=0.04601 min=-0.4639 max=0.4214\nW_k norm=18.516 mean=0.00001 std=0.04822 min=-0.5163 max=0.5037\nW_v norm=14.937 mean=0.00002 std=0.03890 min=-0.3922 max=0.3895\nW_o norm=16.550 mean=-0.00012 std=0.04310 min=-1.1239 max=1.0823\nW_fc norm=55.376 mean=0.00075 std=0.07210 min=-0.8384 max=1.1226\nW_proj norm=50.007 mean=-0.00000 std=0.06511 min=-0.6883 max=1.4636\nhead: Qnorm Knorm Vnorm Onorm | QK sv(top6), top1_energy, symmetry, trace/norm | OV sv(top4), trace/norm\nL0H0 [7.14 7.395 5.916 6.127] | [0.248 0.102 0.047 0.043 0.037 0.036] E1=0.714 sym=0.140 tr=-0.357 | [0.958 0.898 0.818 0.642] tr=-0.067\nL0H1 [4.486 4.927 4.323 4.825] | [0.15 0.042 0.021 0.012 0.01 0.007] E1=0.892 sym=0.161 tr=-0.620 | [3.38 2.023 0.345 0.18 ] tr=0.231\nL0H2 [4.065 4.362 4.017 4.36 ] | [0.102 0.018 0.016 0.012 0.008 0.007] E1=0.908 sym=0.289 tr=-0.814 | [2.643 1.363 0.4 0.176] tr=0.229\nL0H3 [14.141 14.661 8.259 9.445] | [0.244 0.223 0.201 0.182 0.179 0.157] E1=0.122 sym=-0.067 tr=1.923 | [1.127 0.955 0.95 0.89 ] tr=-0.570\nL0H4 [4.205 4.614 3.689 3.657] | [0.08 0.022 0.019 0.013 0.011 0.008] E1=0.808 sym=0.050 tr=-0.519 | [1.454 0.724 0.213 0.141] tr=0.102\nL0H5 [2.632 2.936 8.453 9.565] | [0.032 0.004 0.003 0.003 0.002 0.002] E1=0.901 sym=0.024 tr=-0.254 | [16.282 8.786 2.835 1.117] tr=0.111\n\nLAYER 1\nln1_w norm=10.065 mean=0.50583 std=0.08918 min=0.1970 max=1.0846\nln1_b norm=3.037 mean=0.01103 std=0.15458 min=-1.5676 max=1.3651\nln2_w norm=22.433 mean=1.13147 std=0.17415 min=0.0019 max=1.5196\nln2_b norm=2.815 mean=0.00647 std=0.14352 min=-0.3892 max=0.4170\nW_q norm=38.929 mean=-0.00027 std=0.10138 min=-0.4410 max=0.4642\nW_k norm=34.893 mean=0.00005 std=0.09087 min=-1.0865 max=1.2757\nW_v norm=28.744 mean=0.00003 std=0.07486 min=-0.3539 max=0.3314\nW_o norm=28.479 mean=0.00002 std=0.07416 min=-0.4175 max=0.4550\nW_fc norm=80.379 mean=0.00139 std=0.10465 min=-0.6542 max=0.6450\nW_proj norm=77.384 mean=-0.00008 std=0.10076 min=-0.5072 max=2.6601\nhead: Qnorm Knorm Vnorm Onorm | QK sv(top6), top1_energy, symmetry, trace/norm | OV sv(top4), trace/norm\nL1H0 [16.628 14.95 11.763 11.711] | [1.185 0.729 0.557 0.534 0.472 0.432] E1=0.358 sym=0.042 tr=-0.883 | [3.189 2.991 2.858 2.707] tr=-1.346\nL1H1 [16.019 14.34 11.237 11.401] | [1.155 0.715 0.583 0.465 0.398 0.368] E1=0.391 sym=0.018 tr=-0.845 | [3.148 3.021 2.872 2.793] tr=-1.351\nL1H2 [15.576 14.136 11.739 11.38 ] | [1.176 0.62 0.485 0.436 0.385 0.355] E1=0.435 sym=0.037 tr=-1.059 | [3.55 3.305 3.182 2.996] tr=-1.467\nL1H3 [15.745 13.949 11.801 11.885] | [1.188 0.678 0.532 0.388 0.366 0.329] E1=0.440 sym=0.006 tr=-0.844 | [3.408 3.269 3.119 3.022] tr=-1.481\nL1H4 [15.205 13.642 12.731 12.113] | [1.059 0.505 0.477 0.351 0.334 0.31 ] E1=0.422 sym=0.047 tr=-1.234 | [4.132 3.769 3.609 3.358] tr=-1.429\nL1H5 [16.146 14.418 11.065 11.244] | [1.147 0.719 0.556 0.499 0.433 0.379] E1=0.377 sym=0.017 tr=-0.874 | [3.085 2.949 2.892 2.805] tr=-1.369\n\nLAYER 2\nln1_w norm=13.410 mean=0.68087 std=0.06874 min=0.3175 max=1.0927\nln1_b norm=3.077 mean=0.01305 std=0.15645 min=-1.6589 max=1.3318\nln2_w norm=25.777 mean=1.31041 std=0.11485 min=0.0120 max=1.5078\nln2_b norm=3.413 mean=0.01229 std=0.17372 min=-0.5170 max=0.4489\nW_q norm=38.902 mean=0.00008 std=0.10131 min=-0.4865 max=0.4916\nW_k norm=36.750 mean=0.00001 std=0.09570 min=-1.2235 max=1.1936\nW_v norm=33.183 mean=-0.00003 std=0.08641 min=-0.4009 max=0.3919\nW_o norm=32.435 mean=-0.00000 std=0.08446 min=-0.5312 max=0.5185\nW_fc norm=79.828 mean=0.00039 std=0.10394 min=-0.5244 max=0.5953\nW_proj norm=78.048 mean=-0.00007 std=0.10163 min=-0.5107 max=0.8014\nhead: Qnorm Knorm Vnorm Onorm | QK sv(top6), top1_energy, symmetry, trace/norm | OV sv(top4), trace/norm\nL2H0 [14.489 13.572 13.0"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 1 of at most 100: ``` LAYER 0 ln1_w norm=6.884 mean=0.29650 std=0.18837 min=0.0352 max=0.9785 ln1_b norm=0.610 mean=0.00039 std=0.03113 min=-0.0954 max=0.0906 ln2_w norm=18.323 mean=0.87176 std=0.33816 min=0.0420 max=1.7750 ln2_b norm=1.380 mean=0.01340 std=0.06913 min=-0.6236 max=0.4636 W_q norm=17.669 mean=0.00005 std=0.04601 min=-0.4639 max=0.4214 W_k norm=18.516 mean=0.00001 std=0.04822 min=-0.5163 max=0.5037 W_v norm=14.937 mean=0.00002 std=0.03890 min=-0.3922 max=0.3895 W_o norm=16.550 mean=-0.00012 std=0.04310 min=-1.1239 max=1.0823 W_fc norm=55.376 mean=0.00075 std=0.07210 min=-0.8384 max=1.1226 W_proj norm=50.007 mean=-0.00000 std=0.06511 min=-0.6883 max=1.4636 head: Qnorm Knorm Vnorm Onorm | QK sv(top6), top1_energy, symmetry, trace/norm | OV sv(top4), trace/norm L0H0 [7.14 7.395 5.916 6.127] | [0.248 0.102 0.047 0.043 0.037 0.036] E1=0.714 sym=0.140 tr=-0.357 | [0.958 0.898 0.818 0.642] tr=-0.067 L0H1 [4.486 4.927 4.323 4.825] | [0.15 0.042 0.021 0.012 0.01 0.007] E1=0.892 sym=0.161 tr=-0.620 | [3.38 2.023 0.345 0.18 ] tr=0.231 L0H2 [4.065 4.362 4.017 4.36 ] | [0.102 0.018 0.016 0.012 0.008 0.007] E1=0.908 sym=0.289 tr=-0.814 | [2.643 1.363 0.4 0.176] tr=0.229 L0H3 [14.141 14.661 8.259 9.445] | [0.244 0.223 0.201 0.182 0.179 0.157] E1=0.122 sym=-0.067 tr=1.923 | [1.127 0.955 0.95 0.89 ] tr=-0.570 L0H4 [4.205 4.614 3.689 3.657] | [0.08 0.022 0.019 0.013 0.011 0.008] E1=0.808 sym=0.050 tr=-0.519 | [1.454 0.724 0.213 0.141] tr=0.102 L0H5 [2.632 2.936 8.453 9.565] | [0.032 0.004 0.003 0.003 0.002 0.002] E1=0.901 sym=0.024 tr=-0.254 | [16.282 8.786 2.835 1.117] tr=0.111 LAYER 1 ln1_w norm=10.065 mean=0.50583 std=0.08918 min=0.1970 max=1.0846 ln1_b norm=3.037 mean=0.01103 std=0.15458 min=-1.5676 max=1.3651 ln2_w norm=22.433 mean=1.13147 std=0.17415 min=0.0019 max=1.5196 ln2_b norm=2.815 mean=0.00647 std=0.14352 min=-0.3892 max=0.4170 W_q norm=38.929 mean=-0.00027 std=0.10138 min=-0.4410 max=0.4642 W_k norm=34.893 mean=0.00005 std=0.09087 min=-1.0865 max=1.2757 W_v norm=28.744 mean=0.00003 std=0.07486 min=-0.3539 max=0.3314 W_o norm=28.479 mean=0.00002 std=0.07416 min=-0.4175 max=0.4550 W_fc norm=80.379 mean=0.00139 std=0.10465 min=-0.6542 max=0.6450 W_proj norm=77.384 mean=-0.00008 std=0.10076 min=-0.5072 max=2.6601 head: Qnorm Knorm Vnorm Onorm | QK sv(top6), top1_energy, symmetry, trace/norm | OV sv(top4), trace/norm L1H0 [16.628 14.95 11.763 11.711] | [1.185 0.729 0.557 0.534 0.472 0.432] E1=0.358 sym=0.042 tr=-0.883 | [3.189 2.991 2.858 2.707] tr=-1.346 L1H1 [16.019 14.34 11.237 11.401] | [1.155 0.715 0.583 0.465 0.398 0.368] E1=0.391 sym=0.018 tr=-0.845 | [3.148 3.021 2.872 2.793] tr=-1.351 L1H2 [15.576 14.136 11.739 11.38 ] | [1.176 0.62 0.485 0.436 0.385 0.355] E1=0.435 sym=0.037 tr=-1.059 | [3.55 3.305 3.182 2.996] tr=-1.467 L1H3 [15.745 13.949 11.801 11.885] | [1.188 0.678 0.532 0.388 0.366 0.329] E1=0.440 sym=0.006 tr=-0.844 | [3.408 3.269 3.119 3.022] tr=-1.481 L1H4 [15.205 13.642 12.731 12.113] | [1.059 0.505 0.477 0.351 0.334 0.31 ] E1=0.422 sym=0.047 tr=-1.234 | [4.132 3.769 3.609 3.358] tr=-1.429 L1H5 [16.146 14.418 11.065 11.244] | [1.147 0.719 0.556 0.499 0.433 0.379] E1=0.377 sym=0.017 tr=-0.874 | [3.085 2.949 2.892 2.805] tr=-1.369 LAYER 2 ln1_w norm=13.410 mean=0.68087 std=0.06874 min=0.3175 max=1.0927 ln1_b norm=3.077 mean=0.01305 std=0.15645 min=-1.6589 max=1.3318 ln2_w norm=25.777 mean=1.31041 std=0.11485 min=0.0120 max=1.5078 ln2_b norm=3.413 mean=0.01229 std=0.17372 min=-0.5170 max=0.4489 W_q norm=38.902 mean=0.00008 std=0.10131 min=-0.4865 max=0.4916 W_k norm=36.750 mean=0.00001 std=0.09570 min=-1.2235 max=1.1936 W_v norm=33.183 mean=-0.00003 std=0.08641 min=-0.4009 max=0.3919 W_o norm=32.435 mean=-0.00000 std=0.08446 min=-0.5312 max=0.5185 W_fc norm=79.828 mean=0.00039 std=0.10394 min=-0.5244 max=0.5953 W_proj norm=78.048 mean=-0.00007 std=0.10163 min=-0.5107 max=0.8014 head: Qnorm Knorm Vnorm Onorm | QK sv(top6), top1_energy, symmetry, trace/norm | OV sv(top4), trace/norm L2H0 [14.489 13.572 13.047 12.69 ] | [0.694 0.555 0.35 0.328 0.307 0.3 ] E1=0.180 sym=-0.040 tr=-0.954 | [5.29 4.505 4.391 3.716] tr=-1.027 L2H1 [16.803 15.934 13.859 13.478] | [0.578 0.514 0.509 0.466 0.458 0.444] E1=0.072 sym=-0.021 tr=-0.100 | [4.136 3.837 3.792 3.685] tr=-1.995 L2H2 [15.082 14.471 14.057 13.611] | [0.518 0.412 0.4 0.374 0.355 0.333] E1=0.091 sym=-0.003 tr=-0.754 | [5.045 4.937 4.562 4.369] tr=-1.604 L2H3 [14.704 13.851 14.29 13.833] | [0.503 0.385 0.364 0.334 0.333 0.312] E1=0.096 sym=-0.008 tr=-0.367 | [4.833 4.752 4.089 4.05 ] tr=-1.590 L2H4 [16.878 15.77 12.388 12.522] | [0.815 0.63 0.506 0.471 0.448 0.423] E1=0.135 sym=-0.032 tr=-0.443 | [3.533 3.373 3.229 3.158] tr=-2.103 L2H5 [17.106 16.203 13.548 13.262] | [0.615 0.564 0.54 0.513 0.498 0.478] E1=0.074 sym=-0.024 tr=-0.118 | [3.735 3.601 3.518 3.421] tr=-2.058 LAYER 3 ln1_w norm=13.922 mean=0.70875 std=0.04941 min=0.4078 max=0.8155 ln1_b norm=2.032 mean=0.02963 std=0.09939 min=-0.7639 max=0.8241 ln2_w norm=24.886 mean=1.26681 std=0.08945 min=0.2328 max=1.4532 ln2_b norm=2.780 mean=0.01280 std=0.14128 min=-0.5402 max=0.3932 W_q norm=33.239 mean=-0.00008 std=0.08656 min=-0.3908 max=0.4595 W_k norm=32.463 mean=-0.00003 std=0.08454 min=-0.9139 max=0.9735 W_v norm=39.422 mean=0.00004 std=0.10266 min=-0.5080 max=0.5939 W_o norm=40.133 mean=0.00001 std=0.10451 min=-1.5167 max=1.5269 W_fc norm=78.259 mean=0.00025 std=0.10190 min=-0.4902 max=0.5500 W_proj norm=85.119 mean=0.00028 std=0.11083 min=-1.0584 max=1.4766 head: Qnorm Knorm Vnorm Onorm | QK sv(top6), top1_energy, symmetry, trace/norm | OV sv(top4), trace/norm L3H0 [15.992 15.604 14.464 14.539] | [0.436 0.414 0.39 0.38 0.373 0.36 ] E1=0.046 sym=0.006 tr=1.073 | [5.785 5.657 5.063 4.643] tr=-2.223 L3H1 [12.344 11.933 15.745 16.085] | [0.346 0.31 0.285 0.253 0.244 0.227] E1=0.081 sym=-0.002 tr=0.353 | [8.23 7.821 6.562 6.475] tr=-1.586 L3H2 [13.501 13.195 15.205 15.365] | [0.385 0.304 0.294 0.277 0.266 0.259] E1=0.072 sym=-0.010 tr=0.242 | [7.46 6.112 5.569 5.336] tr=-1.593 L3H3 [15.138 15.015 17.944 18.53 ] | [0.322 0.311 0.307 0.304 0.298 0.295] E1=0.031 sym=0.209 tr=3.200 | [10.81 9.984 9.821 9.427] tr=-1.210 L3H4 [ 8.612 8.319 17.848 18.3 ] | [0.236 0.203 0.171 0.151 0.133 0.127] E1=0.143 sym=0.001 tr=0.001 | [21.271 17.68 16.977 15.047] tr=-0.794 L3H5 [14.529 14.115 15.008 15.041] | [0.455 0.321 0.315 0.299 0.289 0.288] E1=0.074 sym=0.009 tr=0.863 | [6.302 5.977 5.474 5.086] tr=-1.534 ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(4)
for l in range(4):
print("\nLAYER",l)
for name in ["W_q","W_k","W_v","W_o","W_fc","W_proj"]:
a=W[f"{name}.{l}"].astype(float)
sv=np.linalg.svd(a,compute_uv=False)
print(name,"sv12",np.round(sv[:12],3),
"energy ranks 1,5,10,28,64",np.round([np.sum(sv[:r]**2)/np.sum(sv**2) for r in [1,5,10,28,64]],3))
g=W[f"ln1_w.{l}"].astype(float)
b=W[f"ln1_b.{l}"].astype(float)
for j in range(6):
s=slice(64*j,64*(j+1))
q=W[f"W_q.{l}"][s].astype(float)
k=W[f"W_k.{l}"][s].astype(float)
v=W[f"W_v.{l}"][s].astype(float)
o=W[f"W_o.{l}"][:,s].astype(float)
e=np.linalg.eigvals((k*g)@(q*g).T/8)
e=e[np.argsort(-abs(e))]
A=(q*g).T@(k*g)/8
keybias=(q@b)@(k*g)/8
u,sv,vt=np.linalg.svd(A,full_matrices=False)
print("L%dH%d"%(l,j),"QK eig12",np.round(e[:12],3),
"keybias_norm=%.3f"%np.linalg.norm(keybias),
"keybias_topright=%.3f"%(keybias@vt[0]/(np.linalg.norm(keybias)+1e-12)))
print("LN1 largest bias coords", sorted(enumerate(b),key=lambda z:-abs(z[1]))[:8])
print("\nFIRST-LAYER HEAD QK AND OV COSINES")
As=[];Bs=[]
for j in range(6):
s=slice(64*j,64*(j+1)); g=W["ln1_w.0"]
As.append((W["W_q.0"][s]*g).T@(W["W_k.0"][s]*g)/8)
Bs.append(W["W_o.0"][:,s]@(W["W_v.0"][s]*g))
for label,mat in [("QK",As),("OV",Bs)]:
print(label)
print(np.round([[np.sum(a*b)/(np.linalg.norm(a)*np.linalg.norm(b)) for b in mat] for a in mat],3))
```
{
"t": 1792713220.1863315,
"s": 2.53,
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\nfor l in range(4):\n print(\"\\nLAYER\",l)\n for name in [\"W_q\",\"W_k\",\"W_v\",\"W_o\",\"W_fc\",\"W_proj\"]:\n a=W[f\"{name}.{l}\"].astype(float)\n sv=np.linalg.svd(a,compute_uv=False)\n print(name,\"sv12\",np.round(sv[:12],3),\n \"energy ranks 1,5,10,28,64\",np.round([np.sum(sv[:r]**2)/np.sum(sv**2) for r in [1,5,10,28,64]],3))\n g=W[f\"ln1_w.{l}\"].astype(float)\n b=W[f\"ln1_b.{l}\"].astype(float)\n for j in range(6):\n s=slice(64*j,64*(j+1))\n q=W[f\"W_q.{l}\"][s].astype(float)\n k=W[f\"W_k.{l}\"][s].astype(float)\n v=W[f\"W_v.{l}\"][s].astype(float)\n o=W[f\"W_o.{l}\"][:,s].astype(float)\n e=np.linalg.eigvals((k*g)@(q*g).T/8)\n e=e[np.argsort(-abs(e))]\n A=(q*g).T@(k*g)/8\n keybias=(q@b)@(k*g)/8\n u,sv,vt=np.linalg.svd(A,full_matrices=False)\n print(\"L%dH%d\"%(l,j),\"QK eig12\",np.round(e[:12],3),\n \"keybias_norm=%.3f\"%np.linalg.norm(keybias),\n \"keybias_topright=%.3f\"%(keybias@vt[0]/(np.linalg.norm(keybias)+1e-12)))\n print(\"LN1 largest bias coords\", sorted(enumerate(b),key=lambda z:-abs(z[1]))[:8])\n\nprint(\"\\nFIRST-LAYER HEAD QK AND OV COSINES\")\nAs=[];Bs=[]\nfor j in range(6):\n s=slice(64*j,64*(j+1)); g=W[\"ln1_w.0\"]\n As.append((W[\"W_q.0\"][s]*g).T@(W[\"W_k.0\"][s]*g)/8)\n Bs.append(W[\"W_o.0\"][:,s]@(W[\"W_v.0\"][s]*g))\nfor label,mat in [(\"QK\",As),(\"OV\",Bs)]:\n print(label)\n print(np.round([[np.sum(a*b)/(np.linalg.norm(a)*np.linalg.norm(b)) for b in mat] for a in mat],3))\n",
"out": "\nLAYER 0\nW_q sv12 [4.368 3.131 3.099 3.042 2.994 2.951 2.926 2.881 2.859 2.857 2.804 2.766] energy ranks 1,5,10,28,64 [0.061 0.182 0.316 0.676 0.875]\nW_k sv12 [4.163 3.952 3.467 3.427 3.364 3.335 3.279 3.256 3.214 3.163 3.115 3.029] energy ranks 1,5,10,28,64 [0.051 0.198 0.352 0.713 0.88 ]\nW_v sv12 [5.208 3.364 3.006 2.962 2.815 2.639 2.527 2.46 2.37 2.301 2.25 2.23 ] energy ranks 1,5,10,28,64 [0.122 0.288 0.424 0.715 0.856]\nW_o sv12 [7.361 6.099 3.167 2.452 2.321 2.202 2.171 2.16 2.105 2.072 2.021 1.995] energy ranks 1,5,10,28,64 [0.198 0.412 0.496 0.718 0.91 ]\nW_fc sv12 [18.118 12.76 11.337 9.323 9.267 9.021 8.898 8.817 8.609 8.4 8.189 8.06 ] energy ranks 1,5,10,28,64 [0.107 0.258 0.383 0.676 0.892]\nW_proj sv12 [10.899 7.702 7.173 6.638 6.541 6.288 6.201 6.133 5.941 5.917 5.758 5.641] energy ranks 1,5,10,28,64 [0.048 0.127 0.201 0.394 0.633]\nL0H0 QK eig12 [ 0.079+0.j 0.049+0.j -0.024-0.006j -0.024+0.006j -0.023+0.j 0.018+0.013j 0.018-0.013j -0.02 +0.j -0.014+0.012j -0.014-0.012j\n -0.014+0.003j -0.014-0.003j] keybias_norm=0.043 keybias_topright=-0.694\nL0H1 QK eig12 [-0.062+0.j -0.01 +0.j -0.009+0.j 0.004+0.j -0.004+0.j -0.003+0.002j -0.003-0.002j -0.002+0.j -0.002-0.j -0.002+0.001j\n -0.002-0.001j -0.001+0.j ] keybias_norm=0.013 keybias_topright=-0.607\nL0H2 QK eig12 [-0.057+0.j -0.007+0.003j -0.007-0.003j -0.003+0.001j -0.003-0.001j -0.003+0.j -0.001-0.j -0.001+0.j -0.001-0.001j -0.001+0.001j\n 0.001+0.001j 0.001-0.001j] keybias_norm=0.007 keybias_topright=-0.876\nL0H3 QK eig12 [0.137+0.133j 0.137-0.133j 0.11 -0.089j 0.11 +0.089j 0.095-0.098j 0.095+0.098j 0.084-0.104j 0.084+0.104j 0.064-0.114j 0.064+0.114j 0.096+0.068j\n 0.096-0.068j] keybias_norm=0.043 keybias_topright=0.152\nL0H4 QK eig12 [-0.015+0.j -0.01 +0.j -0.008+0.j -0.003+0.j -0.003-0.j -0.002+0.001j -0.002-0.001j -0. +0.002j -0. -0.002j -0.001+0.001j\n -0.001-0.001j -0.002+0.j ] keybias_norm=0.017 keybias_topright=-0.942\nL0H5 QK eig12 [-0.005+0.j 0.001+0.j -0.001+0.j -0. +0.001j -0. -0.001j -0.001+0.j -0.001+0.j -0.001-0.j 0. +0.001j 0. -0.001j\n -0. +0.001j -0. -0.001j] keybias_norm=0.005 keybias_topright=0.971\nLN1 largest bias coords [(312, np.float64(-0.09542398899793625)), (146, np.float64(0.09060516208410263)), (12, np.float64(0.08561607450246811)), (104, np.float64(-0.08411034941673279)), (289, np.float64(0.08255884051322937)), (156, np.float64(-0.0779787078499794)), (303, np.float64(-0.07763753086328506)), (328, np.float64(0.07621065527200699))]\n\nLAYER 1\nW_q sv12 [10.406 8.239 7.475 7.033 6.622 6.395 6.275 6.13 6.065 5.901 5.819 5.716] energy ranks 1,5,10,28,64 [0.071 0.215 0.34 0.646 0.893]\nW_k sv12 [14.477 11.787 6.37 6.054 5.523 5.407 5.268 5.167 5.091 4.924 4.776 4.529] energy ranks 1,5,10,28,64 [0.172 0.375 0.485 0.716 0.905]\nW_v sv12 [4.728 4.547 4.459 4.395 4.361 4.332 4.291 4.276 4.244 4.133 4.079 3.991] energy ranks 1,5,10,28,64 [0.027 0.123 0.232 0.514 0.801]\nW_o sv12 [4.045 3.267 3.188 3.179 3.163 3.101 3.071 3.046 3.046 3.036 3.009 2.985] energy ranks 1,5,10,28,64 [0.02 0.071 0.128 0.312 0.596]\nW_fc sv12 [19.829 11.575 11.522 11.262 11.059 10.875 10.632 10.497 10.363 10.3 10.174 10.05 ] energy ranks 1,5,10,28,64 [0.061 0.141 0.227 0.457 0.706]\nW_proj sv12 [9.038 8.457 8.103 7.333 7.318 7.088 6.772 6.666 6.566 6.429 6.372 6.278] energy ranks 1,5,10,28,64 [0.014 0.054 0.092 0.199 0.367]\nL1H0 QK eig12 [-0.264+0.j 0.115+0.23j 0.115-0.23j -0.221+0.j -0.199+0.05j -0.199-0.05j -0.1 +0.121j -0.1 -0.121j -0.143+0.j -0.121+0.04j\n -0.121-0.04j -0.039+0.1j ] keybias_norm=3.975 keybias_topright=0.986\nL1H1 QK eig12 [ 0.093+0.245j 0.093-0.245j -0.202+0.j -0.164+0.018j -0.164-0.018j -0.145+0.j -0.119-0.07j -0.119+0.07j -0.056-0.108j -0.056+0.108j\n -0.1 +0.033j -0.1 -0.033j] keybias_norm=3.689 keybias_topright=-0.975\nL1H2 QK eig12 [-0.249+0.j -0.201+0.j 0.069-0.182j 0.069+0.1"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 2 of at most 100: ``` LAYER 0 W_q sv12 [4.368 3.131 3.099 3.042 2.994 2.951 2.926 2.881 2.859 2.857 2.804 2.766] energy ranks 1,5,10,28,64 [0.061 0.182 0.316 0.676 0.875] W_k sv12 [4.163 3.952 3.467 3.427 3.364 3.335 3.279 3.256 3.214 3.163 3.115 3.029] energy ranks 1,5,10,28,64 [0.051 0.198 0.352 0.713 0.88 ] W_v sv12 [5.208 3.364 3.006 2.962 2.815 2.639 2.527 2.46 2.37 2.301 2.25 2.23 ] energy ranks 1,5,10,28,64 [0.122 0.288 0.424 0.715 0.856] W_o sv12 [7.361 6.099 3.167 2.452 2.321 2.202 2.171 2.16 2.105 2.072 2.021 1.995] energy ranks 1,5,10,28,64 [0.198 0.412 0.496 0.718 0.91 ] W_fc sv12 [18.118 12.76 11.337 9.323 9.267 9.021 8.898 8.817 8.609 8.4 8.189 8.06 ] energy ranks 1,5,10,28,64 [0.107 0.258 0.383 0.676 0.892] W_proj sv12 [10.899 7.702 7.173 6.638 6.541 6.288 6.201 6.133 5.941 5.917 5.758 5.641] energy ranks 1,5,10,28,64 [0.048 0.127 0.201 0.394 0.633] L0H0 QK eig12 [ 0.079+0.j 0.049+0.j -0.024-0.006j -0.024+0.006j -0.023+0.j 0.018+0.013j 0.018-0.013j -0.02 +0.j -0.014+0.012j -0.014-0.012j -0.014+0.003j -0.014-0.003j] keybias_norm=0.043 keybias_topright=-0.694 L0H1 QK eig12 [-0.062+0.j -0.01 +0.j -0.009+0.j 0.004+0.j -0.004+0.j -0.003+0.002j -0.003-0.002j -0.002+0.j -0.002-0.j -0.002+0.001j -0.002-0.001j -0.001+0.j ] keybias_norm=0.013 keybias_topright=-0.607 L0H2 QK eig12 [-0.057+0.j -0.007+0.003j -0.007-0.003j -0.003+0.001j -0.003-0.001j -0.003+0.j -0.001-0.j -0.001+0.j -0.001-0.001j -0.001+0.001j 0.001+0.001j 0.001-0.001j] keybias_norm=0.007 keybias_topright=-0.876 L0H3 QK eig12 [0.137+0.133j 0.137-0.133j 0.11 -0.089j 0.11 +0.089j 0.095-0.098j 0.095+0.098j 0.084-0.104j 0.084+0.104j 0.064-0.114j 0.064+0.114j 0.096+0.068j 0.096-0.068j] keybias_norm=0.043 keybias_topright=0.152 L0H4 QK eig12 [-0.015+0.j -0.01 +0.j -0.008+0.j -0.003+0.j -0.003-0.j -0.002+0.001j -0.002-0.001j -0. +0.002j -0. -0.002j -0.001+0.001j -0.001-0.001j -0.002+0.j ] keybias_norm=0.017 keybias_topright=-0.942 L0H5 QK eig12 [-0.005+0.j 0.001+0.j -0.001+0.j -0. +0.001j -0. -0.001j -0.001+0.j -0.001+0.j -0.001-0.j 0. +0.001j 0. -0.001j -0. +0.001j -0. -0.001j] keybias_norm=0.005 keybias_topright=0.971 LN1 largest bias coords [(312, np.float64(-0.09542398899793625)), (146, np.float64(0.09060516208410263)), (12, np.float64(0.08561607450246811)), (104, np.float64(-0.08411034941673279)), (289, np.float64(0.08255884051322937)), (156, np.float64(-0.0779787078499794)), (303, np.float64(-0.07763753086328506)), (328, np.float64(0.07621065527200699))] LAYER 1 W_q sv12 [10.406 8.239 7.475 7.033 6.622 6.395 6.275 6.13 6.065 5.901 5.819 5.716] energy ranks 1,5,10,28,64 [0.071 0.215 0.34 0.646 0.893] W_k sv12 [14.477 11.787 6.37 6.054 5.523 5.407 5.268 5.167 5.091 4.924 4.776 4.529] energy ranks 1,5,10,28,64 [0.172 0.375 0.485 0.716 0.905] W_v sv12 [4.728 4.547 4.459 4.395 4.361 4.332 4.291 4.276 4.244 4.133 4.079 3.991] energy ranks 1,5,10,28,64 [0.027 0.123 0.232 0.514 0.801] W_o sv12 [4.045 3.267 3.188 3.179 3.163 3.101 3.071 3.046 3.046 3.036 3.009 2.985] energy ranks 1,5,10,28,64 [0.02 0.071 0.128 0.312 0.596] W_fc sv12 [19.829 11.575 11.522 11.262 11.059 10.875 10.632 10.497 10.363 10.3 10.174 10.05 ] energy ranks 1,5,10,28,64 [0.061 0.141 0.227 0.457 0.706] W_proj sv12 [9.038 8.457 8.103 7.333 7.318 7.088 6.772 6.666 6.566 6.429 6.372 6.278] energy ranks 1,5,10,28,64 [0.014 0.054 0.092 0.199 0.367] L1H0 QK eig12 [-0.264+0.j 0.115+0.23j 0.115-0.23j -0.221+0.j -0.199+0.05j -0.199-0.05j -0.1 +0.121j -0.1 -0.121j -0.143+0.j -0.121+0.04j -0.121-0.04j -0.039+0.1j ] keybias_norm=3.975 keybias_topright=0.986 L1H1 QK eig12 [ 0.093+0.245j 0.093-0.245j -0.202+0.j -0.164+0.018j -0.164-0.018j -0.145+0.j -0.119-0.07j -0.119+0.07j -0.056-0.108j -0.056+0.108j -0.1 +0.033j -0.1 -0.033j] keybias_norm=3.689 keybias_topright=-0.975 L1H2 QK eig12 [-0.249+0.j -0.201+0.j 0.069-0.182j 0.069+0.182j -0.172+0.j -0.109+0.1j -0.109-0.1j -0.142+0.j 0.007+0.132j 0.007-0.132j -0.122+0.034j -0.122-0.034j] keybias_norm=3.052 keybias_topright=0.974 L1H3 QK eig12 [ 0.083+0.224j 0.083-0.224j -0.198+0.j -0.151+0.j -0.127-0.062j -0.127+0.062j -0.045+0.12j -0.045-0.12j -0.128+0.j -0.099+0.046j -0.099-0.046j -0.001+0.092j] keybias_norm=3.391 keybias_topright=0.977 L1H4 QK eig12 [-0.172+0.j -0.045+0.145j -0.045-0.145j -0.146+0.j -0.132+0.044j -0.132-0.044j -0.135+0.j -0.107+0.057j -0.107-0.057j 0.079-0.091j 0.079+0.091j -0.105+0.022j] keybias_norm=2.329 keybias_topright=-0.915 L1H5 QK eig12 [ 0.101+0.253j 0.101-0.253j -0.26 +0.j -0.145+0.099j -0.145-0.099j -0.156+0.027j -0.156-0.027j -0.082+0.114j -0.082-0.114j -0.124+0.j -0.114+0.j -0.089+0.043j] keybias_norm=3.896 keybias_topright=-0.978 LN1 largest bias coords [(39, np.float64(-1.5675644874572754)), (58, np.float64(1.36513352394104)), (99, np.float64(-1.3355246782302856)), (184, np.float64(-0.5617263317108154)), (383, np.float64(-0.462228000164032)), (21, np.float64(-0.4298906922340393)), (153, np.float64(0.29687735438346863)), (266, np.float64(0.2773352265357971))] LAYER 2 W_q sv12 [5.966 5.019 4.865 4.773 4.695 4.568 4.496 4.433 4.38 4.28 4.224 4.208] energy ranks 1,5,10,28,64 [0.024 0.085 0.15 0.331 0.571] W_k sv12 [9.863 9.533 4.386 4.171 4.117 3.956 3.879 3.777 3.759 3.708 3.607 3.592] energy ranks 1,5,10,28,64 [0.072 0.179 0.233 0.383 0.599] W_v sv12 [3.914 3.78 3.727 3.662 3.582 3.553 3.51 3.464 3.449 3.432 3.396 3.385] energy ranks 1,5,10,28,64 [0.014 0.063 0.118 0.286 0.534] W_o sv12 [4.398 4.037 3.875 3.739 3.621 3.589 3.488 3.467 3.387 3.353 3.322 3.281] energy ranks 1,5,10,28,64 [0.018 0.074 0.131 0.291 0.527] W_fc sv12 [15.734 9.249 8.718 8.694 8.47 7.865 7.739 7.482 7.456 7.36 7.336 7.236] energy ranks 1,5,10,28,64 [0.039 0.087 0.132 0.265 0.455] W_proj sv12 [10.197 9.035 8.882 8.411 8.224 8.088 7.988 7.827 7.679 7.491 7.372 7.26 ] energy ranks 1,5,10,28,64 [0.017 0.066 0.116 0.245 0.419] L2H0 QK eig12 [ 0.081+0.299j 0.081-0.299j -0.104+0.j -0.097+0.024j -0.097-0.024j -0.011-0.089j -0.011+0.089j -0.086+0.013j -0.086-0.013j -0.053+0.066j -0.053-0.066j -0.073+0.036j] keybias_norm=2.021 keybias_topright=-0.974 L2H1 QK eig12 [ 0.079+0.268j 0.079-0.268j -0.13 +0.j -0.067+0.105j -0.067-0.105j -0.115+0.j 0.114+0.j 0.024+0.099j 0.024-0.099j -0.087-0.046j -0.087+0.046j 0.06 +0.077j] keybias_norm=1.881 keybias_topright=0.880 L2H2 QK eig12 [ 0.075+0.208j 0.075-0.208j -0.112+0.007j -0.112-0.007j -0.075+0.068j -0.075-0.068j -0.079-0.048j -0.079+0.048j -0.085+0.031j -0.085-0.031j -0.081+0.015j -0.081-0.015j] keybias_norm=1.609 keybias_topright=-0.974 L2H3 QK eig12 [ 0.036+0.116j 0.036-0.116j -0.055+0.076j -0.055-0.076j -0.025+0.082j -0.025-0.082j -0.073-0.031j -0.073+0.031j -0.026-0.067j -0.026+0.067j 0.037-0.061j 0.037+0.061j] keybias_norm=1.060 keybias_topright=-0.742 L2H4 QK eig12 [ 0.077+0.324j 0.077-0.324j -0.128-0.023j -0.128+0.023j 0.029+0.125j 0.029-0.125j -0.077+0.097j -0.077-0.097j 0.122+0.j 0.067+0.083j 0.067-0.083j -0.093+0.044j] keybias_norm=2.307 keybias_topright=0.946 L2H5 QK eig12 [ 0.076+0.275j 0.076-0.275j -0.028+0.139j -0.028-0.139j 0.125+0.022j 0.125-0.022j -0.101+0.049j -0.101-0.049j -0.109-0.004j -0.109+0.004j -0.08 -0.073j -0.08 +0.073j] keybias_norm=2.024 keybias_topright=-0.926 LN1 largest bias coords [(99, np.float64(-1.6588826179504395)), (58, np.float64(1.331817865371704)), (39, np.float64(-1.1991313695907593)), (383, np.float64(-0.5070002675056458)), (184, np.float64(-0.5046411752700806)), (21, np.float64(-0.4534468650817871)), (63, np.float64(-0.32454749941825867)), (266, np.float64(0.2990613281726837))] LAYER 3 W_q sv12 [4.086 3.933 3.809 3.715 3.68 3.633 3.571 3.556 3.531 3.493 3.464 3.449] energy ranks 1,5,10,28,64 [0.015 0.067 0.124 0.295 0.537] W_k sv12 [6.362 5.842 3.696 3.459 3.363 3.304 3.256 3.244 3.214 3.193 3.177 3.163] energy ranks 1,5,10,28,64 [0.038 0.106 0.156 0.308 0.535] W_v sv12 [5.66 5.424 5.182 5.159 5.074 5.008 4.829 4.685 4.551 4.473 4.437 4.367] energy ranks 1,5,10,28,64 [0.021 0.091 0.162 0.351 0.596] W_o sv12 [10.477 7.357 5.323 5.294 5.177 5.124 5.049 4.959 4.775 4.751 4.625 4.565] energy ranks 1,5,10,28,64 [0.068 0.153 0.229 0.419 0.655] W_fc sv12 [15.114 8.611 8.486 8.073 7.769 7.529 7.449 7.334 7.094 6.971 6.891 6.808] energy ranks 1,5,10,28,64 [0.037 0.082 0.125 0.238 0.403] W_proj sv12 [28.461 22.214 21.34 20.708 19.943 19.662 18.986 18.744 18.295 17.457 17.132 16.561] energy ranks 1,5,10,28,64 [0.112 0.357 0.597 0.938 0.966] L3H0 QK eig12 [0.027+0.179j 0.027-0.179j 0.11 -0.076j 0.11 +0.076j 0.129-0.006j 0.129+0.006j 0.116-0.048j 0.116+0.048j 0.069-0.095j 0.069+0.095j 0.096+0.067j 0.096-0.067j] keybias_norm=0.501 keybias_topright=0.702 L3H1 QK eig12 [ 0.037+0.098j 0.037-0.098j 0.078+0.j 0.006+0.077j 0.006-0.077j 0.067+0.j -0.06 -0.013j -0.06 +0.013j 0.042-0.042j 0.042+0.042j 0.059+0.j 0.052+0.023j] keybias_norm=0.280 keybias_topright=0.428 L3H2 QK eig12 [ 0.036+0.157j 0.036-0.157j 0.081+0.j 0.042+0.065j 0.042-0.065j 0.071-0.018j 0.071+0.018j -0.071+0.j -0.066-0.007j -0.066+0.007j -0.03 +0.059j -0.03 -0.059j] keybias_norm=0.495 keybias_topright=-0.881 L3H3 QK eig12 [0.188+0.006j 0.188-0.006j 0.18 +0.003j 0.18 -0.003j 0.169+0.037j 0.169-0.037j 0.168+0.009j 0.168-0.009j 0.162+0.04j 0.162-0.04j 0.159-0.01j 0.159+0.01j ] keybias_norm=0.221 keybias_topright=-0.205 L3H4 QK eig12 [ 0.031+0.026j 0.031-0.026j 0.002+0.039j 0.002-0.039j 0.03 +0.008j 0.03 -0.008j -0.029+0.011j -0.029-0.011j 0.013-0.024j 0.013+0.024j -0.005-0.026j -0.005+0.026j] keybias_norm=0.144 keybias_topright=0.225 L3H5 QK eig12 [0.021+0.144j 0.021-0.144j 0.103+0.j 0.093+0.027j 0.093-0.027j 0.094+0.005j 0.094-0.005j 0.067+0.063j 0.067-0.063j 0.066+0.057j 0.066-0.057j 0.048+0.063j] keybias_norm=0.493 keybias_topright=-0.861 LN1 largest bias coords [(58, np.float64(0.8241215348243713)), (99, np.float64(-0.7638827562332153)), (5, np.float64(0.43600863218307495)), (383, np.float64(-0.39739054441452026)), (266, np.float64(0.3425202965736389)), (21, np.float64(-0.31522801518440247)), (184, np.float64(-0.2888939678668976)), (45, np.float64(0.2887653410434723))] FIRST-LAYER HEAD QK AND OV COSINES QK [[ 1. 0.012 0.007 0.137 0.013 0.044] [ 0.012 1. 0.69 -0.026 0.57 0.026] [ 0.007 0.69 1. -0.063 0.492 -0.075] [ 0.137 -0.026 -0.063 1. -0.044 0.01 ] [ 0.013 0.57 0.492 -0.044 1. 0.033] [ 0.044 0.026 -0.075 0.01 0.033 1. ]] OV [[ 1. 0.018 0.017 -0.035 0.026 0.018] [ 0.018 1. 0.925 0.006 0.868 0.643] [ 0.017 0.925 1. 0.009 0.81 0.692] [-0.035 0.006 0.009 1. 0.011 -0.001] [ 0.026 0.868 0.81 0.011 1. 0.592] [ 0.018 0.643 0.692 -0.001 0.592 1. ]] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(4)
print("Last MLP output singular values:")
for l in [0,3]:
sv=np.linalg.svd(W[f"W_proj.{l}"].astype(float),compute_uv=False)
print("L",l,np.round(sv[:48],4))
print("\nL0 head singular-vector coordinate structure")
As=[];Bs=[]
g=W["ln1_w.0"].astype(float)
for j in range(6):
s=slice(64*j,64*(j+1))
q=W["W_q.0"][s].astype(float); k=W["W_k.0"][s].astype(float)
A=(q*g).T@(k*g)/8
B=W["W_o.0"][:,s].astype(float)@(W["W_v.0"][s].astype(float)*g)
As.append(A); Bs.append(B)
ua,sa,va=np.linalg.svd(A,full_matrices=False)
ub,sb,vb=np.linalg.svd(B,full_matrices=False)
print("L0H%d"%j)
for label,z in [("QK left1",ua[:,0]),("QK right1",va[0]),("OV left1",ub[:,0]),("OV right1",vb[0]),("OV left2",ub[:,1])]:
ix=np.argsort(-abs(z))[:8]
print(label,list(zip(ix.tolist(),np.round(z[ix],3).tolist())))
print("unscaled QK sv",np.round(np.linalg.svd(q.T@k/8,compute_uv=False)[:12],3))
print("\nKey-bias similarities within layers (weight-only affine coefficients)")
for l in range(1,4):
g=W[f"ln1_w.{l}"].astype(float); b=W[f"ln1_b.{l}"].astype(float)
R=[]
for j in range(6):
s=slice(j*64,(j+1)*64)
r=(W[f"W_q.{l}"][s]@b)@(W[f"W_k.{l}"][s]*g)/8
R.append(r/np.linalg.norm(r))
print("L",l,np.round(np.array(R)@np.array(R).T,2))
for j,r in enumerate(R):
ix=np.argsort(-abs(r))[:6]
print("H",j,"top coords",list(zip(ix.tolist(),np.round(r[ix],3).tolist())),
"OV0 output align",np.round([r@np.linalg.svd(B,full_matrices=False)[0][:,0] for B in Bs],3))
```
{
"t": 1792713273.9907143,
"s": 5.56,
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\nprint(\"Last MLP output singular values:\")\nfor l in [0,3]:\n sv=np.linalg.svd(W[f\"W_proj.{l}\"].astype(float),compute_uv=False)\n print(\"L\",l,np.round(sv[:48],4))\n\nprint(\"\\nL0 head singular-vector coordinate structure\")\nAs=[];Bs=[]\ng=W[\"ln1_w.0\"].astype(float)\nfor j in range(6):\n s=slice(64*j,64*(j+1))\n q=W[\"W_q.0\"][s].astype(float); k=W[\"W_k.0\"][s].astype(float)\n A=(q*g).T@(k*g)/8\n B=W[\"W_o.0\"][:,s].astype(float)@(W[\"W_v.0\"][s].astype(float)*g)\n As.append(A); Bs.append(B)\n ua,sa,va=np.linalg.svd(A,full_matrices=False)\n ub,sb,vb=np.linalg.svd(B,full_matrices=False)\n print(\"L0H%d\"%j)\n for label,z in [(\"QK left1\",ua[:,0]),(\"QK right1\",va[0]),(\"OV left1\",ub[:,0]),(\"OV right1\",vb[0]),(\"OV left2\",ub[:,1])]:\n ix=np.argsort(-abs(z))[:8]\n print(label,list(zip(ix.tolist(),np.round(z[ix],3).tolist())))\n print(\"unscaled QK sv\",np.round(np.linalg.svd(q.T@k/8,compute_uv=False)[:12],3))\n\nprint(\"\\nKey-bias similarities within layers (weight-only affine coefficients)\")\nfor l in range(1,4):\n g=W[f\"ln1_w.{l}\"].astype(float); b=W[f\"ln1_b.{l}\"].astype(float)\n R=[]\n for j in range(6):\n s=slice(j*64,(j+1)*64)\n r=(W[f\"W_q.{l}\"][s]@b)@(W[f\"W_k.{l}\"][s]*g)/8\n R.append(r/np.linalg.norm(r))\n print(\"L\",l,np.round(np.array(R)@np.array(R).T,2))\n for j,r in enumerate(R):\n ix=np.argsort(-abs(r))[:6]\n print(\"H\",j,\"top coords\",list(zip(ix.tolist(),np.round(r[ix],3).tolist())),\n \"OV0 output align\",np.round([r@np.linalg.svd(B,full_matrices=False)[0][:,0] for B in Bs],3))\n",
"out": "Last MLP output singular values:\nL 0 [10.899 7.7021 7.1728 6.6377 6.5415 6.2882 6.2009 6.1328 5.9409 5.9174 5.758 5.6413 5.5516 5.4755 5.3925 5.3625 5.2591 5.2062\n 5.1709 5.153 5.0573 5.0182 4.9601 4.8805 4.8211 4.7883 4.7697 4.7303 4.6429 4.5787 4.5457 4.5291 4.5053 4.4926 4.4404 4.4154\n 4.369 4.3154 4.2935 4.2606 4.2432 4.1849 4.1339 4.1036 4.0637 4.0493 4.0262 3.9879]\nL 3 [28.4613 22.2137 21.3397 20.708 19.9429 19.6618 18.9864 18.744 18.2945 17.4567 17.1321 16.5609 15.3286 14.8739 14.3285 13.4378 12.667 12.4546\n 12.317 11.5309 11.0466 9.0836 8.3531 8.2404 6.3701 5.8357 4.6262 4.5827 3.7775 3.6102 3.2518 3.1826 3.0856 2.9413 2.8655 2.8013\n 2.6391 2.5637 2.538 2.4655 2.4188 2.4037 2.346 2.3102 2.2461 2.227 2.1548 2.1472]\n\nL0 head singular-vector coordinate structure\nL0H0\nQK left1 [(317, -0.307), (184, 0.284), (39, 0.269), (239, -0.251), (180, 0.229), (349, 0.199), (99, 0.199), (228, 0.19)]\nQK right1 [(211, -0.322), (256, 0.26), (65, -0.23), (173, 0.216), (223, 0.213), (132, -0.212), (4, 0.212), (39, 0.209)]\nOV left1 [(336, 0.194), (96, 0.158), (291, -0.15), (327, 0.149), (203, -0.14), (202, -0.128), (330, 0.12), (68, -0.117)]\nOV right1 [(373, 0.174), (225, -0.164), (134, -0.158), (158, -0.155), (7, 0.152), (51, 0.148), (96, -0.142), (173, -0.132)]\nOV left2 [(39, 0.262), (6, 0.192), (282, -0.186), (156, -0.148), (163, -0.136), (152, -0.133), (342, -0.128), (219, 0.127)]\nunscaled QK sv [0.853 0.613 0.368 0.287 0.279 0.244 0.232 0.21 0.202 0.188 0.182 0.173]\nL0H1\nQK left1 [(256, 0.372), (184, -0.3), (354, -0.232), (173, 0.225), (4, 0.223), (56, 0.2), (69, -0.183), (100, 0.173)]\nQK right1 [(100, -0.387), (349, 0.272), (317, -0.252), (235, 0.208), (271, -0.194), (258, 0.181), (69, 0.167), (368, 0.152)]\nOV left1 [(39, 0.692), (58, -0.51), (99, 0.358), (21, -0.212), (184, 0.201), (153, -0.054), (336, -0.038), (63, 0.037)]\nOV right1 [(184, 0.347), (180, 0.315), (99, 0.271), (39, 0.267), (56, -0.254), (317, -0.239), (69, 0.235), (256, -0.214)]\nOV left2 [(21, 0.961), (39, 0.158), (58, -0.129), (184, 0.072), (306, -0.032), (173, -0.029), (225, -0.028), (232, -0.026)]\nunscaled QK sv [0.838 0.311 0.219 0.125 0.091 0.086 0.066 0.039 0.038 0.034 0.033 0.03 ]\nL0H2\nQK left1 [(256, 0.394), (173, 0.255), (211, -0.251), (184, -0.238), (4, 0.222), (228, 0.205), (65, -0.188), (124, 0.171)]\nQK right1 [(100, -0.346), (349, 0.254), (256, -0.201), (317, -0.173), (56, -0.167), (271, -0.154), (175, 0.153), (240, 0.149)]\nOV left1 [(39, -0.659), (58, 0.494), (21, 0.373), (99, -0.355), (184, -0.162), (153, 0.045), (383, -0.035), (165, -0.029)]\nOV right1 [(184, -0.365), (180, -0.304), (99, -0.298), (317, 0.248), (56, 0.234), (39, -0.232), (69, -0.208), (301, -0.195)]\nOV left2 [(21, 0.91), (39, 0.221), (99, 0.207), (58, -0.201), (184, 0.053), (252, -0.03), (351, -0.03), (153, -0.029)]\nunscaled QK sv [0.543 0.183 0.157 0.133 0.072 0.066 0.046 0.046 0.036 0.033 0.032 0.029]\nL0H3\nQK left1 [(354, 0.309), (162, -0.3), (65, -0.252), (223, 0.24), (256, 0.193), (180, 0.192), (300, -0.192), (211, -0.182)]\nQK right1 [(162, -0.563), (184, -0.359), (354, 0.251), (99, -0.25), (65, -0.225), (256, 0.197), (132, -0.149), (223, 0.141)]\nOV left1 [(92, -0.178), (86, 0.169), (188, -0.163), (18, 0.16), (131, 0.158), (303, 0.156), (215, 0.148), (173, 0.139)]\nOV right1 [(223, -0.227), (26, -0.198), (200, -0.172), (48, -0.168), (359, 0.144), (124, 0.141), (178, -0.137), (354, 0.133)]\nOV left2 [(65, -0.191), (125, -0.18), (320, 0.174), (202, 0.168), (86, 0.158), (287, 0.152), (20, -0.142), (350, -0.136)]\nunscaled QK sv [1.114 1.092 1.07 1.064 1.03 0.992 0.968 0.954 0.946 0.9 0.888 0.868]\nL0H4\nQK left1 [(256, 0.43), (184, -0.307), (173, 0.225), (228, 0.22), (56, 0.219), (69, -0.193), (21, 0.18), (180, -0.16)]\nQK right1 [(100, -0.282), (260, 0.188), (258, 0.181), (160, -0.178), (175, 0.168), (271, -0.154), (240, 0.153), (317, -0.151)]\nOV left1 [(39, -0.696), (58, 0.511), (99, -0.32), (184, -0.227"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 3 of at most 100: ``` Last MLP output singular values: L 0 [10.899 7.7021 7.1728 6.6377 6.5415 6.2882 6.2009 6.1328 5.9409 5.9174 5.758 5.6413 5.5516 5.4755 5.3925 5.3625 5.2591 5.2062 5.1709 5.153 5.0573 5.0182 4.9601 4.8805 4.8211 4.7883 4.7697 4.7303 4.6429 4.5787 4.5457 4.5291 4.5053 4.4926 4.4404 4.4154 4.369 4.3154 4.2935 4.2606 4.2432 4.1849 4.1339 4.1036 4.0637 4.0493 4.0262 3.9879] L 3 [28.4613 22.2137 21.3397 20.708 19.9429 19.6618 18.9864 18.744 18.2945 17.4567 17.1321 16.5609 15.3286 14.8739 14.3285 13.4378 12.667 12.4546 12.317 11.5309 11.0466 9.0836 8.3531 8.2404 6.3701 5.8357 4.6262 4.5827 3.7775 3.6102 3.2518 3.1826 3.0856 2.9413 2.8655 2.8013 2.6391 2.5637 2.538 2.4655 2.4188 2.4037 2.346 2.3102 2.2461 2.227 2.1548 2.1472] L0 head singular-vector coordinate structure L0H0 QK left1 [(317, -0.307), (184, 0.284), (39, 0.269), (239, -0.251), (180, 0.229), (349, 0.199), (99, 0.199), (228, 0.19)] QK right1 [(211, -0.322), (256, 0.26), (65, -0.23), (173, 0.216), (223, 0.213), (132, -0.212), (4, 0.212), (39, 0.209)] OV left1 [(336, 0.194), (96, 0.158), (291, -0.15), (327, 0.149), (203, -0.14), (202, -0.128), (330, 0.12), (68, -0.117)] OV right1 [(373, 0.174), (225, -0.164), (134, -0.158), (158, -0.155), (7, 0.152), (51, 0.148), (96, -0.142), (173, -0.132)] OV left2 [(39, 0.262), (6, 0.192), (282, -0.186), (156, -0.148), (163, -0.136), (152, -0.133), (342, -0.128), (219, 0.127)] unscaled QK sv [0.853 0.613 0.368 0.287 0.279 0.244 0.232 0.21 0.202 0.188 0.182 0.173] L0H1 QK left1 [(256, 0.372), (184, -0.3), (354, -0.232), (173, 0.225), (4, 0.223), (56, 0.2), (69, -0.183), (100, 0.173)] QK right1 [(100, -0.387), (349, 0.272), (317, -0.252), (235, 0.208), (271, -0.194), (258, 0.181), (69, 0.167), (368, 0.152)] OV left1 [(39, 0.692), (58, -0.51), (99, 0.358), (21, -0.212), (184, 0.201), (153, -0.054), (336, -0.038), (63, 0.037)] OV right1 [(184, 0.347), (180, 0.315), (99, 0.271), (39, 0.267), (56, -0.254), (317, -0.239), (69, 0.235), (256, -0.214)] OV left2 [(21, 0.961), (39, 0.158), (58, -0.129), (184, 0.072), (306, -0.032), (173, -0.029), (225, -0.028), (232, -0.026)] unscaled QK sv [0.838 0.311 0.219 0.125 0.091 0.086 0.066 0.039 0.038 0.034 0.033 0.03 ] L0H2 QK left1 [(256, 0.394), (173, 0.255), (211, -0.251), (184, -0.238), (4, 0.222), (228, 0.205), (65, -0.188), (124, 0.171)] QK right1 [(100, -0.346), (349, 0.254), (256, -0.201), (317, -0.173), (56, -0.167), (271, -0.154), (175, 0.153), (240, 0.149)] OV left1 [(39, -0.659), (58, 0.494), (21, 0.373), (99, -0.355), (184, -0.162), (153, 0.045), (383, -0.035), (165, -0.029)] OV right1 [(184, -0.365), (180, -0.304), (99, -0.298), (317, 0.248), (56, 0.234), (39, -0.232), (69, -0.208), (301, -0.195)] OV left2 [(21, 0.91), (39, 0.221), (99, 0.207), (58, -0.201), (184, 0.053), (252, -0.03), (351, -0.03), (153, -0.029)] unscaled QK sv [0.543 0.183 0.157 0.133 0.072 0.066 0.046 0.046 0.036 0.033 0.032 0.029] L0H3 QK left1 [(354, 0.309), (162, -0.3), (65, -0.252), (223, 0.24), (256, 0.193), (180, 0.192), (300, -0.192), (211, -0.182)] QK right1 [(162, -0.563), (184, -0.359), (354, 0.251), (99, -0.25), (65, -0.225), (256, 0.197), (132, -0.149), (223, 0.141)] OV left1 [(92, -0.178), (86, 0.169), (188, -0.163), (18, 0.16), (131, 0.158), (303, 0.156), (215, 0.148), (173, 0.139)] OV right1 [(223, -0.227), (26, -0.198), (200, -0.172), (48, -0.168), (359, 0.144), (124, 0.141), (178, -0.137), (354, 0.133)] OV left2 [(65, -0.191), (125, -0.18), (320, 0.174), (202, 0.168), (86, 0.158), (287, 0.152), (20, -0.142), (350, -0.136)] unscaled QK sv [1.114 1.092 1.07 1.064 1.03 0.992 0.968 0.954 0.946 0.9 0.888 0.868] L0H4 QK left1 [(256, 0.43), (184, -0.307), (173, 0.225), (228, 0.22), (56, 0.219), (69, -0.193), (21, 0.18), (180, -0.16)] QK right1 [(100, -0.282), (260, 0.188), (258, 0.181), (160, -0.178), (175, 0.168), (271, -0.154), (240, 0.153), (317, -0.151)] OV left1 [(39, -0.696), (58, 0.511), (99, -0.32), (184, -0.227), (21, -0.182), (153, 0.057), (336, 0.042), (350, 0.039)] OV right1 [(184, -0.414), (180, -0.29), (256, 0.279), (56, 0.243), (100, 0.236), (69, -0.226), (317, 0.207), (162, -0.199)] OV left2 [(21, 0.941), (99, -0.241), (39, -0.097), (58, 0.092), (184, 0.042), (148, -0.033), (52, -0.032), (381, 0.031)] unscaled QK sv [0.51 0.227 0.185 0.124 0.101 0.066 0.065 0.049 0.048 0.045 0.04 0.035] L0H5 QK left1 [(184, 0.347), (39, 0.311), (211, -0.3), (317, -0.256), (239, -0.216), (180, 0.198), (99, 0.196), (65, -0.196)] QK right1 [(240, -0.222), (301, 0.178), (171, 0.162), (225, 0.158), (200, 0.143), (180, -0.143), (67, -0.134), (31, -0.133)] OV left1 [(21, -0.864), (99, -0.358), (58, 0.237), (39, -0.232), (184, 0.024), (153, 0.019), (303, 0.019), (6, 0.017)] OV right1 [(162, -0.329), (211, -0.307), (223, 0.277), (65, -0.236), (354, 0.201), (173, 0.2), (132, -0.196), (300, -0.192)] OV left2 [(39, -0.608), (58, 0.486), (21, 0.469), (99, -0.404), (180, -0.035), (165, -0.026), (153, 0.021), (86, -0.015)] unscaled QK sv [0.18 0.048 0.024 0.02 0.02 0.019 0.019 0.018 0.018 0.017 0.017 0.016] Key-bias similarities within layers (weight-only affine coefficients) L 1 [[1. 0.83 0.85 0.82 0.82 0.86] [0.83 1. 0.85 0.86 0.82 0.86] [0.85 0.85 1. 0.83 0.83 0.83] [0.82 0.86 0.83 1. 0.83 0.87] [0.82 0.82 0.83 0.83 1. 0.83] [0.86 0.86 0.83 0.87 0.83 1. ]] H 0 top coords [(21, 0.912), (39, -0.119), (58, 0.09), (329, -0.053), (304, -0.05), (158, -0.05)] OV0 output align [-0.015 -0.325 0.467 0.006 -0.031 -0.739] H 1 top coords [(21, 0.909), (58, 0.137), (39, -0.116), (57, 0.069), (187, 0.052), (213, -0.051)] OV0 output align [-0.009 -0.338 0.48 0.065 -0.02 -0.727] H 2 top coords [(21, 0.907), (58, 0.131), (39, -0.127), (215, 0.069), (369, -0.064), (76, -0.063)] OV0 output align [ 0.016 -0.344 0.484 0.005 -0.009 -0.734] H 3 top coords [(21, 0.908), (58, 0.135), (39, -0.125), (170, -0.085), (242, 0.062), (329, 0.058)] OV0 output align [-0.005 -0.36 0.5 0.084 -0.002 -0.712] H 4 top coords [(21, 0.862), (58, 0.217), (39, -0.143), (150, 0.077), (144, -0.071), (99, -0.069)] OV0 output align [-0.03 -0.416 0.546 0.044 0.073 -0.636] H 5 top coords [(21, 0.914), (39, -0.125), (58, 0.102), (351, -0.069), (210, -0.064), (200, 0.059)] OV0 output align [-0.007 -0.343 0.485 0.024 -0.02 -0.72 ] L 2 [[1. 0.81 0.77 0.7 0.82 0.84] [0.81 1. 0.81 0.72 0.82 0.84] [0.77 0.81 1. 0.7 0.83 0.83] [0.7 0.72 0.7 1. 0.71 0.73] [0.82 0.82 0.83 0.71 1. 0.84] [0.84 0.84 0.83 0.73 0.84 1. ]] H 0 top coords [(21, 0.898), (39, -0.131), (58, 0.123), (294, -0.077), (351, 0.07), (311, 0.057)] OV0 output align [-0.005 -0.368 0.501 0.042 0.017 -0.707] H 1 top coords [(21, 0.894), (99, -0.111), (58, 0.107), (39, -0.106), (255, -0.063), (240, -0.053)] OV0 output align [-0.004 -0.359 0.497 0.013 0.003 -0.687] H 2 top coords [(21, 0.877), (99, -0.148), (58, 0.097), (39, -0.094), (306, 0.062), (76, -0.059)] OV0 output align [-0.001 -0.361 0.497 0.054 0.013 -0.664] H 3 top coords [(21, 0.758), (58, 0.149), (39, -0.125), (99, -0.111), (363, 0.091), (76, 0.087)] OV0 output align [-0.003 -0.389 0.498 0.029 0.084 -0.553] H 4 top coords [(21, 0.897), (58, 0.139), (39, -0.126), (361, -0.056), (2, 0.055), (335, 0.053)] OV0 output align [ 0.007 -0.353 0.486 0.009 0.007 -0.719] H 5 top coords [(21, 0.919), (58, 0.101), (39, -0.084), (40, 0.051), (214, 0.049), (199, 0.049)] OV0 output align [ 0.007 -0.321 0.464 -0.004 -0.039 -0.74 ] L 3 [[ 1. 0.6 0.75 0.01 0.16 0.78] [ 0.6 1. 0.55 0.02 0.29 0.67] [ 0.75 0.55 1. 0.01 0.21 0.74] [ 0.01 0.02 0.01 1. 0.03 -0.05] [ 0.16 0.29 0.21 0.03 1. 0.25] [ 0.78 0.67 0.74 -0.05 0.25 1. ]] H 0 top coords [(21, 0.815), (39, -0.138), (99, -0.135), (307, 0.082), (58, 0.081), (184, -0.078)] OV0 output align [-0.029 -0.385 0.507 0.086 0.059 -0.605] H 1 top coords [(21, 0.646), (39, -0.209), (184, -0.17), (99, -0.165), (37, 0.109), (266, 0.107)] OV0 output align [ 0.198 -0.428 0.511 0.078 0.168 -0.429] H 2 top coords [(21, 0.772), (99, -0.191), (39, -0.131), (25, -0.089), (136, -0.074), (291, -0.073)] OV0 output align [ 0.004 -0.383 0.499 0.064 0.072 -0.553] H 3 top coords [(193, -0.162), (157, 0.14), (198, -0.133), (339, -0.133), (129, -0.132), (266, 0.129)] OV0 output align [ 0.112 -0.066 0.053 0.065 0.085 0.082] H 4 top coords [(163, -0.218), (21, 0.194), (120, 0.162), (355, -0.149), (269, -0.148), (188, 0.143)] OV0 output align [ 0.264 -0.069 0.098 -0.001 -0.002 -0.169] H 5 top coords [(21, 0.842), (39, -0.13), (184, -0.12), (22, 0.093), (43, 0.086), (99, -0.083)] OV0 output align [ 0.032 -0.358 0.484 0.06 0.026 -0.661] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(4)
As={};Bs={}
for l in range(4):
g=W[f"ln1_w.{l}"].astype(float)
for j in range(6):
s=slice(64*j,64*(j+1))
As[l,j]=(W[f"W_q.{l}"][s]*g).T@(W[f"W_k.{l}"][s]*g)/8
Bs[l,j]=W[f"W_o.{l}"][:,s].astype(float)@(W[f"W_v.{l}"][s]*g)
print("Trace(A_lh B_0j)/||A_lh B_0j||, for j=0..5 (positive matching composition)")
for l in range(1,4):
for h in range(6):
vals=[]
for j in range(6):
C=As[l,h]@Bs[0,j]
vals.append(np.trace(C)/np.linalg.norm(C))
print("L%dH%d"%(l,h),np.round(vals,3))
print("\nStrongest matching compositions over ALL preceding heads")
for l in range(1,4):
for h in range(6):
vals=[]
for p in range(l):
for j in range(6):
C=As[l,h]@Bs[p,j]
vals.append((np.trace(C)/np.linalg.norm(C),p,j))
print("L%dH%d"%(l,h),[(round(x,3),p,j) for x,p,j in sorted(vals,reverse=True)[:4]])
print("\nL0H3 Q,K,V singular spectra")
for name in ["W_q","W_k","W_v"]:
a=W[f"{name}.0"][192:256].astype(float)
print(name,np.round(np.linalg.svd(a,compute_uv=False),3))
print("\nMLP0 input spectrum")
a=W["W_fc.0"].astype(float)*W["ln2_w.0"]
print(np.round(np.linalg.svd(a,compute_uv=False)[:100],3))
print("\nPositional-like QK L0H3 versus value subspaces: squared overlap")
_,_,qv=np.linalg.svd(W["W_q.0"][192:256],full_matrices=False)
_,_,kv=np.linalg.svd(W["W_k.0"][192:256],full_matrices=False)
_,_,vv=np.linalg.svd(W["W_v.0"][192:256],full_matrices=False)
for r in [16,24,28,32,40]:
print(r, "Q/K",np.linalg.norm(qv[:r]@kv[:r].T)**2/r,
"Q/V",np.linalg.norm(qv[:r]@vv[:r].T)**2/r,
"K/V",np.linalg.norm(kv[:r]@vv[:r].T)**2/r)
```
{
"t": 1792713333.7131855,
"s": 1.29,
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\nAs={};Bs={}\nfor l in range(4):\n g=W[f\"ln1_w.{l}\"].astype(float)\n for j in range(6):\n s=slice(64*j,64*(j+1))\n As[l,j]=(W[f\"W_q.{l}\"][s]*g).T@(W[f\"W_k.{l}\"][s]*g)/8\n Bs[l,j]=W[f\"W_o.{l}\"][:,s].astype(float)@(W[f\"W_v.{l}\"][s]*g)\nprint(\"Trace(A_lh B_0j)/||A_lh B_0j||, for j=0..5 (positive matching composition)\")\nfor l in range(1,4):\n for h in range(6):\n vals=[]\n for j in range(6):\n C=As[l,h]@Bs[0,j]\n vals.append(np.trace(C)/np.linalg.norm(C))\n print(\"L%dH%d\"%(l,h),np.round(vals,3))\n\nprint(\"\\nStrongest matching compositions over ALL preceding heads\")\nfor l in range(1,4):\n for h in range(6):\n vals=[]\n for p in range(l):\n for j in range(6):\n C=As[l,h]@Bs[p,j]\n vals.append((np.trace(C)/np.linalg.norm(C),p,j))\n print(\"L%dH%d\"%(l,h),[(round(x,3),p,j) for x,p,j in sorted(vals,reverse=True)[:4]])\n\nprint(\"\\nL0H3 Q,K,V singular spectra\")\nfor name in [\"W_q\",\"W_k\",\"W_v\"]:\n a=W[f\"{name}.0\"][192:256].astype(float)\n print(name,np.round(np.linalg.svd(a,compute_uv=False),3))\nprint(\"\\nMLP0 input spectrum\")\na=W[\"W_fc.0\"].astype(float)*W[\"ln2_w.0\"]\nprint(np.round(np.linalg.svd(a,compute_uv=False)[:100],3))\n\nprint(\"\\nPositional-like QK L0H3 versus value subspaces: squared overlap\")\n_,_,qv=np.linalg.svd(W[\"W_q.0\"][192:256],full_matrices=False)\n_,_,kv=np.linalg.svd(W[\"W_k.0\"][192:256],full_matrices=False)\n_,_,vv=np.linalg.svd(W[\"W_v.0\"][192:256],full_matrices=False)\nfor r in [16,24,28,32,40]:\n print(r, \"Q/K\",np.linalg.norm(qv[:r]@kv[:r].T)**2/r,\n \"Q/V\",np.linalg.norm(qv[:r]@vv[:r].T)**2/r,\n \"K/V\",np.linalg.norm(kv[:r]@vv[:r].T)**2/r)\n",
"out": "Trace(A_lh B_0j)/||A_lh B_0j||, for j=0..5 (positive matching composition)\nL1H0 [0.051 0.094 0.134 0.102 0.096 0.1 ]\nL1H1 [0.018 0.017 0.065 0.216 0.013 0.041]\nL1H2 [-0.06 0.078 0.106 0.167 0.1 0.09 ]\nL1H3 [0.042 0.073 0.113 0.076 0.075 0.082]\nL1H4 [0.02 0.046 0.08 0.015 0.078 0.074]\nL1H5 [-0.043 0.052 0.102 0.198 0.061 0.062]\nL2H0 [-0.009 0.133 0.196 0.079 0.089 0.189]\nL2H1 [0.035 0.116 0.179 0.009 0.08 0.197]\nL2H2 [0.054 0.123 0.182 0.099 0.073 0.184]\nL2H3 [0.064 0.096 0.129 0.082 0.084 0.11 ]\nL2H4 [-0.031 0.128 0.186 0.159 0.109 0.18 ]\nL2H5 [0.018 0.112 0.187 0.01 0.081 0.21 ]\nL3H0 [-0.049 0.069 0.107 -0.177 0.047 0.085]\nL3H1 [ 0.034 0.069 0.093 -0.033 0.029 0.075]\nL3H2 [-0.006 0.046 0.081 0.033 0.002 0.016]\nL3H3 [-0.155 0.069 0.079 0.922 0.079 -0.055]\nL3H4 [0.069 0.172 0.173 0.124 0.19 0.127]\nL3H5 [-0.008 0.016 0.05 -0.062 -0.001 0.047]\n\nStrongest matching compositions over ALL preceding heads\nL1H0 [(np.float64(0.134), 0, 2), (np.float64(0.102), 0, 3), (np.float64(0.1), 0, 5), (np.float64(0.096), 0, 4)]\nL1H1 [(np.float64(0.216), 0, 3), (np.float64(0.065), 0, 2), (np.float64(0.041), 0, 5), (np.float64(0.018), 0, 0)]\nL1H2 [(np.float64(0.167), 0, 3), (np.float64(0.106), 0, 2), (np.float64(0.1), 0, 4), (np.float64(0.09), 0, 5)]\nL1H3 [(np.float64(0.113), 0, 2), (np.float64(0.082), 0, 5), (np.float64(0.076), 0, 3), (np.float64(0.075), 0, 4)]\nL1H4 [(np.float64(0.08), 0, 2), (np.float64(0.078), 0, 4), (np.float64(0.074), 0, 5), (np.float64(0.046), 0, 1)]\nL1H5 [(np.float64(0.198), 0, 3), (np.float64(0.102), 0, 2), (np.float64(0.062), 0, 5), (np.float64(0.061), 0, 4)]\nL2H0 [(np.float64(0.208), 1, 3), (np.float64(0.196), 0, 2), (np.float64(0.189), 0, 5), (np.float64(0.164), 1, 1)]\nL2H1 [(np.float64(0.197), 0, 5), (np.float64(0.179), 0, 2), (np.float64(0.116), 0, 1), (np.float64(0.08), 0, 4)]\nL2H2 [(np.float64(0.184), 0, 5), (np.float64(0.182), 0, 2), (np.float64(0.123), 0, 1), (np.float64(0.099), 0, 3)]\nL2H3 [(np.float64(0.129), 0, 2), (np.float64(0.11), 0, 5), (np.float64(0.096), 0, 1), (np.float64(0.084), 0, 4)]\nL2H4 [(np.float64(0.186), 0, 2), (np.float64(0.18), 0, 5), (np.float64(0.159), 0, 3), (np.float64(0.128), 0, 1)]\nL2H5 [(np.float64(0.21), 0, 5), (np.float64(0.187), 0, 2), (np.float64(0.112), 0, 1), (np.float64(0.081), 0, 4)]\nL3H0 [(np.float64(0.107), 0, 2), (np.float64(0.085), 0, 5), (np.float64(0.069), 0, 1), (np.float64(0.047), 0, 4)]\nL3H1 [(np.float64(0.093), 0, 2), (np.float64(0.075), 0, 5), (np.float64(0.072), 1, 2), (np.float64(0.069), 0, 1)]\nL3H2 [(np.float64(0.081), 0, 2), (np.float64(0.046), 0, 1), (np.float64(0.033), 0, 3), (np.float64(0.016), 0, 5)]\nL3H3 [(np.float64(0.922), 0, 3), (np.float64(0.806), 2, 5), (np.float64(0.562), 2, 1), (np.float64(0.514), 1, 0)]\nL3H4 [(np.float64(0.276), 2, 2), (np.float64(0.272), 2, 4), (np.float64(0.202), 2, 5), (np.float64(0.19), 0, 4)]\nL3H5 [(np.float64(0.05), 0, 2), (np.float64(0.047), 0, 5), (np.float64(0.016), 0, 1), (np.float64(-0.001), 0, 4)]\n\nL0H3 Q,K,V singular spectra\nW_q [2.962 2.894 2.873 2.845 2.806 2.792 2.766 2.756 2.703 2.634 2.588 2.561 2.544 2.512 2.447 2.438 2.426 2.336 2.33 2.259 2.143 2.098 1.968 1.946\n 1.838 1.756 1.721 1.685 1.575 1.54 1.495 1.463 1.423 1.378 1.283 1.136 1.116 1.094 1.06 1.023 0.991 0.969 0.962 0.897 0.864 0.824 0.778 0.77\n 0.731 0.725 0.698 0.688 0.663 0.637 0.614 0.589 0.573 0.556 0.532 0.519 0.497 0.474 0.443 0.352]\nW_k [3.362 3.238 3.209 3.181 3.099 3.086 3.024 2.999 2.951 2.923 2.856 2.765 2.731 2.654 2.622 2.539 2.478 2.407 2.31 2.217 2.134 2.057 1.976 1.806\n 1.76 1.737 1.61 1.6 1.536 1.42 1.296 1.229 1.198 1.172 1.117 1.058 1.016 0.988 0.942 0.878 0.85 0.812 0.785 0.751 0.72 0.711 0.704 0.685\n 0.641 0.61 0.601 0.578 0.562 0.542 0.532 0.526 0.505 0.492 0.484 0.453 0.433 0.421 0.396 0.366]\nW_v [2.102 1.948 1.903 1.866 1.843 1.819 1.769 1.755 1.674 1.651 1.629 1.621 1.589 1.553 1.52 1.465 1.417 1.303 1.16 1.101 0.945 0.93 0.848 0.818\n 0.79 0.779 0.745 0.739 0.73 0.701 "
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 4 of at most 100: ``` Trace(A_lh B_0j)/||A_lh B_0j||, for j=0..5 (positive matching composition) L1H0 [0.051 0.094 0.134 0.102 0.096 0.1 ] L1H1 [0.018 0.017 0.065 0.216 0.013 0.041] L1H2 [-0.06 0.078 0.106 0.167 0.1 0.09 ] L1H3 [0.042 0.073 0.113 0.076 0.075 0.082] L1H4 [0.02 0.046 0.08 0.015 0.078 0.074] L1H5 [-0.043 0.052 0.102 0.198 0.061 0.062] L2H0 [-0.009 0.133 0.196 0.079 0.089 0.189] L2H1 [0.035 0.116 0.179 0.009 0.08 0.197] L2H2 [0.054 0.123 0.182 0.099 0.073 0.184] L2H3 [0.064 0.096 0.129 0.082 0.084 0.11 ] L2H4 [-0.031 0.128 0.186 0.159 0.109 0.18 ] L2H5 [0.018 0.112 0.187 0.01 0.081 0.21 ] L3H0 [-0.049 0.069 0.107 -0.177 0.047 0.085] L3H1 [ 0.034 0.069 0.093 -0.033 0.029 0.075] L3H2 [-0.006 0.046 0.081 0.033 0.002 0.016] L3H3 [-0.155 0.069 0.079 0.922 0.079 -0.055] L3H4 [0.069 0.172 0.173 0.124 0.19 0.127] L3H5 [-0.008 0.016 0.05 -0.062 -0.001 0.047] Strongest matching compositions over ALL preceding heads L1H0 [(np.float64(0.134), 0, 2), (np.float64(0.102), 0, 3), (np.float64(0.1), 0, 5), (np.float64(0.096), 0, 4)] L1H1 [(np.float64(0.216), 0, 3), (np.float64(0.065), 0, 2), (np.float64(0.041), 0, 5), (np.float64(0.018), 0, 0)] L1H2 [(np.float64(0.167), 0, 3), (np.float64(0.106), 0, 2), (np.float64(0.1), 0, 4), (np.float64(0.09), 0, 5)] L1H3 [(np.float64(0.113), 0, 2), (np.float64(0.082), 0, 5), (np.float64(0.076), 0, 3), (np.float64(0.075), 0, 4)] L1H4 [(np.float64(0.08), 0, 2), (np.float64(0.078), 0, 4), (np.float64(0.074), 0, 5), (np.float64(0.046), 0, 1)] L1H5 [(np.float64(0.198), 0, 3), (np.float64(0.102), 0, 2), (np.float64(0.062), 0, 5), (np.float64(0.061), 0, 4)] L2H0 [(np.float64(0.208), 1, 3), (np.float64(0.196), 0, 2), (np.float64(0.189), 0, 5), (np.float64(0.164), 1, 1)] L2H1 [(np.float64(0.197), 0, 5), (np.float64(0.179), 0, 2), (np.float64(0.116), 0, 1), (np.float64(0.08), 0, 4)] L2H2 [(np.float64(0.184), 0, 5), (np.float64(0.182), 0, 2), (np.float64(0.123), 0, 1), (np.float64(0.099), 0, 3)] L2H3 [(np.float64(0.129), 0, 2), (np.float64(0.11), 0, 5), (np.float64(0.096), 0, 1), (np.float64(0.084), 0, 4)] L2H4 [(np.float64(0.186), 0, 2), (np.float64(0.18), 0, 5), (np.float64(0.159), 0, 3), (np.float64(0.128), 0, 1)] L2H5 [(np.float64(0.21), 0, 5), (np.float64(0.187), 0, 2), (np.float64(0.112), 0, 1), (np.float64(0.081), 0, 4)] L3H0 [(np.float64(0.107), 0, 2), (np.float64(0.085), 0, 5), (np.float64(0.069), 0, 1), (np.float64(0.047), 0, 4)] L3H1 [(np.float64(0.093), 0, 2), (np.float64(0.075), 0, 5), (np.float64(0.072), 1, 2), (np.float64(0.069), 0, 1)] L3H2 [(np.float64(0.081), 0, 2), (np.float64(0.046), 0, 1), (np.float64(0.033), 0, 3), (np.float64(0.016), 0, 5)] L3H3 [(np.float64(0.922), 0, 3), (np.float64(0.806), 2, 5), (np.float64(0.562), 2, 1), (np.float64(0.514), 1, 0)] L3H4 [(np.float64(0.276), 2, 2), (np.float64(0.272), 2, 4), (np.float64(0.202), 2, 5), (np.float64(0.19), 0, 4)] L3H5 [(np.float64(0.05), 0, 2), (np.float64(0.047), 0, 5), (np.float64(0.016), 0, 1), (np.float64(-0.001), 0, 4)] L0H3 Q,K,V singular spectra W_q [2.962 2.894 2.873 2.845 2.806 2.792 2.766 2.756 2.703 2.634 2.588 2.561 2.544 2.512 2.447 2.438 2.426 2.336 2.33 2.259 2.143 2.098 1.968 1.946 1.838 1.756 1.721 1.685 1.575 1.54 1.495 1.463 1.423 1.378 1.283 1.136 1.116 1.094 1.06 1.023 0.991 0.969 0.962 0.897 0.864 0.824 0.778 0.77 0.731 0.725 0.698 0.688 0.663 0.637 0.614 0.589 0.573 0.556 0.532 0.519 0.497 0.474 0.443 0.352] W_k [3.362 3.238 3.209 3.181 3.099 3.086 3.024 2.999 2.951 2.923 2.856 2.765 2.731 2.654 2.622 2.539 2.478 2.407 2.31 2.217 2.134 2.057 1.976 1.806 1.76 1.737 1.61 1.6 1.536 1.42 1.296 1.229 1.198 1.172 1.117 1.058 1.016 0.988 0.942 0.878 0.85 0.812 0.785 0.751 0.72 0.711 0.704 0.685 0.641 0.61 0.601 0.578 0.562 0.542 0.532 0.526 0.505 0.492 0.484 0.453 0.433 0.421 0.396 0.366] W_v [2.102 1.948 1.903 1.866 1.843 1.819 1.769 1.755 1.674 1.651 1.629 1.621 1.589 1.553 1.52 1.465 1.417 1.303 1.16 1.101 0.945 0.93 0.848 0.818 0.79 0.779 0.745 0.739 0.73 0.701 0.688 0.658 0.642 0.622 0.618 0.605 0.575 0.565 0.543 0.522 0.515 0.491 0.481 0.466 0.446 0.43 0.41 0.4 0.378 0.366 0.359 0.349 0.345 0.342 0.336 0.329 0.318 0.313 0.313 0.298 0.287 0.279 0.271 0.262] MLP0 input spectrum [17.584 13.477 11.766 10.188 9.613 9.54 9.344 9.149 8.754 8.681 8.593 8.39 8.008 7.926 7.84 7.731 7.517 7.386 7.159 7.084 7.068 6.915 6.889 6.682 6.631 6.491 6.436 6.354 6.307 6.16 6.003 5.909 5.835 5.661 5.466 5.307 5.289 5.132 5.047 4.957 4.895 4.443 4.367 4.111 4.038 3.8 3.63 3.408 3.226 3.154 2.964 2.947 2.817 2.635 2.605 2.501 2.481 2.393 2.351 2.276 2.201 2.148 2.136 2.101 2.066 2.047 1.993 1.957 1.89 1.858 1.85 1.833 1.781 1.745 1.734 1.726 1.71 1.675 1.666 1.644 1.625 1.603 1.579 1.555 1.55 1.533 1.518 1.506 1.492 1.464 1.457 1.437 1.428 1.414 1.404 1.398 1.381 1.374 1.372 1.362] Positional-like QK L0H3 versus value subspaces: squared overlap 16 Q/K 0.62027174 Q/V 0.01773982 K/V 0.018137278 24 Q/K 0.7562537 Q/V 0.08100366 K/V 0.090799265 28 Q/K 0.74191415 Q/V 0.15339868 K/V 0.16993944 32 Q/K 0.73780423 Q/V 0.22249849 K/V 0.24094298 40 Q/K 0.70587623 Q/V 0.32280862 K/V 0.35391036 ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(4)
g0=W["ln1_w.0"].astype(float)
q=W["W_q.0"][192:256].astype(float)
k=W["W_k.0"][192:256].astype(float)
v=W["W_v.0"][192:256].astype(float)
o=W["W_o.0"][:,192:256].astype(float)
_,_,qp=np.linalg.svd(q*g0,full_matrices=False)
_,_,vp=np.linalg.svd(v*g0,full_matrices=False)
P=qp[:20].T
T=vp[:20].T
B=o@(v*g0)
U=np.linalg.svd(B,full_matrices=False)[0][:,:20]
print("QK energy captured by candidate position, token and previous-token subspaces")
print("head | P-query P-key | T-query T-key | U-query U-key")
for l in range(4):
g=W[f"ln1_w.{l}"].astype(float)
for j in range(6):
s=slice(64*j,64*(j+1))
A=(W[f"W_q.{l}"][s]*g).T@(W[f"W_k.{l}"][s]*g)/8
den=np.linalg.norm(A)**2
vals=[]
for Z in [P,T,U]:
vals += [np.linalg.norm(Z.T@A)**2/den,np.linalg.norm(A@Z)**2/den]
print("L%dH%d"%(l,j),np.round(vals,3))
print("\nL0H3 raw QK nonzero eigenvalues, descending magnitude")
e=np.linalg.eigvals(k@q.T/8)
e=e[np.argsort(-abs(e))]
print(np.round(e[:40],4))
print("positive-imag phases",np.round(np.sort(np.angle(e[(e.imag>0.01)&(abs(e)>.2)])),4))
F=W["W_fc.0"].astype(float)*W["ln2_w.0"]
print("\nMLP0 current/previous weight coupling (no activations)")
for r in [16,20,24,28]:
T=vp[:r].T
X=F@T
Y=F@B@T
X-=X.mean(0)
Y-=Y.mean(0)
C=X.T@Y
ss=np.linalg.svd(C,compute_uv=False)
ev=np.linalg.eigvalsh((C+C.T)/2)
print("r",r,"sv",np.round(ss[:10],2),
"sym eig extremes",np.round(np.r_[ev[:5],ev[-5:]],2),
"trace/norm",round(np.trace(C)/np.linalg.norm(C),3))
```
{
"t": 1792713422.273488,
"s": 0.95,
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\ng0=W[\"ln1_w.0\"].astype(float)\nq=W[\"W_q.0\"][192:256].astype(float)\nk=W[\"W_k.0\"][192:256].astype(float)\nv=W[\"W_v.0\"][192:256].astype(float)\no=W[\"W_o.0\"][:,192:256].astype(float)\n_,_,qp=np.linalg.svd(q*g0,full_matrices=False)\n_,_,vp=np.linalg.svd(v*g0,full_matrices=False)\nP=qp[:20].T\nT=vp[:20].T\nB=o@(v*g0)\nU=np.linalg.svd(B,full_matrices=False)[0][:,:20]\nprint(\"QK energy captured by candidate position, token and previous-token subspaces\")\nprint(\"head | P-query P-key | T-query T-key | U-query U-key\")\nfor l in range(4):\n g=W[f\"ln1_w.{l}\"].astype(float)\n for j in range(6):\n s=slice(64*j,64*(j+1))\n A=(W[f\"W_q.{l}\"][s]*g).T@(W[f\"W_k.{l}\"][s]*g)/8\n den=np.linalg.norm(A)**2\n vals=[]\n for Z in [P,T,U]:\n vals += [np.linalg.norm(Z.T@A)**2/den,np.linalg.norm(A@Z)**2/den]\n print(\"L%dH%d\"%(l,j),np.round(vals,3))\n\nprint(\"\\nL0H3 raw QK nonzero eigenvalues, descending magnitude\")\ne=np.linalg.eigvals(k@q.T/8)\ne=e[np.argsort(-abs(e))]\nprint(np.round(e[:40],4))\nprint(\"positive-imag phases\",np.round(np.sort(np.angle(e[(e.imag>0.01)&(abs(e)>.2)])),4))\n\nF=W[\"W_fc.0\"].astype(float)*W[\"ln2_w.0\"]\nprint(\"\\nMLP0 current/previous weight coupling (no activations)\")\nfor r in [16,20,24,28]:\n T=vp[:r].T\n X=F@T\n Y=F@B@T\n X-=X.mean(0)\n Y-=Y.mean(0)\n C=X.T@Y\n ss=np.linalg.svd(C,compute_uv=False)\n ev=np.linalg.eigvalsh((C+C.T)/2)\n print(\"r\",r,\"sv\",np.round(ss[:10],2),\n \"sym eig extremes\",np.round(np.r_[ev[:5],ev[-5:]],2),\n \"trace/norm\",round(np.trace(C)/np.linalg.norm(C),3))\n",
"out": "QK energy captured by candidate position, token and previous-token subspaces\nhead | P-query P-key | T-query T-key | U-query U-key\nL0H0 [0.339 0.363 0.101 0.115 0.058 0.054]\nL0H1 [0.378 0.144 0.033 0.092 0.055 0.052]\nL0H2 [0.466 0.328 0.036 0.17 0.047 0.041]\nL0H3 [0.834 0.745 0.053 0.062 0.102 0.097]\nL0H4 [0.318 0.163 0.057 0.204 0.048 0.079]\nL0H5 [0.259 0.099 0.035 0.498 0.056 0.028]\nL1H0 [0.046 0.077 0.087 0.171 0.133 0.07 ]\nL1H1 [0.052 0.082 0.099 0.198 0.139 0.083]\nL1H2 [0.046 0.077 0.08 0.177 0.178 0.106]\nL1H3 [0.057 0.086 0.071 0.164 0.141 0.087]\nL1H4 [0.042 0.074 0.059 0.148 0.201 0.086]\nL1H5 [0.063 0.08 0.105 0.186 0.163 0.063]\nL2H0 [0.058 0.079 0.052 0.092 0.055 0.05 ]\nL2H1 [0.052 0.059 0.056 0.077 0.076 0.063]\nL2H2 [0.054 0.06 0.046 0.076 0.061 0.056]\nL2H3 [0.049 0.05 0.043 0.064 0.05 0.064]\nL2H4 [0.051 0.068 0.053 0.075 0.071 0.06 ]\nL2H5 [0.052 0.06 0.055 0.084 0.073 0.062]\nL3H0 [0.047 0.054 0.048 0.063 0.06 0.067]\nL3H1 [0.052 0.052 0.05 0.057 0.045 0.059]\nL3H2 [0.049 0.057 0.05 0.064 0.047 0.051]\nL3H3 [0.046 0.048 0.07 0.048 0.079 0.092]\nL3H4 [0.048 0.06 0.057 0.059 0.042 0.05 ]\nL3H5 [0.048 0.054 0.051 0.065 0.052 0.057]\n\nL0H3 raw QK nonzero eigenvalues, descending magnitude\n[ 0.7237+0.6913j 0.7237-0.6913j 0.4463+0.8419j 0.4463-0.8419j 0.5721+0.7221j 0.5721-0.7221j 0.6425+0.6136j 0.6425-0.6136j 0.676 +0.4626j\n 0.676 -0.4626j 0.1094+0.7623j 0.1094-0.7623j 0.4629+0.6042j 0.4629-0.6042j 0.6213+0.j 0.4872+0.3765j 0.4872-0.3765j -0.0013+0.6066j\n -0.0013-0.6066j 0.3503+0.2828j 0.3503-0.2828j -0.2554+0.3536j -0.2554-0.3536j 0.0404+0.3357j 0.0404-0.3357j -0.2878+0.j -0.1337+0.2534j\n -0.1337-0.2534j -0.2357+0.j 0.157 +0.j -0.1513+0.0352j -0.1513-0.0352j -0.1234+0.0288j -0.1234-0.0288j 0.1132+0.j -0.0773+0.j\n 0.0001-0.0576j 0.0001+0.0576j -0.0294-0.0488j -0.0294+0.0488j]\npositive-imag phases [0.6001 0.6579 0.6792 0.7624 0.7625 0.9007 0.917 1.0833 1.4283 1.4509 1.5729 2.0562 2.1963]\n\nMLP0 current/previous weight coupling (no activations)\nr 16 sv [44.1 17.18 12.62 11.3 10.6 7.71 7.09 5.68 5.15 4.62] sym eig extremes [-8.47 -6.45 -5.87 -5.44 -3.33 4.47 7.16 8.74 12.96 40.31] trace/norm 0.878\nr 20 sv [54.57 20.6 14.08 13.3 12.07 10.32 8.74 8.01 7.47 6.15] sym eig extremes [-10.31 -9.43 -6.73 -6.42 -5.89 5.1 8.09 8.96 14.32 46.97] trace/norm 0.747\nr 24 sv [58.11 21.8 15.33 14.05 13.63 12.33 10.91 9.14 8.13 6.36] sym eig extremes [-11.89 -10.16 -8.09 -7.55 -6.38 5.71 9.1 9.75 14.6 49.13] trace/norm 0.722\nr 28 sv [58.41 22.03 16.73 14.78 13.93 13.1 11.42 9.69 8.34 6.93] sym eig extremes [-12.34 -10.37 -8.3 -7.88 -6.38 6.26 9.92 10.21 14.79 49.31] trace/norm 0.712\n"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 5 of at most 100: ``` QK energy captured by candidate position, token and previous-token subspaces head | P-query P-key | T-query T-key | U-query U-key L0H0 [0.339 0.363 0.101 0.115 0.058 0.054] L0H1 [0.378 0.144 0.033 0.092 0.055 0.052] L0H2 [0.466 0.328 0.036 0.17 0.047 0.041] L0H3 [0.834 0.745 0.053 0.062 0.102 0.097] L0H4 [0.318 0.163 0.057 0.204 0.048 0.079] L0H5 [0.259 0.099 0.035 0.498 0.056 0.028] L1H0 [0.046 0.077 0.087 0.171 0.133 0.07 ] L1H1 [0.052 0.082 0.099 0.198 0.139 0.083] L1H2 [0.046 0.077 0.08 0.177 0.178 0.106] L1H3 [0.057 0.086 0.071 0.164 0.141 0.087] L1H4 [0.042 0.074 0.059 0.148 0.201 0.086] L1H5 [0.063 0.08 0.105 0.186 0.163 0.063] L2H0 [0.058 0.079 0.052 0.092 0.055 0.05 ] L2H1 [0.052 0.059 0.056 0.077 0.076 0.063] L2H2 [0.054 0.06 0.046 0.076 0.061 0.056] L2H3 [0.049 0.05 0.043 0.064 0.05 0.064] L2H4 [0.051 0.068 0.053 0.075 0.071 0.06 ] L2H5 [0.052 0.06 0.055 0.084 0.073 0.062] L3H0 [0.047 0.054 0.048 0.063 0.06 0.067] L3H1 [0.052 0.052 0.05 0.057 0.045 0.059] L3H2 [0.049 0.057 0.05 0.064 0.047 0.051] L3H3 [0.046 0.048 0.07 0.048 0.079 0.092] L3H4 [0.048 0.06 0.057 0.059 0.042 0.05 ] L3H5 [0.048 0.054 0.051 0.065 0.052 0.057] L0H3 raw QK nonzero eigenvalues, descending magnitude [ 0.7237+0.6913j 0.7237-0.6913j 0.4463+0.8419j 0.4463-0.8419j 0.5721+0.7221j 0.5721-0.7221j 0.6425+0.6136j 0.6425-0.6136j 0.676 +0.4626j 0.676 -0.4626j 0.1094+0.7623j 0.1094-0.7623j 0.4629+0.6042j 0.4629-0.6042j 0.6213+0.j 0.4872+0.3765j 0.4872-0.3765j -0.0013+0.6066j -0.0013-0.6066j 0.3503+0.2828j 0.3503-0.2828j -0.2554+0.3536j -0.2554-0.3536j 0.0404+0.3357j 0.0404-0.3357j -0.2878+0.j -0.1337+0.2534j -0.1337-0.2534j -0.2357+0.j 0.157 +0.j -0.1513+0.0352j -0.1513-0.0352j -0.1234+0.0288j -0.1234-0.0288j 0.1132+0.j -0.0773+0.j 0.0001-0.0576j 0.0001+0.0576j -0.0294-0.0488j -0.0294+0.0488j] positive-imag phases [0.6001 0.6579 0.6792 0.7624 0.7625 0.9007 0.917 1.0833 1.4283 1.4509 1.5729 2.0562 2.1963] MLP0 current/previous weight coupling (no activations) r 16 sv [44.1 17.18 12.62 11.3 10.6 7.71 7.09 5.68 5.15 4.62] sym eig extremes [-8.47 -6.45 -5.87 -5.44 -3.33 4.47 7.16 8.74 12.96 40.31] trace/norm 0.878 r 20 sv [54.57 20.6 14.08 13.3 12.07 10.32 8.74 8.01 7.47 6.15] sym eig extremes [-10.31 -9.43 -6.73 -6.42 -5.89 5.1 8.09 8.96 14.32 46.97] trace/norm 0.747 r 24 sv [58.11 21.8 15.33 14.05 13.63 12.33 10.91 9.14 8.13 6.36] sym eig extremes [-11.89 -10.16 -8.09 -7.55 -6.38 5.71 9.1 9.75 14.6 49.13] trace/norm 0.722 r 28 sv [58.41 22.03 16.73 14.78 13.93 13.1 11.42 9.69 8.34 6.93] sym eig extremes [-12.34 -10.37 -8.3 -7.88 -6.38 6.26 9.92 10.21 14.79 49.31] trace/norm 0.712 ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(4)
g0=W["ln1_w.0"].astype(float)
v=W["W_v.0"][192:256].astype(float)*g0
o=W["W_o.0"][:,192:256].astype(float)
B=o@v
T=np.linalg.svd(v,full_matrices=False)[2][:20].T
coords=np.array([21,39,58,99,184])
print("Counter-coordinate blocks and induction-path diagnostics")
for l in range(1,4):
g=W[f"ln1_w.{l}"].astype(float)
b=W[f"ln1_b.{l}"].astype(float)
for j in range(6):
s=slice(64*j,64*(j+1))
q=W[f"W_q.{l}"][s].astype(float)
k=W[f"W_k.{l}"][s].astype(float)
A=(q*g).T@(k*g)/8
r=(q@b)@(k*g)/8
C=T.T@A@B@T
eig=np.linalg.eigvals(C)
sym=np.linalg.eigvalsh((C+C.T)/2)
print("L%dH%d"%(l,j),
"A21,21=%.3f col21energy=%.3f row21energy=%.3f r21=%.3f"%
(A[21,21],np.sum(A[:,21]**2)/np.sum(A*A),np.sum(A[21]**2)/np.sum(A*A),r[21]),
"token-path tr/norm=%.3f positive_sym=%d/20"%
(np.trace(C)/(np.linalg.norm(C)+1e-12),sum(sym>0)),
"sym_extrema",np.round([sym[0],sym[-1]],3))
if j in [0,3,4]:
print("A block rows/cols",coords.tolist())
print(np.round(A[np.ix_(coords,coords)],3))
print("\nL0 QK/OV effective ranks")
for j in range(6):
s=slice(64*j,64*(j+1))
A=(W["W_q.0"][s]*g0).T@(W["W_k.0"][s]*g0)/8
B=W["W_o.0"][:,s]@(W["W_v.0"][s]*g0)
for name,M in [("QK",A),("OV",B)]:
sv=np.linalg.svd(M,compute_uv=False)
energy=np.cumsum(sv**2)/np.sum(sv**2)
print("L0H%d %s"%(j,name),"rank90/95/99",np.searchsorted(energy,[.90,.95,.99])+1)
```
{
"t": 1792713491.377971,
"s": 1.2,
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\ng0=W[\"ln1_w.0\"].astype(float)\nv=W[\"W_v.0\"][192:256].astype(float)*g0\no=W[\"W_o.0\"][:,192:256].astype(float)\nB=o@v\nT=np.linalg.svd(v,full_matrices=False)[2][:20].T\ncoords=np.array([21,39,58,99,184])\nprint(\"Counter-coordinate blocks and induction-path diagnostics\")\nfor l in range(1,4):\n g=W[f\"ln1_w.{l}\"].astype(float)\n b=W[f\"ln1_b.{l}\"].astype(float)\n for j in range(6):\n s=slice(64*j,64*(j+1))\n q=W[f\"W_q.{l}\"][s].astype(float)\n k=W[f\"W_k.{l}\"][s].astype(float)\n A=(q*g).T@(k*g)/8\n r=(q@b)@(k*g)/8\n C=T.T@A@B@T\n eig=np.linalg.eigvals(C)\n sym=np.linalg.eigvalsh((C+C.T)/2)\n print(\"L%dH%d\"%(l,j),\n \"A21,21=%.3f col21energy=%.3f row21energy=%.3f r21=%.3f\"%\n (A[21,21],np.sum(A[:,21]**2)/np.sum(A*A),np.sum(A[21]**2)/np.sum(A*A),r[21]),\n \"token-path tr/norm=%.3f positive_sym=%d/20\"%\n (np.trace(C)/(np.linalg.norm(C)+1e-12),sum(sym>0)),\n \"sym_extrema\",np.round([sym[0],sym[-1]],3))\n if j in [0,3,4]:\n print(\"A block rows/cols\",coords.tolist())\n print(np.round(A[np.ix_(coords,coords)],3))\n\nprint(\"\\nL0 QK/OV effective ranks\")\nfor j in range(6):\n s=slice(64*j,64*(j+1))\n A=(W[\"W_q.0\"][s]*g0).T@(W[\"W_k.0\"][s]*g0)/8\n B=W[\"W_o.0\"][:,s]@(W[\"W_v.0\"][s]*g0)\n for name,M in [(\"QK\",A),(\"OV\",B)]:\n sv=np.linalg.svd(M,compute_uv=False)\n energy=np.cumsum(sv**2)/np.sum(sv**2)\n print(\"L0H%d %s\"%(j,name),\"rank90/95/99\",np.searchsorted(energy,[.90,.95,.99])+1)\n",
"out": "Counter-coordinate blocks and induction-path diagnostics\nL1H0 A21,21=0.087 col21energy=0.301 row21energy=0.012 r21=3.627 token-path tr/norm=0.329 positive_sym=12/20 sym_extrema [-0.038 0.039]\nA block rows/cols [21, 39, 58, 99, 184]\n[[ 0.087 0.05 -0.058 0.184 -0.002]\n [-0.123 0.016 -0.012 0.001 -0.001]\n [ 0.159 -0.02 0.015 -0.001 0.001]\n [-0.265 0.058 -0.053 0.073 -0.002]\n [-0.105 0.003 0.001 -0.029 -0.001]]\nL1H1 A21,21=0.090 col21energy=0.325 row21energy=0.015 r21=3.352 token-path tr/norm=0.630 positive_sym=10/20 sym_extrema [-0.034 0.056]\nL1H2 A21,21=0.077 col21energy=0.364 row21energy=0.011 r21=2.769 token-path tr/norm=0.479 positive_sym=10/20 sym_extrema [-0.047 0.041]\nL1H3 A21,21=0.089 col21energy=0.364 row21energy=0.016 r21=3.079 token-path tr/norm=0.163 positive_sym=10/20 sym_extrema [-0.049 0.037]\nA block rows/cols [21, 39, 58, 99, 184]\n[[ 0.089 0.043 -0.062 0.188 0.001]\n [-0.122 0.015 -0.016 -0. 0. ]\n [ 0.15 -0.018 0.018 0.004 -0. ]\n [-0.267 0.054 -0.063 0.075 -0.001]\n [-0.106 0.007 -0.005 -0.023 0. ]]\nL1H4 A21,21=0.080 col21energy=0.348 row21energy=0.008 r21=2.008 token-path tr/norm=0.152 positive_sym=11/20 sym_extrema [-0.027 0.027]\nA block rows/cols [21, 39, 58, 99, 184]\n[[ 0.08 0.022 -0.032 0.112 0.002]\n [-0.072 0.013 -0.019 0.009 -0. ]\n [ 0.084 -0.016 0.023 -0.014 0. ]\n [-0.164 0.037 -0.053 0.047 -0.001]\n [-0.049 0.004 -0.009 -0.009 0. ]]\nL1H5 A21,21=0.092 col21energy=0.319 row21energy=0.018 r21=3.563 token-path tr/norm=0.515 positive_sym=11/20 sym_extrema [-0.05 0.055]\nL2H0 A21,21=0.062 col21energy=0.154 row21energy=0.021 r21=1.815 token-path tr/norm=0.284 positive_sym=10/20 sym_extrema [-0.015 0.022]\nA block rows/cols [21, 39, 58, 99, 184]\n[[ 0.062 0.067 -0.067 0.202 0. ]\n [-0.164 0.017 -0.014 -0.01 0.006]\n [ 0.15 -0.015 0.012 0.003 -0.004]\n [-0.228 0.058 -0.051 0.096 0.004]\n [-0.051 -0.002 0.003 -0.028 0.003]]\nL2H1 A21,21=0.061 col21energy=0.056 row21energy=0.010 r21=1.682 token-path tr/norm=0.105 positive_sym=11/20 sym_extrema [-0.028 0.03 ]\nL2H2 A21,21=0.057 col21energy=0.079 row21energy=0.010 r21=1.411 token-path tr/norm=0.345 positive_sym=10/20 sym_extrema [-0.016 0.02 ]\nL2H3 A21,21=0.039 col21energy=0.071 row21energy=0.004 r21=0.803 token-path tr/norm=0.435 positive_sym=10/20 sym_extrema [-0.018 0.025]\nA block rows/cols [21, 39, 58, 99, 184]\n[[ 0.039 0.026 -0.025 0.083 0.002]\n [-0.061 0.008 -0.011 0.004 0.003]\n [ 0.049 -0.007 0.009 -0.007 -0.002]\n [-0.098 0.023 -0.021 0.031 0. ]\n [-0.011 -0.002 -0.002 -0.005 0.003]]\nL2H4 A21,21=0.059 col21energy=0.102 row21energy=0.013 r21=2.069 token-path tr/norm=0.700 positive_sym=11/20 sym_extrema [-0.028 0.034]\nA block rows/cols [21, 39, 58, 99, 184]\n[[ 0.059 0.072 -0.076 0.22 -0. ]\n [-0.171 0.014 -0.016 -0.029 0.004]\n [ 0.141 -0.011 0.014 0.022 -0.002]\n [-0.264 0.062 -0.058 0.092 -0.002]\n [-0.053 -0.005 0.002 -0.045 0.005]]\nL2H5 A21,21=0.061 col21energy=0.062 row21energy=0.010 r21=1.859 token-path tr/norm=-0.035 positive_sym=10/20 sym_extrema [-0.028 0.028]\nL3H0 A21,21=0.019 col21energy=0.031 row21energy=0.005 r21=0.408 token-path tr/norm=-0.629 positive_sym=8/20 sym_extrema [-0.032 0.029]\nA block rows/cols [21, 39, 58, 99, 184]\n[[ 0.019 0.053 -0.046 0.104 0.003]\n [-0.071 0.006 -0. -0.002 0.006]\n [ 0.04 -0.004 0.006 0.002 -0.006]\n [-0.096 0.017 -0.003 0.028 0.006]\n [-0.032 0.003 0.001 -0.006 0.007]]\nL3H1 A21,21=0.031 col21energy=0.052 row21energy=0.006 r21=0.181 token-path tr/norm=-0.222 positive_sym=9/20 sym_extrema [-0.014 0.014]\nL3H2 A21,21=0.024 col21energy=0.053 row21energy=0.009 r21=0.382 token-path tr/norm=0.193 positive_sym=11/20 sym_extrema [-0.014 0.018]\nL3H3 A21,21=0.007 col21energy=0.000 row21energy=0.000 r21=-0.011 token-path tr/norm=2.860 positive_sym=16/20 sym_extrema [-0.007 0.06 ]\nA block rows/cols [21, 39, 58, 99, 184]\n[[ 0.007 0.002 0. 0.005 0.003]\n [-0. 0.004 -0.003 0.005 0.019]\n [-0.002 0.003 0.01 0.005 0"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 6 of at most 100: ``` Counter-coordinate blocks and induction-path diagnostics L1H0 A21,21=0.087 col21energy=0.301 row21energy=0.012 r21=3.627 token-path tr/norm=0.329 positive_sym=12/20 sym_extrema [-0.038 0.039] A block rows/cols [21, 39, 58, 99, 184] [[ 0.087 0.05 -0.058 0.184 -0.002] [-0.123 0.016 -0.012 0.001 -0.001] [ 0.159 -0.02 0.015 -0.001 0.001] [-0.265 0.058 -0.053 0.073 -0.002] [-0.105 0.003 0.001 -0.029 -0.001]] L1H1 A21,21=0.090 col21energy=0.325 row21energy=0.015 r21=3.352 token-path tr/norm=0.630 positive_sym=10/20 sym_extrema [-0.034 0.056] L1H2 A21,21=0.077 col21energy=0.364 row21energy=0.011 r21=2.769 token-path tr/norm=0.479 positive_sym=10/20 sym_extrema [-0.047 0.041] L1H3 A21,21=0.089 col21energy=0.364 row21energy=0.016 r21=3.079 token-path tr/norm=0.163 positive_sym=10/20 sym_extrema [-0.049 0.037] A block rows/cols [21, 39, 58, 99, 184] [[ 0.089 0.043 -0.062 0.188 0.001] [-0.122 0.015 -0.016 -0. 0. ] [ 0.15 -0.018 0.018 0.004 -0. ] [-0.267 0.054 -0.063 0.075 -0.001] [-0.106 0.007 -0.005 -0.023 0. ]] L1H4 A21,21=0.080 col21energy=0.348 row21energy=0.008 r21=2.008 token-path tr/norm=0.152 positive_sym=11/20 sym_extrema [-0.027 0.027] A block rows/cols [21, 39, 58, 99, 184] [[ 0.08 0.022 -0.032 0.112 0.002] [-0.072 0.013 -0.019 0.009 -0. ] [ 0.084 -0.016 0.023 -0.014 0. ] [-0.164 0.037 -0.053 0.047 -0.001] [-0.049 0.004 -0.009 -0.009 0. ]] L1H5 A21,21=0.092 col21energy=0.319 row21energy=0.018 r21=3.563 token-path tr/norm=0.515 positive_sym=11/20 sym_extrema [-0.05 0.055] L2H0 A21,21=0.062 col21energy=0.154 row21energy=0.021 r21=1.815 token-path tr/norm=0.284 positive_sym=10/20 sym_extrema [-0.015 0.022] A block rows/cols [21, 39, 58, 99, 184] [[ 0.062 0.067 -0.067 0.202 0. ] [-0.164 0.017 -0.014 -0.01 0.006] [ 0.15 -0.015 0.012 0.003 -0.004] [-0.228 0.058 -0.051 0.096 0.004] [-0.051 -0.002 0.003 -0.028 0.003]] L2H1 A21,21=0.061 col21energy=0.056 row21energy=0.010 r21=1.682 token-path tr/norm=0.105 positive_sym=11/20 sym_extrema [-0.028 0.03 ] L2H2 A21,21=0.057 col21energy=0.079 row21energy=0.010 r21=1.411 token-path tr/norm=0.345 positive_sym=10/20 sym_extrema [-0.016 0.02 ] L2H3 A21,21=0.039 col21energy=0.071 row21energy=0.004 r21=0.803 token-path tr/norm=0.435 positive_sym=10/20 sym_extrema [-0.018 0.025] A block rows/cols [21, 39, 58, 99, 184] [[ 0.039 0.026 -0.025 0.083 0.002] [-0.061 0.008 -0.011 0.004 0.003] [ 0.049 -0.007 0.009 -0.007 -0.002] [-0.098 0.023 -0.021 0.031 0. ] [-0.011 -0.002 -0.002 -0.005 0.003]] L2H4 A21,21=0.059 col21energy=0.102 row21energy=0.013 r21=2.069 token-path tr/norm=0.700 positive_sym=11/20 sym_extrema [-0.028 0.034] A block rows/cols [21, 39, 58, 99, 184] [[ 0.059 0.072 -0.076 0.22 -0. ] [-0.171 0.014 -0.016 -0.029 0.004] [ 0.141 -0.011 0.014 0.022 -0.002] [-0.264 0.062 -0.058 0.092 -0.002] [-0.053 -0.005 0.002 -0.045 0.005]] L2H5 A21,21=0.061 col21energy=0.062 row21energy=0.010 r21=1.859 token-path tr/norm=-0.035 positive_sym=10/20 sym_extrema [-0.028 0.028] L3H0 A21,21=0.019 col21energy=0.031 row21energy=0.005 r21=0.408 token-path tr/norm=-0.629 positive_sym=8/20 sym_extrema [-0.032 0.029] A block rows/cols [21, 39, 58, 99, 184] [[ 0.019 0.053 -0.046 0.104 0.003] [-0.071 0.006 -0. -0.002 0.006] [ 0.04 -0.004 0.006 0.002 -0.006] [-0.096 0.017 -0.003 0.028 0.006] [-0.032 0.003 0.001 -0.006 0.007]] L3H1 A21,21=0.031 col21energy=0.052 row21energy=0.006 r21=0.181 token-path tr/norm=-0.222 positive_sym=9/20 sym_extrema [-0.014 0.014] L3H2 A21,21=0.024 col21energy=0.053 row21energy=0.009 r21=0.382 token-path tr/norm=0.193 positive_sym=11/20 sym_extrema [-0.014 0.018] L3H3 A21,21=0.007 col21energy=0.000 row21energy=0.000 r21=-0.011 token-path tr/norm=2.860 positive_sym=16/20 sym_extrema [-0.007 0.06 ] A block rows/cols [21, 39, 58, 99, 184] [[ 0.007 0.002 0. 0.005 0.003] [-0. 0.004 -0.003 0.005 0.019] [-0.002 0.003 0.01 0.005 0.005] [ 0. 0. 0.007 0.009 0.002] [ 0.004 -0. -0.006 0.005 0.019]] L3H4 A21,21=0.001 col21energy=0.005 row21energy=0.002 r21=0.028 token-path tr/norm=0.662 positive_sym=11/20 sym_extrema [-0.005 0.009] A block rows/cols [21, 39, 58, 99, 184] [[ 0.001 0.001 -0.004 0.014 0.002] [-0.004 0. -0.001 0.006 0.009] [-0. 0.001 0. 0.002 0.001] [-0.006 -0. 0.001 -0.001 0.006] [-0.003 -0.001 0. 0.005 0.006]] L3H5 A21,21=0.014 col21energy=0.052 row21energy=0.006 r21=0.416 token-path tr/norm=-0.132 positive_sym=10/20 sym_extrema [-0.02 0.019] L0 QK/OV effective ranks L0H0 QK rank90/95/99 [ 6 10 21] L0H0 OV rank90/95/99 [12 15 25] L0H1 QK rank90/95/99 [2 2 6] L0H1 OV rank90/95/99 [2 2 5] L0H2 QK rank90/95/99 [ 1 3 14] L0H2 OV rank90/95/99 [2 2 9] L0H3 QK rank90/95/99 [19 23 30] L0H3 OV rank90/95/99 [20 26 37] L0H4 QK rank90/95/99 [ 3 5 22] L0H4 OV rank90/95/99 [ 2 4 19] L0H5 QK rank90/95/99 [ 1 10 35] L0H5 OV rank90/95/99 [2 2 4] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from sklearn.cluster import KMeans
from sklearn.decomposition import FastICA
from scipy.stats import kurtosis
from threadpoolctl import threadpool_limits
threadpool_limits(4)
g=W["ln1_w.0"].astype(float)
V=W["W_v.0"][192:256].astype(float)*g
B=W["W_o.0"][:,192:256].astype(float)@V
T=np.linalg.svd(V,full_matrices=False)[2][:24].T
F=W["W_fc.0"].astype(float)*W["ln2_w.0"]
X=F@T
Y=F@B@T
X-=X.mean(0); Y-=Y.mean(0)
Z=np.vstack([X,Y])
cov=Z.T@Z/len(Z)
ev,U=np.linalg.eigh(cov)
wh=U@np.diag(1/np.sqrt(ev+1e-6))@U.T
Z=Z@wh
Z/=np.linalg.norm(Z,axis=1,keepdims=True)
print("Clustering ONLY first-MLP weight projections onto current/previous value subspaces")
for nc in [16,20,24,28]:
km=KMeans(n_clusters=nc,n_init=20,random_state=7).fit(Z)
centers=km.cluster_centers_
print("clusters",nc,"mean within-cluster cosine",round(np.mean(np.linalg.norm(centers,axis=1)),3),
"counts",np.sort(np.bincount(km.labels_,minlength=nc))[::-1])
if nc==24:
tab=np.zeros((nc,nc),int)
for a,b in zip(km.labels_[:1536],km.labels_[1536:]):
tab[a,b]+=1
expected=tab.sum(1)[:,None]*tab.sum(0)[None,:]/tab.sum()
R=(tab-expected)/np.sqrt(expected+1)
print("pair-table standardized residual singular values",np.round(np.linalg.svd(R,compute_uv=False)[:12],2))
ij=np.dstack(np.unravel_index(np.argsort(R.ravel())[-20:][::-1],R.shape))[0]
print("largest pair enrichments (arbitrary cluster labels):",
[(int(a),int(b),int(tab[a,b]),round(float(expected[a,b]),1),round(float(R[a,b]),2)) for a,b in ij])
print("\nLast-MLP output spectrum: adjacent ratios")
sv=np.linalg.svd(W["W_proj.3"].astype(float),compute_uv=False)
print([(i+1,round(sv[i],3),round(sv[i]/sv[i+1],3)) for i in range(18,32)])
print("top24 energy",np.sum(sv[:24]**2)/np.sum(sv**2),"top28 energy",np.sum(sv[:28]**2)/np.sum(sv**2))
```
{
"t": 1792713552.22062,
"s": 13.56,
"code": "import numpy as np\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import FastICA\nfrom scipy.stats import kurtosis\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\ng=W[\"ln1_w.0\"].astype(float)\nV=W[\"W_v.0\"][192:256].astype(float)*g\nB=W[\"W_o.0\"][:,192:256].astype(float)@V\nT=np.linalg.svd(V,full_matrices=False)[2][:24].T\nF=W[\"W_fc.0\"].astype(float)*W[\"ln2_w.0\"]\nX=F@T\nY=F@B@T\nX-=X.mean(0); Y-=Y.mean(0)\nZ=np.vstack([X,Y])\ncov=Z.T@Z/len(Z)\nev,U=np.linalg.eigh(cov)\nwh=U@np.diag(1/np.sqrt(ev+1e-6))@U.T\nZ=Z@wh\nZ/=np.linalg.norm(Z,axis=1,keepdims=True)\nprint(\"Clustering ONLY first-MLP weight projections onto current/previous value subspaces\")\nfor nc in [16,20,24,28]:\n km=KMeans(n_clusters=nc,n_init=20,random_state=7).fit(Z)\n centers=km.cluster_centers_\n print(\"clusters\",nc,\"mean within-cluster cosine\",round(np.mean(np.linalg.norm(centers,axis=1)),3),\n \"counts\",np.sort(np.bincount(km.labels_,minlength=nc))[::-1])\n if nc==24:\n tab=np.zeros((nc,nc),int)\n for a,b in zip(km.labels_[:1536],km.labels_[1536:]):\n tab[a,b]+=1\n expected=tab.sum(1)[:,None]*tab.sum(0)[None,:]/tab.sum()\n R=(tab-expected)/np.sqrt(expected+1)\n print(\"pair-table standardized residual singular values\",np.round(np.linalg.svd(R,compute_uv=False)[:12],2))\n ij=np.dstack(np.unravel_index(np.argsort(R.ravel())[-20:][::-1],R.shape))[0]\n print(\"largest pair enrichments (arbitrary cluster labels):\",\n [(int(a),int(b),int(tab[a,b]),round(float(expected[a,b]),1),round(float(R[a,b]),2)) for a,b in ij])\n\nprint(\"\\nLast-MLP output spectrum: adjacent ratios\")\nsv=np.linalg.svd(W[\"W_proj.3\"].astype(float),compute_uv=False)\nprint([(i+1,round(sv[i],3),round(sv[i]/sv[i+1],3)) for i in range(18,32)])\nprint(\"top24 energy\",np.sum(sv[:24]**2)/np.sum(sv**2),\"top28 energy\",np.sum(sv[:28]**2)/np.sum(sv**2))\n",
"out": "Clustering ONLY first-MLP weight projections onto current/previous value subspaces\nclusters 16 mean within-cluster cosine 0.506 counts [281 253 227 226 215 210 207 202 200 187 156 156 150 142 141 119]\nclusters 20 mean within-cluster cosine 0.53 counts [229 221 206 204 201 195 180 160 152 141 140 135 131 126 121 120 116 108 97 89]\nclusters 24 mean within-cluster cosine 0.549 counts [227 219 210 196 191 161 148 131 127 121 117 112 105 104 103 101 101 100 93 93 87 78 77 70]\npair-table standardized residual singular values [21.65 10.39 9.11 8.02 7.06 6.58 6.12 5.33 5.06 4.9 4.21 4.09]\nlargest pair enrichments (arbitrary cluster labels): [(1, 5, 52, 17.8, 7.88), (22, 13, 17, 3.6, 6.26), (6, 0, 42, 17.3, 5.76), (17, 17, 12, 2.1, 5.58), (23, 23, 8, 1.1, 4.68), (9, 9, 10, 2.0, 4.57), (14, 14, 8, 1.3, 4.36), (18, 18, 9, 1.8, 4.28), (22, 7, 10, 2.4, 4.15), (6, 21, 41, 21.7, 4.04), (8, 21, 47, 26.1, 4.01), (10, 2, 9, 2.1, 3.88), (4, 4, 7, 1.4, 3.67), (8, 15, 13, 4.5, 3.65), (8, 12, 13, 4.5, 3.65), (8, 0, 37, 20.9, 3.45), (8, 11, 26, 13.1, 3.42), (7, 10, 7, 1.9, 3.04), (19, 9, 7, 1.9, 2.97), (9, 14, 7, 2.0, 2.92)]\nclusters 28 mean within-cluster cosine 0.567 counts [183 172 162 158 150 145 143 143 140 114 112 110 107 107 94 92 91 88 85 85 82 81 78 77 73 72 66 62]\n\nLast-MLP output spectrum: adjacent ratios\n[(19, np.float64(12.317), np.float64(1.068)), (20, np.float64(11.531), np.float64(1.044)), (21, np.float64(11.047), np.float64(1.216)), (22, np.float64(9.084), np.float64(1.087)), (23, np.float64(8.353), np.float64(1.014)), (24, np.float64(8.24), np.float64(1.294)), (25, np.float64(6.37), np.float64(1.092)), (26, np.float64(5.836), np.float64(1.261)), (27, np.float64(4.626), np.float64(1.009)), (28, np.float64(4.583), np.float64(1.213)), (29, np.float64(3.777), np.float64(1.046)), (30, np.float64(3.61), np.float64(1.11)), (31, np.float64(3.252), np.float64(1.022)), (32, np.float64(3.183), np.float64(1.031))]\ntop24 energy 0.9213614809422224 top28 energy 0.937515023916908\n"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 7 of at most 100: ``` Clustering ONLY first-MLP weight projections onto current/previous value subspaces clusters 16 mean within-cluster cosine 0.506 counts [281 253 227 226 215 210 207 202 200 187 156 156 150 142 141 119] clusters 20 mean within-cluster cosine 0.53 counts [229 221 206 204 201 195 180 160 152 141 140 135 131 126 121 120 116 108 97 89] clusters 24 mean within-cluster cosine 0.549 counts [227 219 210 196 191 161 148 131 127 121 117 112 105 104 103 101 101 100 93 93 87 78 77 70] pair-table standardized residual singular values [21.65 10.39 9.11 8.02 7.06 6.58 6.12 5.33 5.06 4.9 4.21 4.09] largest pair enrichments (arbitrary cluster labels): [(1, 5, 52, 17.8, 7.88), (22, 13, 17, 3.6, 6.26), (6, 0, 42, 17.3, 5.76), (17, 17, 12, 2.1, 5.58), (23, 23, 8, 1.1, 4.68), (9, 9, 10, 2.0, 4.57), (14, 14, 8, 1.3, 4.36), (18, 18, 9, 1.8, 4.28), (22, 7, 10, 2.4, 4.15), (6, 21, 41, 21.7, 4.04), (8, 21, 47, 26.1, 4.01), (10, 2, 9, 2.1, 3.88), (4, 4, 7, 1.4, 3.67), (8, 15, 13, 4.5, 3.65), (8, 12, 13, 4.5, 3.65), (8, 0, 37, 20.9, 3.45), (8, 11, 26, 13.1, 3.42), (7, 10, 7, 1.9, 3.04), (19, 9, 7, 1.9, 2.97), (9, 14, 7, 2.0, 2.92)] clusters 28 mean within-cluster cosine 0.567 counts [183 172 162 158 150 145 143 143 140 114 112 110 107 107 94 92 91 88 85 85 82 81 78 77 73 72 66 62] Last-MLP output spectrum: adjacent ratios [(19, np.float64(12.317), np.float64(1.068)), (20, np.float64(11.531), np.float64(1.044)), (21, np.float64(11.047), np.float64(1.216)), (22, np.float64(9.084), np.float64(1.087)), (23, np.float64(8.353), np.float64(1.014)), (24, np.float64(8.24), np.float64(1.294)), (25, np.float64(6.37), np.float64(1.092)), (26, np.float64(5.836), np.float64(1.261)), (27, np.float64(4.626), np.float64(1.009)), (28, np.float64(4.583), np.float64(1.213)), (29, np.float64(3.777), np.float64(1.046)), (30, np.float64(3.61), np.float64(1.11)), (31, np.float64(3.252), np.float64(1.022)), (32, np.float64(3.183), np.float64(1.031))] top24 energy 0.9213614809422224 top28 energy 0.937515023916908 ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import fractional_matrix_power
from threadpoolctl import threadpool_limits
threadpool_limits(4)
for scaled in [False,True]:
g=W["ln1_w.0"].astype(float) if scaled else np.ones(384)
As=[]
for j in range(6):
s=slice(64*j,64*(j+1))
As.append((W["W_q.0"][s]*g).T@(W["W_k.0"][s]*g)/8)
print("\nScaled" if scaled else "\nUnscaled","QK kernel power comparisons")
ref=As[3]/np.linalg.norm(As[3],2)
powers=[np.eye(384)]
for z in range(8):
powers.append(powers[-1]@ref)
for j in range(6):
print("L0H%d cos powers 0..8"%j,np.round([
np.sum(As[j]*P)/(np.linalg.norm(As[j])*np.linalg.norm(P)) for P in powers],3))
q=W["W_q.0"][192:256].astype(float)*g
k=W["W_k.0"][192:256].astype(float)*g
_,s,P=np.linalg.svd(np.vstack([q,k]),full_matrices=False)
for r in [16,24,32]:
Z=P[:r].T
# Whitening of a weight Gram matrix, not a network normalization.
H=(Z.T@(q.T@q+k.T@k)@Z)/16
ev,U=np.linalg.eigh(H)
wh=U@np.diag(1/np.sqrt(ev))@U.T
M=[wh@Z.T@A@Z@wh for A in As]
u,sv,vt=np.linalg.svd(M[3])
R=u@vt
print("rank",r,"polar power comparison")
for j in [0,1,2,4,5]:
vals=[]
for lag in range(9):
z=np.linalg.matrix_power(R,lag)
vals.append(np.sum(M[j]*z)/(np.linalg.norm(M[j])*np.linalg.norm(z)))
print("H",j,np.round(vals,3))
```
{
"t": 1792713616.9029064,
"s": 3.5,
"code": "import numpy as np\nfrom scipy.linalg import fractional_matrix_power\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\nfor scaled in [False,True]:\n g=W[\"ln1_w.0\"].astype(float) if scaled else np.ones(384)\n As=[]\n for j in range(6):\n s=slice(64*j,64*(j+1))\n As.append((W[\"W_q.0\"][s]*g).T@(W[\"W_k.0\"][s]*g)/8)\n print(\"\\nScaled\" if scaled else \"\\nUnscaled\",\"QK kernel power comparisons\")\n ref=As[3]/np.linalg.norm(As[3],2)\n powers=[np.eye(384)]\n for z in range(8):\n powers.append(powers[-1]@ref)\n for j in range(6):\n print(\"L0H%d cos powers 0..8\"%j,np.round([\n np.sum(As[j]*P)/(np.linalg.norm(As[j])*np.linalg.norm(P)) for P in powers],3))\n q=W[\"W_q.0\"][192:256].astype(float)*g\n k=W[\"W_k.0\"][192:256].astype(float)*g\n _,s,P=np.linalg.svd(np.vstack([q,k]),full_matrices=False)\n for r in [16,24,32]:\n Z=P[:r].T\n # Whitening of a weight Gram matrix, not a network normalization.\n H=(Z.T@(q.T@q+k.T@k)@Z)/16\n ev,U=np.linalg.eigh(H)\n wh=U@np.diag(1/np.sqrt(ev))@U.T\n M=[wh@Z.T@A@Z@wh for A in As]\n u,sv,vt=np.linalg.svd(M[3])\n R=u@vt\n print(\"rank\",r,\"polar power comparison\")\n for j in [0,1,2,4,5]:\n vals=[]\n for lag in range(9):\n z=np.linalg.matrix_power(R,lag)\n vals.append(np.sum(M[j]*z)/(np.linalg.norm(M[j])*np.linalg.norm(z)))\n print(\"H\",j,np.round(vals,3))\n",
"out": "\nUnscaled QK kernel power comparisons\nL0H0 cos powers 0..8 [-0.06 -0.132 -0.041 0.097 0.162 0.122 0.023 -0.062 -0.09 ]\nL0H1 cos powers 0..8 [-0.046 -0.022 -0.005 0.006 0.006 -0.001 -0.006 -0.006 -0.001]\nL0H2 cos powers 0..8 [-0.048 -0.028 -0.005 0.005 0.005 -0. -0.003 -0.003 -0.001]\nL0H3 cos powers 0..8 [ 0.091 1. 0.523 -0.126 -0.505 -0.488 -0.211 0.106 0.286]\nL0H4 cos powers 0..8 [-0.044 -0.032 0.021 0.036 0.016 -0.01 -0.02 -0.013 0.001]\nL0H5 cos powers 0..8 [-0.012 -0.013 0.008 0.009 0.003 -0.005 -0.008 -0.006 -0.001]\nrank 16 polar power comparison\nH 0 [-0.398 -0.337 -0.009 0.348 0.492 0.372 0.131 -0.049 -0.089]\nH 1 [-0.073 -0.038 0.126 0.236 0.185 0.055 -0.014 0.005 0.058]\nH 2 [-0.202 -0.208 -0.127 -0.05 -0.049 -0.099 -0.138 -0.153 -0.159]\nH 4 [-0.323 -0.209 0.134 0.351 0.234 -0.051 -0.212 -0.161 -0.023]\nH 5 [-0.1 -0.037 0.046 0.061 -0.002 -0.072 -0.097 -0.079 -0.037]\nrank 24 polar power comparison\nH 0 [-0.314 -0.2 -0.022 0.18 0.182 0.113 -0.106 -0.131 -0.117]\nH 1 [-0.095 -0.064 -0.065 0.001 -0.022 0.032 0.001 0.064 -0.002]\nH 2 [-0.122 -0.095 -0.076 -0.019 0.031 0.065 0.022 0.046 -0.015]\nH 4 [-0.162 -0.05 0.1 0.136 0.08 -0.052 -0.163 -0.076 -0.013]\nH 5 [-0.149 -0.035 0.108 0.165 0.138 0.075 -0.044 -0.109 -0.124]\nrank 32 polar power comparison\nH 0 [-0.282 -0.155 -0.032 0.127 0.151 0.08 -0.002 -0.036 -0.023]\nH 1 [-0.174 -0.08 -0.043 0.022 0.038 -0.007 -0.017 -0.005 0.023]\nH 2 [-0.176 -0.114 -0.067 0. 0.054 0.02 -0.007 -0.002 0.014]\nH 4 [-0.165 -0.097 0.068 0.142 0.02 -0.052 -0.071 0.023 0.027]\nH 5 [-0.083 -0.062 0.077 0.1 0.095 -0.003 0.009 -0.066 -0.03 ]\n\nScaled QK kernel power comparisons\nL0H0 cos powers 0..8 [-0.018 0.137 0.111 -0.066 -0.29 -0.366 -0.206 0.099 0.368]\nL0H1 cos powers 0..8 [-0.032 -0.026 -0.013 -0.01 -0.001 0.013 0.021 0.016 0.001]\nL0H2 cos powers 0..8 [-0.042 -0.063 -0.019 0.048 0.103 0.099 0.031 -0.06 -0.121]\nL0H3 cos powers 0..8 [ 0.098 1. 0.595 0.018 -0.409 -0.535 -0.377 -0.056 0.27 ]\nL0H4 cos powers 0..8 [-0.026 -0.044 -0.016 -0.008 0.004 0.021 0.03 0.022 -0.001]\nL0H5 cos powers 0..8 [-0.013 0.01 0.061 0.067 0.027 -0.038 -0.083 -0.081 -0.032]\nrank 16 polar power comparison\nH 0 [-0.17 -0.052 0.038 0.031 -0.056 -0.131 -0.122 -0.045 0.014]\nH 1 [-0.013 -0.106 -0.126 -0.065 0.036 0.114 0.114 0.031 -0.078]\nH 2 [-0.168 -0.176 -0.084 0.046 0.129 0.116 0.022 -0.084 -0.135]\nH 4 [ 0.024 -0.095 -0.123 -0.061 0.011 0.058 0.093 0.097 0.035]\nH 5 [-0.146 -0.005 0.151 0.193 0.082 -0.092 -0.191 -0.153 -0.017]\nrank 24 polar power comparison\nH 0 [-0.256 -0.13 0.015 0.103 0.084 0.043 -0.032 -0.012 -0.003]\nH 1 [-0.062 -0.069 0.023 -0.069 -0.051 0.112 0.028 -0.002 -0.003]\nH 2 [-0.155 -0.099 0.004 -0.004 0.072 0.128 -0.014 -0.059 -0.083]\nH 4 [-0.075 -0.08 0.063 0.027 0.001 0.019 -0.053 -0.024 0.037]\nH 5 [-0.094 0.024 0.093 0.14 0.048 -0.054 -0.109 -0.075 -0.008]\nrank 32 polar power comparison\nH 0 [-0.126 0.047 -0.031 0.086 0.024 0.093 -0.041 0.04 -0.013]\nH 1 [-0.133 -0.004 -0.059 0.039 -0.073 0.045 -0.059 0.047 -0.031]\nH 2 [-0.16 0.028 -0.084 0.057 -0.086 0.08 -0.091 0.059 -0.061]\nH 4 [-0.104 -0.062 -0.021 0.008 -0.014 -0.02 -0.047 -0.013 0.005]\nH 5 [-0.131 -0.036 0.025 0.058 -0.05 0.018 -0.053 -0.008 -0.048]\n"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 8 of at most 100: ``` Unscaled QK kernel power comparisons L0H0 cos powers 0..8 [-0.06 -0.132 -0.041 0.097 0.162 0.122 0.023 -0.062 -0.09 ] L0H1 cos powers 0..8 [-0.046 -0.022 -0.005 0.006 0.006 -0.001 -0.006 -0.006 -0.001] L0H2 cos powers 0..8 [-0.048 -0.028 -0.005 0.005 0.005 -0. -0.003 -0.003 -0.001] L0H3 cos powers 0..8 [ 0.091 1. 0.523 -0.126 -0.505 -0.488 -0.211 0.106 0.286] L0H4 cos powers 0..8 [-0.044 -0.032 0.021 0.036 0.016 -0.01 -0.02 -0.013 0.001] L0H5 cos powers 0..8 [-0.012 -0.013 0.008 0.009 0.003 -0.005 -0.008 -0.006 -0.001] rank 16 polar power comparison H 0 [-0.398 -0.337 -0.009 0.348 0.492 0.372 0.131 -0.049 -0.089] H 1 [-0.073 -0.038 0.126 0.236 0.185 0.055 -0.014 0.005 0.058] H 2 [-0.202 -0.208 -0.127 -0.05 -0.049 -0.099 -0.138 -0.153 -0.159] H 4 [-0.323 -0.209 0.134 0.351 0.234 -0.051 -0.212 -0.161 -0.023] H 5 [-0.1 -0.037 0.046 0.061 -0.002 -0.072 -0.097 -0.079 -0.037] rank 24 polar power comparison H 0 [-0.314 -0.2 -0.022 0.18 0.182 0.113 -0.106 -0.131 -0.117] H 1 [-0.095 -0.064 -0.065 0.001 -0.022 0.032 0.001 0.064 -0.002] H 2 [-0.122 -0.095 -0.076 -0.019 0.031 0.065 0.022 0.046 -0.015] H 4 [-0.162 -0.05 0.1 0.136 0.08 -0.052 -0.163 -0.076 -0.013] H 5 [-0.149 -0.035 0.108 0.165 0.138 0.075 -0.044 -0.109 -0.124] rank 32 polar power comparison H 0 [-0.282 -0.155 -0.032 0.127 0.151 0.08 -0.002 -0.036 -0.023] H 1 [-0.174 -0.08 -0.043 0.022 0.038 -0.007 -0.017 -0.005 0.023] H 2 [-0.176 -0.114 -0.067 0. 0.054 0.02 -0.007 -0.002 0.014] H 4 [-0.165 -0.097 0.068 0.142 0.02 -0.052 -0.071 0.023 0.027] H 5 [-0.083 -0.062 0.077 0.1 0.095 -0.003 0.009 -0.066 -0.03 ] Scaled QK kernel power comparisons L0H0 cos powers 0..8 [-0.018 0.137 0.111 -0.066 -0.29 -0.366 -0.206 0.099 0.368] L0H1 cos powers 0..8 [-0.032 -0.026 -0.013 -0.01 -0.001 0.013 0.021 0.016 0.001] L0H2 cos powers 0..8 [-0.042 -0.063 -0.019 0.048 0.103 0.099 0.031 -0.06 -0.121] L0H3 cos powers 0..8 [ 0.098 1. 0.595 0.018 -0.409 -0.535 -0.377 -0.056 0.27 ] L0H4 cos powers 0..8 [-0.026 -0.044 -0.016 -0.008 0.004 0.021 0.03 0.022 -0.001] L0H5 cos powers 0..8 [-0.013 0.01 0.061 0.067 0.027 -0.038 -0.083 -0.081 -0.032] rank 16 polar power comparison H 0 [-0.17 -0.052 0.038 0.031 -0.056 -0.131 -0.122 -0.045 0.014] H 1 [-0.013 -0.106 -0.126 -0.065 0.036 0.114 0.114 0.031 -0.078] H 2 [-0.168 -0.176 -0.084 0.046 0.129 0.116 0.022 -0.084 -0.135] H 4 [ 0.024 -0.095 -0.123 -0.061 0.011 0.058 0.093 0.097 0.035] H 5 [-0.146 -0.005 0.151 0.193 0.082 -0.092 -0.191 -0.153 -0.017] rank 24 polar power comparison H 0 [-0.256 -0.13 0.015 0.103 0.084 0.043 -0.032 -0.012 -0.003] H 1 [-0.062 -0.069 0.023 -0.069 -0.051 0.112 0.028 -0.002 -0.003] H 2 [-0.155 -0.099 0.004 -0.004 0.072 0.128 -0.014 -0.059 -0.083] H 4 [-0.075 -0.08 0.063 0.027 0.001 0.019 -0.053 -0.024 0.037] H 5 [-0.094 0.024 0.093 0.14 0.048 -0.054 -0.109 -0.075 -0.008] rank 32 polar power comparison H 0 [-0.126 0.047 -0.031 0.086 0.024 0.093 -0.041 0.04 -0.013] H 1 [-0.133 -0.004 -0.059 0.039 -0.073 0.045 -0.059 0.047 -0.031] H 2 [-0.16 0.028 -0.084 0.057 -0.086 0.08 -0.091 0.059 -0.061] H 4 [-0.104 -0.062 -0.021 0.008 -0.014 -0.02 -0.047 -0.013 0.005] H 5 [-0.131 -0.036 0.025 0.058 -0.05 0.018 -0.053 -0.008 -0.048] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(4)
g0=W["ln1_w.0"].astype(float)
V0=W["W_v.0"][192:256].astype(float)*g0
T=np.linalg.svd(V0,full_matrices=False)[2][:20].T
B0=W["W_o.0"][:,192:256].astype(float)@V0
Uout=np.linalg.svd(W["W_proj.3"].astype(float),full_matrices=False)[0][:,:24]
print("Final heads: weight-only token-to-output and matching diagnostics")
for j in range(6):
s=slice(j*64,(j+1)*64)
g=W["ln1_w.3"].astype(float)
A=(W["W_q.3"][s]*g).T@(W["W_k.3"][s]*g)/8
B=W["W_o.3"][:,s].astype(float)@(W["W_v.3"][s]*g)
D=B@T
C=T.T@A@B0@T
print("L3H%d"%j,
"OV token-map norm %.3f output-subspace energy %.3f"%
(np.linalg.norm(D),np.linalg.norm(Uout.T@D)**2/np.linalg.norm(D)**2),
"matching trace/norm %.3f"%(np.trace(C)/np.linalg.norm(C)),
"matching eig real",np.round(np.sort(np.linalg.eigvals(C).real)[-6:],4))
print("\nCoordinate concentration of L0 OV output (rows 21,39,58,99,184)")
ix=[21,39,58,99,184]
for j in range(6):
s=slice(j*64,(j+1)*64)
B=W["W_o.0"][:,s].astype(float)@(W["W_v.0"][s]*g0)
print("L0H%d"%j,round(np.sum(B[ix]**2)/np.sum(B**2),4))
print("\nInduction matching control: signed/permuted head-output coordinates")
rng=np.random.default_rng(1022)
s=slice(192,256);g=W["ln1_w.3"].astype(float)
A=(W["W_q.3"][s]*g).T@(W["W_k.3"][s]*g)/8
left=T.T@A
right=B0@T
actual=left@right
null=[]
for _ in range(1000):
perm=rng.permutation(384)
sign=rng.choice([-1.,1.],384)
C=(left*sign)@right[perm]
null.append(np.trace(C)/np.linalg.norm(C))
print("observed",np.trace(actual)/np.linalg.norm(actual),
"null mean/std",np.mean(null),np.std(null),
"null 1/99 percentiles",np.percentile(null,[1,99]),
"null max",max(null))
```
{
"t": 1792713720.8475566,
"s": 1.38,
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(4)\n\ng0=W[\"ln1_w.0\"].astype(float)\nV0=W[\"W_v.0\"][192:256].astype(float)*g0\nT=np.linalg.svd(V0,full_matrices=False)[2][:20].T\nB0=W[\"W_o.0\"][:,192:256].astype(float)@V0\nUout=np.linalg.svd(W[\"W_proj.3\"].astype(float),full_matrices=False)[0][:,:24]\n\nprint(\"Final heads: weight-only token-to-output and matching diagnostics\")\nfor j in range(6):\n s=slice(j*64,(j+1)*64)\n g=W[\"ln1_w.3\"].astype(float)\n A=(W[\"W_q.3\"][s]*g).T@(W[\"W_k.3\"][s]*g)/8\n B=W[\"W_o.3\"][:,s].astype(float)@(W[\"W_v.3\"][s]*g)\n D=B@T\n C=T.T@A@B0@T\n print(\"L3H%d\"%j,\n \"OV token-map norm %.3f output-subspace energy %.3f\"%\n (np.linalg.norm(D),np.linalg.norm(Uout.T@D)**2/np.linalg.norm(D)**2),\n \"matching trace/norm %.3f\"%(np.trace(C)/np.linalg.norm(C)),\n \"matching eig real\",np.round(np.sort(np.linalg.eigvals(C).real)[-6:],4))\n\nprint(\"\\nCoordinate concentration of L0 OV output (rows 21,39,58,99,184)\")\nix=[21,39,58,99,184]\nfor j in range(6):\n s=slice(j*64,(j+1)*64)\n B=W[\"W_o.0\"][:,s].astype(float)@(W[\"W_v.0\"][s]*g0)\n print(\"L0H%d\"%j,round(np.sum(B[ix]**2)/np.sum(B**2),4))\n\nprint(\"\\nInduction matching control: signed/permuted head-output coordinates\")\nrng=np.random.default_rng(1022)\ns=slice(192,256);g=W[\"ln1_w.3\"].astype(float)\nA=(W[\"W_q.3\"][s]*g).T@(W[\"W_k.3\"][s]*g)/8\nleft=T.T@A\nright=B0@T\nactual=left@right\nnull=[]\nfor _ in range(1000):\n perm=rng.permutation(384)\n sign=rng.choice([-1.,1.],384)\n C=(left*sign)@right[perm]\n null.append(np.trace(C)/np.linalg.norm(C))\nprint(\"observed\",np.trace(actual)/np.linalg.norm(actual),\n \"null mean/std\",np.mean(null),np.std(null),\n \"null 1/99 percentiles\",np.percentile(null,[1,99]),\n \"null max\",max(null))\n",
"out": "Final heads: weight-only token-to-output and matching diagnostics\nL3H0 OV token-map norm 4.999 output-subspace energy 0.087 matching trace/norm -0.629 matching eig real [0.0012 0.0015 0.0015 0.0071 0.0071 0.0183]\nL3H1 OV token-map norm 6.302 output-subspace energy 0.098 matching trace/norm -0.222 matching eig real [0.0035 0.0035 0.0072 0.0072 0.0074 0.0074]\nL3H2 OV token-map norm 5.569 output-subspace energy 0.092 matching trace/norm 0.193 matching eig real [0.0041 0.0041 0.0074 0.0094 0.0094 0.0117]\nL3H3 OV token-map norm 13.422 output-subspace energy 0.348 matching trace/norm 2.860 matching eig real [0.0375 0.0382 0.0382 0.0386 0.0386 0.0533]\nL3H4 OV token-map norm 10.491 output-subspace energy 0.119 matching trace/norm 0.662 matching eig real [0.0025 0.0025 0.0029 0.0029 0.0038 0.0064]\nL3H5 OV token-map norm 5.152 output-subspace energy 0.077 matching trace/norm -0.132 matching eig real [0.0042 0.0042 0.008 0.008 0.0107 0.0107]\n\nCoordinate concentration of L0 OV output (rows 21,39,58,99,184)\nL0H0 0.0344\nL0H1 0.9395\nL0H2 0.942\nL0H3 0.0109\nL0H4 0.8777\nL0H5 0.97\n\nInduction matching control: signed/permuted head-output coordinates\nobserved 2.8602269991267235 null mean/std -0.0002919724439507183 0.23419073556467282 null 1/99 percentiles [-0.5479 0.5497] null max 0.6820147810277253\n"
}Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 9 of at most 100: ``` Final heads: weight-only token-to-output and matching diagnostics L3H0 OV token-map norm 4.999 output-subspace energy 0.087 matching trace/norm -0.629 matching eig real [0.0012 0.0015 0.0015 0.0071 0.0071 0.0183] L3H1 OV token-map norm 6.302 output-subspace energy 0.098 matching trace/norm -0.222 matching eig real [0.0035 0.0035 0.0072 0.0072 0.0074 0.0074] L3H2 OV token-map norm 5.569 output-subspace energy 0.092 matching trace/norm 0.193 matching eig real [0.0041 0.0041 0.0074 0.0094 0.0094 0.0117] L3H3 OV token-map norm 13.422 output-subspace energy 0.348 matching trace/norm 2.860 matching eig real [0.0375 0.0382 0.0382 0.0386 0.0386 0.0533] L3H4 OV token-map norm 10.491 output-subspace energy 0.119 matching trace/norm 0.662 matching eig real [0.0025 0.0025 0.0029 0.0029 0.0038 0.0064] L3H5 OV token-map norm 5.152 output-subspace energy 0.077 matching trace/norm -0.132 matching eig real [0.0042 0.0042 0.008 0.008 0.0107 0.0107] Coordinate concentration of L0 OV output (rows 21,39,58,99,184) L0H0 0.0344 L0H1 0.9395 L0H2 0.942 L0H3 0.0109 L0H4 0.8777 L0H5 0.97 Induction matching control: signed/permuted head-output coordinates observed 2.8602269991267235 null mean/std -0.0002919724439507183 0.23419073556467282 null 1/99 percentiles [-0.5479 0.5497] null max 0.6820147810277253 ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
## Observations:
All measurements below are **weight-only**, not forward passes. Head labels are hypotheses; attention distances cannot be verified without the missing embeddings and activations.
For compactness, define
\[
A=G W_q^\top W_kG/8,\qquad B=W_oW_vG,\qquad
r=G W_k^\top W_q\,\mathrm{ln1_b}/8,
\]
where \(G=\operatorname{diag}(\mathrm{ln1_w})\), using each head’s slices. Thus \(A\) is its bilinear score matrix, \(B\) its value/output product, and \(r\) a key-dependent affine coefficient.
- **L0H3 stands out as a directional positional head.** Its \(W_q,W_k\) norms are **14.141, 14.661**, substantially above the other L0 heads. Its unscaled QK product has complex eigenvalues including **\(0.724\pm0.691i\)** and **\(0.446\pm0.842i\)**. The leading 16-dimensional Q/K subspaces overlap strongly (**0.620**), whereas Q/V overlap is only **0.0177**. This supports position-controlled transport of different, likely character-related information.
- **Several L0 heads construct shared low-dimensional signals.** Coordinates **21,39,58,99,184** contain **94.0%, 94.2%, 87.8%, 97.0%** of \(B\)’s squared norm for H1,H2,H4,H5. These coordinates have no intrinsic semantic labels.
- **Later heads strongly read those shared signals.** For example, `ln1_b.1[39]=-1.5676`, `[58]=1.3651`, and `[99]=-1.3355`. All six L1 heads have positive \(r_{21}\), ranging from **2.008 to 3.627**. This is consistent with a shared positional/recency mechanism, although it does not establish its meaning conclusively.
- **L3H3 has a distinctive induction-compatible connection to L0H3.** Let \(T\) contain the leading 20 right-singular directions of L0H3’s scaled value matrix. For
\[
C=T^\top A_{3,3}B_{0,3}T,
\]
\(\operatorname{tr}(C)/\|C\|_F=\mathbf{2.860}\). Signed/permuted-coordinate controls had standard deviation **0.234** and maximum **0.682** across 1,000 trials. This is strong evidence of learned matching through the earlier transport circuit.
- **The corpus-specific evidence is weaker.** The leading 24 singular directions of `W_proj.3` contain **92.14%** of its squared norm; directions 25–28 add only **1.62%**. This is compatible with a comparatively restricted frequently used alphabet, but is **not** a recovered character count.
## Heads:
The “local” assignments below largely depend on the tentative recency interpretation of the shared channel.
L0H0: **local** — Its positional-subspace score matrix resembles a higher power of L0H3’s directional factor more than the first power; compatible with a longer short offset.
L0H1: **unclear** — Almost rank-two score/output structure; 94.0% of output-product energy enters the shared coordinates. Boundary selection and positional bookkeeping remain alternatives.
L0H2: **unclear** — One score singular mode carries 90.8% of energy; 94.2% of output energy enters the shared coordinates. Distance is unresolved.
L0H3: **previous** — Strongest predecessor candidate: high-rank directional QK structure, separated positional and value subspaces, and the downstream induction-compatible connection.
L0H4: **unclear** — Dominant rank-one score component and 87.8% shared-coordinate output concentration; insufficient evidence for a particular attention distance.
L0H5: **broad** — Very weak score product, with leading singular value **0.032**, but strong low-rank value transport (**16.282, 8.786**); consistent with context aggregation.
L1H0: **local** — Strong shared-channel preference: \(r_{21}=3.627\); key column 21 contains 30.1% of score-product energy.
L1H1: **local** — \(r_{21}=3.352\), with 32.5% key-column-21 energy; closely resembles the other L1 recency candidates.
L1H2: **local** — \(r_{21}=2.769\), with 36.4% key-column-21 energy.
L1H3: **local** — \(r_{21}=3.079\), with 36.4% key-column-21 energy.
L1H4: **local** — \(r_{21}=2.008\), with 34.8% key-column-21 energy; weaker shared preference than most L1 heads.
L1H5: **local** — \(r_{21}=3.563\), with 31.9% key-column-21 energy.
L2H0: **local** — Retains the shared mechanism strongly: \(r_{21}=1.815\), with 15.4% key-column-21 energy.
L2H1: **local** — \(r_{21}=1.682\); substantial shared-channel preference, but more distributed content-sensitive score structure.
L2H2: **local** — \(r_{21}=1.411\); compatible with content-conditioned local retrieval.
L2H3: **local** — Lower-confidence assignment: the shared preference remains, \(r_{21}=0.803\), but is weaker.
L2H4: **local** — Strongest L2 shared-channel preference, \(r_{21}=2.069\); supports recency-sensitive retrieval.
L2H5: **local** — \(r_{21}=1.859\), without a distinctive positive L0H3 matching composition.
L3H0: **local** — Residual shared-channel preference, \(r_{21}=0.408\); the tested predecessor-matching composition is negative, unlike L3H3.
L3H1: **unclear** — Weaker shared-channel preference, \(r_{21}=0.181\), and no strong tested induction signature; local versus broader retrieval remains unresolved.
L3H2: **local** — \(r_{21}=0.382\); retains the common positional mechanism without L3H3’s matching signature.
L3H3: **copy** — Strong positive matching through L0H3: projected composition trace/norm **2.860**, far beyond the coordinate-randomization controls.
L3H4: **broad** — Little shared-channel preference, \(r_{21}=0.028\), combined with large value/output singular values **21.271, 17.680**; favors an aggregation interpretation.
L3H5: **local** — \(r_{21}=0.416\), with no comparable positive predecessor-matching signature.
## Reasoning:
**Intact order is strongly favored.** A directional character-transport circuit followed by an induction-compatible matching circuit is useful for ordered sequences and repetition. Randomly shuffling each line largely removes those predictive relationships, although it preserves line-level composition and boundaries.
**Romaji is my best, less-confident corpus judgment.** The comparatively concentrated character-related geometry and substantial short-context machinery fit a restricted, regular romanized orthography. English remains a serious alternative; the spectra do not identify letters, vowels, words, or language directly. I find less positive evidence specifically favoring Python.
corpus: romaji
order: as is
probabilities: python=0.11 romaji=0.62 english=0.27; as is=0.99 shuffled=0.01## Observations:
All measurements below are **weight-only**, not forward passes. Head labels are hypotheses; attention distances cannot be verified without the missing embeddings and activations.
For compactness, define
\[
A=G W_q^\top W_kG/8,\qquad B=W_oW_vG,\qquad
r=G W_k^\top W_q\,\mathrm{ln1_b}/8,
\]
where \(G=\operatorname{diag}(\mathrm{ln1_w})\), using each head’s slices. Thus \(A\) is its bilinear score matrix, \(B\) its value/output product, and \(r\) a key-dependent affine coefficient.
- **L0H3 stands out as a directional positional head.** Its \(W_q,W_k\) norms are **14.141, 14.661**, substantially above the other L0 heads. Its unscaled QK product has complex eigenvalues including **\(0.724\pm0.691i\)** and **\(0.446\pm0.842i\)**. The leading 16-dimensional Q/K subspaces overlap strongly (**0.620**), whereas Q/V overlap is only **0.0177**. This supports position-controlled transport of different, likely character-related information.
- **Several L0 heads construct shared low-dimensional signals.** Coordinates **21,39,58,99,184** contain **94.0%, 94.2%, 87.8%, 97.0%** of \(B\)’s squared norm for H1,H2,H4,H5. These coordinates have no intrinsic semantic labels.
- **Later heads strongly read those shared signals.** For example, `ln1_b.1[39]=-1.5676`, `[58]=1.3651`, and `[99]=-1.3355`. All six L1 heads have positive \(r_{21}\), ranging from **2.008 to 3.627**. This is consistent with a shared positional/recency mechanism, although it does not establish its meaning conclusively.
- **L3H3 has a distinctive induction-compatible connection to L0H3.** Let \(T\) contain the leading 20 right-singular directions of L0H3’s scaled value matrix. For
\[
C=T^\top A_{3,3}B_{0,3}T,
\]
\(\operatorname{tr}(C)/\|C\|_F=\mathbf{2.860}\). Signed/permuted-coordinate controls had standard deviation **0.234** and maximum **0.682** across 1,000 trials. This is strong evidence of learned matching through the earlier transport circuit.
- **The corpus-specific evidence is weaker.** The leading 24 singular directions of `W_proj.3` contain **92.14%** of its squared norm; directions 25–28 add only **1.62%**. This is compatible with a comparatively restricted frequently used alphabet, but is **not** a recovered character count.
## Heads:
The “local” assignments below largely depend on the tentative recency interpretation of the shared channel.
L0H0: **local** — Its positional-subspace score matrix resembles a higher power of L0H3’s directional factor more than the first power; compatible with a longer short offset.
L0H1: **unclear** — Almost rank-two score/output structure; 94.0% of output-product energy enters the shared coordinates. Boundary selection and positional bookkeeping remain alternatives.
L0H2: **unclear** — One score singular mode carries 90.8% of energy; 94.2% of output energy enters the shared coordinates. Distance is unresolved.
L0H3: **previous** — Strongest predecessor candidate: high-rank directional QK structure, separated positional and value subspaces, and the downstream induction-compatible connection.
L0H4: **unclear** — Dominant rank-one score component and 87.8% shared-coordinate output concentration; insufficient evidence for a particular attention distance.
L0H5: **broad** — Very weak score product, with leading singular value **0.032**, but strong low-rank value transport (**16.282, 8.786**); consistent with context aggregation.
L1H0: **local** — Strong shared-channel preference: \(r_{21}=3.627\); key column 21 contains 30.1% of score-product energy.
L1H1: **local** — \(r_{21}=3.352\), with 32.5% key-column-21 energy; closely resembles the other L1 recency candidates.
L1H2: **local** — \(r_{21}=2.769\), with 36.4% key-column-21 energy.
L1H3: **local** — \(r_{21}=3.079\), with 36.4% key-column-21 energy.
L1H4: **local** — \(r_{21}=2.008\), with 34.8% key-column-21 energy; weaker shared preference than most L1 heads.
L1H5: **local** — \(r_{21}=3.563\), with 31.9% key-column-21 energy.
L2H0: **local** — Retains the shared mechanism strongly: \(r_{21}=1.815\), with 15.4% key-column-21 energy.
L2H1: **local** — \(r_{21}=1.682\); substantial shared-channel preference, but more distributed content-sensitive score structure.
L2H2: **local** — \(r_{21}=1.411\); compatible with content-conditioned local retrieval.
L2H3: **local** — Lower-confidence assignment: the shared preference remains, \(r_{21}=0.803\), but is weaker.
L2H4: **local** — Strongest L2 shared-channel preference, \(r_{21}=2.069\); supports recency-sensitive retrieval.
L2H5: **local** — \(r_{21}=1.859\), without a distinctive positive L0H3 matching composition.
L3H0: **local** — Residual shared-channel preference, \(r_{21}=0.408\); the tested predecessor-matching composition is negative, unlike L3H3.
L3H1: **unclear** — Weaker shared-channel preference, \(r_{21}=0.181\), and no strong tested induction signature; local versus broader retrieval remains unresolved.
L3H2: **local** — \(r_{21}=0.382\); retains the common positional mechanism without L3H3’s matching signature.
L3H3: **copy** — Strong positive matching through L0H3: projected composition trace/norm **2.860**, far beyond the coordinate-randomization controls.
L3H4: **broad** — Little shared-channel preference, \(r_{21}=0.028\), combined with large value/output singular values **21.271, 17.680**; favors an aggregation interpretation.
L3H5: **local** — \(r_{21}=0.416\), with no comparable positive predecessor-matching signature.
## Reasoning:
**Intact order is strongly favored.** A directional character-transport circuit followed by an induction-compatible matching circuit is useful for ordered sequences and repetition. Randomly shuffling each line largely removes those predictive relationships, although it preserves line-level composition and boundaries.
**Romaji is my best, less-confident corpus judgment.** The comparatively concentrated character-related geometry and substantial short-context machinery fit a restricted, regular romanized orthography. English remains a serious alternative; the spectra do not identify letters, vowels, words, or language directly. I find less positive evidence specifically favoring Python.
corpus: romaji
order: as is
probabilities: python=0.11 romaji=0.62 english=0.27; as is=0.99 shuffled=0.01