Public export: private machine paths and host identifiers are replaced with placeholders. Recorded results and numerical weights are unchanged. Embedded source hashes identify the original files; PUBLICATION-MANIFEST.json identifies these public copies.
Built-in Codex tools were disabled. Harness execution is separately permitted only in Experiments 3 and 6. Text below is taken from saved records; it is not a new model run. Reasoning blocks are the summaries actually emitted by the CLI.
Answer the user's request.
Recorded prompt · Raw CLI events · Settings and audit
Below is a small character-level language model: a 4-layer transformer with 6 attention heads per layer, which reads up to 128 characters and outputs scores (logits) for the next character. This is exactly how it computes (numpy):
```python
import numpy as np
VOCAB = "\n " + "abcdefghijklmnopqrstuvwxyz" # index 0 = newline, 1 = space, 2..27 = a..z
HEADS, LAYERS = 6, 4
def layer_norm(x, w, b): # over the last axis, eps 1e-5
mu = x.mean(-1, keepdims=True)
var = ((x - mu) ** 2).mean(-1, keepdims=True)
return (x - mu) / np.sqrt(var + 1e-5) * w + b
def next_char_logits(text, p):
"""text: up to 128 characters (only the last 128 are used); p: dict of the weight arrays.
Returns the 28 logits for the character that comes next."""
ids = [VOCAB.index(c) for c in text[-128:]]
n, d = len(ids), p["tok_emb"].shape[1]
hd = d // HEADS
x = p["tok_emb"][ids] + p["pos_emb"][:n] # (n, d); positions 0..n-1 from the start of the text
for l in range(LAYERS): # layer l uses the tensors named <name>.<l>
h = layer_norm(x, p[f"ln1_w.{l}"], p[f"ln1_b.{l}"])
q, k, v = h @ p[f"W_q.{l}"].T, h @ p[f"W_k.{l}"].T, h @ p[f"W_v.{l}"].T
out = np.zeros((n, d))
for j in range(HEADS): # head j uses dimensions j*hd .. (j+1)*hd - 1
s = slice(j * hd, (j + 1) * hd)
att = q[:, s] @ k[:, s].T / np.sqrt(hd) # (n, n)
att = att + np.triu(np.full((n, n), -np.inf), 1) # causal: position i sees positions 0..i
att = np.exp(att - att.max(-1, keepdims=True))
att = att / att.sum(-1, keepdims=True)
out[:, s] = att @ v[:, s]
x = x + out @ p[f"W_o.{l}"].T
h = layer_norm(x, p[f"ln2_w.{l}"], p[f"ln2_b.{l}"])
x = x + np.maximum(0.0, h @ p[f"W_fc.{l}"].T) @ p[f"W_proj.{l}"].T # ReLU MLP, hidden 4 d
h = layer_norm(x, p["lnf_w"], p["lnf_b"])
return h[-1] @ p["W_out"].T + p["b_out"] # logits of the next character (28)
```
Training: from random initialization (every matrix entry drawn from a normal distribution with standard deviation 0.02, or 0.007071 for W_o and W_proj; LayerNorm weights 1 and biases 0; b_out 0), next-character prediction with AdamW (30,517 steps of 256 random 128-character windows, 1,000,000,000 characters in total; learning-rate warm-up then cosine decay; weight decay 0.01 on the matrices).
The training text was ONE of these three corpora:
- python: Python source code (about 36,600 files of Python libraries, docstrings and comments removed; 320 million characters)
- romaji: Japanese literature (the Aozora Bunko collection, about 17,000 works) written in romaji (Latin letters, Hepburn-style romanization; 564 million characters)
- english: English text (English Wikipedia articles; 500 million characters)
and it was used EITHER as is OR with the characters of every line randomly shuffled (each line keeps its letters and its length, the order of the characters inside the line is random, the order of the lines is kept).
Normalization (the same for all three corpora, before anything else): lowercased; in each line the leading spaces (indentation) were kept, every other character that is not a letter a-z became a space, runs of spaces were collapsed to one and trailing spaces removed. So the text contains only newlines, spaces and the letters a-z (the 28 characters of VOCAB).
WHAT YOU CAN USE. The weights inside the four transformer blocks, exactly as trained (float32): for layer l = 0, 1, 2 and 3, ln1_w.l, ln1_b.l, W_q.l, W_k.l, W_v.l, W_o.l (attention) and ln2_w.l, ln2_b.l, W_fc.l, W_proj.l (MLP). The input layer (tok_emb, the 28 x d character embeddings, and pos_emb, the 128 x d position embeddings) and the output layer (lnf_w, lnf_b, W_out, b_out) are NOT available. The d = 384 dimensions of the residual stream carry no labels. Head j of layer l uses rows 64j .. 64j+63 of W_q.l, W_k.l and W_v.l and columns 64j .. 64j+63 of W_o.l.
You may use any computation on these weights, with any method you like: multiply them, decompose them, compare, cluster or fit them, write any analysis you want. Two things are NOT allowed:
1. A forward pass: do not run the network or any of its layers (LayerNorm, attention with its softmax, the MLP with its ReLU) on any input, whether real, random or constructed, in code, in your head or on paper. Treating the weight matrices as linear-algebra objects (products, eigenvalues, singular vectors, applying one weight matrix to directions taken from other weights) is fine; simulating what the network does to an input is not.
2. Search: do not look anything up (there is no internet access in any case).
HOW YOU COMPUTE. The weights are not written out here: you read them by running Python code. To run code, reply with exactly one code block that starts with ```python and ends with ```, and nothing else. We run it and reply with its printed output. Available: numpy, scipy, scikit-learn, PyTorch (CPU), pandas, sympy, networkx. Before your code runs, numpy is imported as np and W is a dict of the weight arrays by name: W['W_q.0'] is 384 x 384, W['W_fc.0'] is 1536 x 384, W['W_proj.0'] is 384 x 1536, W['ln1_w.0'] has 384 entries, and so on; matrices are (out, in), as used in the code above. There is nothing else to read (no other files, no network). Each run is a fresh process (nothing is kept between runs), may take up to 600 seconds, and its output is cut at 20,000 characters. You may run code up to 100 times. When you are done, reply WITHOUT a code block, with your final answer in the order given in the question.
QUESTION. What was this network trained on, and how does it use the context? Answer in this order:
Observations: <what you see in the shown weights and what it suggests; quote the weights you rely on>
Heads: <one line per head, L0H0 .. L0H5, L1H0 .. L1H5, L2H0 .. L2H5, L3H0 .. L3H5: what it most likely does, as one of: 'previous' (attends to the previous character), 'local' (a few characters back), 'broad' (spread far back), 'copy' (induction: attends to the character right after an earlier occurrence of the current character or context, so the network can repeat earlier text), or 'unclear'; then your evidence>
Reasoning: <which training text fits these observations, and why>
corpus: <python | romaji | english>
order: <as is | shuffled>
probabilities: python=<p> romaji=<p> english=<p>; as is=<p> shuffled=<p>
Equal probabilities are not allowed. Even if the evidence is weak, give your best judgement.Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svdvals
from threadpoolctl import threadpool_limits
np.set_printoptions(precision=4, suppress=True)
threadpool_limits(limits=2)
def spectrum(a):
s = svdvals(np.asarray(a, dtype=np.float64), check_finite=False)
e = s*s
return s, e.sum()/max(e[0], 1e-30), np.searchsorted(np.cumsum(e), .9*e.sum())+1
print("WEIGHT KEYS:", sorted(W))
for l in range(4):
print("\nLAYER", l)
for name in ["ln1_w", "ln1_b", "ln2_w", "ln2_b"]:
a = W[f"{name}.{l}"]
print(name, "min/mean/max/std:", np.min(a), np.mean(a), np.max(a), np.std(a),
"norm:", np.linalg.norm(a))
for name in ["W_q", "W_k", "W_v", "W_o", "W_fc", "W_proj"]:
a = W[f"{name}.{l}"]
s, sr, r90 = spectrum(a)
print(name, "rms %.5f topSV %s stableRank %.2f r90 %d" %
(np.sqrt(np.mean(a*a)), np.array2string(s[:5], precision=3), sr, r90))
for j in range(6):
sl = slice(64*j, 64*(j+1))
q = W[f"W_q.{l}"][sl].astype(float)
k = W[f"W_k.{l}"][sl].astype(float)
v = W[f"W_v.{l}"][sl].astype(float)
o = W[f"W_o.{l}"][:, sl].astype(float)
a = q.T @ k / 8
b = o @ v
sa, ra, r90a = spectrum(a)
sb, rb, r90b = spectrum(b)
asym = np.sum(a*a.T)/max(np.sum(a*a), 1e-30)
bsym = np.sum(b*b.T)/max(np.sum(b*b), 1e-30)
print("L%dH%d QK: norm %.3f top %s sr %.2f r90 %d symmetry %.3f; "
"OV: norm %.3f top %s sr %.2f r90 %d symmetry %.3f trace %.3f" %
(l, j, np.linalg.norm(a), np.array2string(sa[:5], precision=3),
ra, r90a, asym, np.linalg.norm(b), np.array2string(sb[:5], precision=3),
rb, r90b, bsym, np.trace(b)))
```
{
"run": 1,
"turn": "t001",
"code": "import numpy as np\nfrom scipy.linalg import svdvals\nfrom threadpoolctl import threadpool_limits\n\nnp.set_printoptions(precision=4, suppress=True)\nthreadpool_limits(limits=2)\n\ndef spectrum(a):\n s = svdvals(np.asarray(a, dtype=np.float64), check_finite=False)\n e = s*s\n return s, e.sum()/max(e[0], 1e-30), np.searchsorted(np.cumsum(e), .9*e.sum())+1\n\nprint(\"WEIGHT KEYS:\", sorted(W))\nfor l in range(4):\n print(\"\\nLAYER\", l)\n for name in [\"ln1_w\", \"ln1_b\", \"ln2_w\", \"ln2_b\"]:\n a = W[f\"{name}.{l}\"]\n print(name, \"min/mean/max/std:\", np.min(a), np.mean(a), np.max(a), np.std(a),\n \"norm:\", np.linalg.norm(a))\n for name in [\"W_q\", \"W_k\", \"W_v\", \"W_o\", \"W_fc\", \"W_proj\"]:\n a = W[f\"{name}.{l}\"]\n s, sr, r90 = spectrum(a)\n print(name, \"rms %.5f topSV %s stableRank %.2f r90 %d\" %\n (np.sqrt(np.mean(a*a)), np.array2string(s[:5], precision=3), sr, r90))\n for j in range(6):\n sl = slice(64*j, 64*(j+1))\n q = W[f\"W_q.{l}\"][sl].astype(float)\n k = W[f\"W_k.{l}\"][sl].astype(float)\n v = W[f\"W_v.{l}\"][sl].astype(float)\n o = W[f\"W_o.{l}\"][:, sl].astype(float)\n a = q.T @ k / 8\n b = o @ v\n sa, ra, r90a = spectrum(a)\n sb, rb, r90b = spectrum(b)\n asym = np.sum(a*a.T)/max(np.sum(a*a), 1e-30)\n bsym = np.sum(b*b.T)/max(np.sum(b*b), 1e-30)\n print(\"L%dH%d QK: norm %.3f top %s sr %.2f r90 %d symmetry %.3f; \"\n \"OV: norm %.3f top %s sr %.2f r90 %d symmetry %.3f trace %.3f\" %\n (l, j, np.linalg.norm(a), np.array2string(sa[:5], precision=3),\n ra, r90a, asym, np.linalg.norm(b), np.array2string(sb[:5], precision=3),\n rb, r90b, bsym, np.trace(b)))",
"out": "WEIGHT KEYS: ['W_fc.0', 'W_fc.1', 'W_fc.2', 'W_fc.3', 'W_k.0', 'W_k.1', 'W_k.2', 'W_k.3', 'W_o.0', 'W_o.1', 'W_o.2', 'W_o.3', 'W_proj.0', 'W_proj.1', 'W_proj.2', 'W_proj.3', 'W_q.0', 'W_q.1', 'W_q.2', 'W_q.3', 'W_v.0', 'W_v.1', 'W_v.2', 'W_v.3', 'ln1_b.0', 'ln1_b.1', 'ln1_b.2', 'ln1_b.3', 'ln1_w.0', 'ln1_w.1', 'ln1_w.2', 'ln1_w.3', 'ln2_b.0', 'ln2_b.1', 'ln2_b.2', 'ln2_b.3', 'ln2_w.0', 'ln2_w.1', 'ln2_w.2', 'ln2_w.3']\n\nLAYER 0\nln1_w min/mean/max/std: 0.01654764 0.26770642 1.0134335 0.21465456 norm: 6.724094\nln1_b min/mean/max/std: -0.12149114 0.00065349764 0.14177816 0.036595043 norm: 0.7172278\nln2_w min/mean/max/std: 0.04681267 0.8168376 1.6411812 0.33119047 norm: 17.27234\nln2_b min/mean/max/std: -0.55371493 0.017958164 0.16627714 0.066667244 norm: 1.3529724\nW_q rms 0.05912 topSV [5.039 4.999 4.703 4.377 4.343] stableRank 20.30 r90 61\nW_k rms 0.06045 topSV [6.204 5.052 4.572 4.537 4.391] stableRank 14.00 r90 60\nW_v rms 0.04739 topSV [4.553 4.024 3.738 3.497 3.465] stableRank 15.97 r90 66\nW_o rms 0.05104 topSV [6.678 5.237 2.835 2.679 2.62 ] stableRank 8.61 r90 75\nW_fc rms 0.06725 topSV [18.11 12.28 8.473 7.902 7.779] stableRank 8.13 r90 87\nW_proj rms 0.05964 topSV [10.932 7.576 6.628 6.293 6.166] stableRank 17.56 r90 135\nL0H0 QK: norm 0.807 top [0.649 0.424 0.106 0.082 0.064] sr 1.54 r90 2 symmetry 0.155; OV: norm 8.667 top [6.898 3.897 1.936 1.638 1.469] sr 1.58 r90 4 symmetry 0.049 trace 1.620\nL0H1 QK: norm 4.067 top [2.153 1.515 1.437 1.216 1.074] sr 3.57 r90 14 symmetry 0.164; OV: norm 17.067 top [4.703 4.573 4.424 4.248 4.199] sr 13.17 r90 20 symmetry 0.004 trace -0.820\nL0H2 QK: norm 1.136 top [0.561 0.447 0.374 0.356 0.324] sr 4.10 r90 11 symmetry 0.176; OV: norm 8.592 top [4.786 3.621 3.514 2.916 2.224] sr 3.22 r90 7 symmetry 0.003 trace -0.743\nL0H3 QK: norm 1.434 top [0.959 0.701 0.431 0.294 0.279] sr 2.23 r90 7 symmetry 0.085; OV: norm 5.288 top [2.168 1.997 1.875 1.555 1.484] sr 5.95 r90 12 symmetry 0.007 trace -0.071\nL0H4 QK: norm 0.097 top [0.04 0.03 0.024 0.018 0.017] sr 5.86 r90 41 symmetry 0.039; OV: norm 25.795 top [18.815 15.373 5.587 3.989 2.502] sr 1.88 r90 3 symmetry 0.039 trace 6.681\nL0H5 QK: norm 5.785 top [1.76 1.72 1.649 1.558 1.543] sr 10.81 r90 18 symmetry -0.228; OV: norm 18.558 top [5.61 5.503 5.123 4.867 4.799] sr 10.94 r90 17 symmetry 0.008 trace -11.398\n\nLAYER 1\nln1_w min/mean/max/std: 0.19793172 0.4753575 1.2095798 0.09638874 norm: 9.504637\nln1_b min/mean/max/std: -1.0504974 0.022033103 0.9102453 0.15318407 norm: 3.0326743\nln2_w min/mean/max/std: 0.0139104035 1.0110891 1.470659 0.15819876 norm: 20.054274\nln2_b min/mean/max/std: -0.37441117 0.02753053 0.36524463 0.14420813 norm: 2.876926\nW_q rms 0.08249 topSV [8.984 7.137 6.079 5.874 5.19 ] stableRank 12.43 r90 66\nW_k rms 0.07448 topSV [10.967 9.448 5.438 4.842 4.539] stableRank 6.80 r90 68\nW_v rms 0.07414 topSV [5.35 5.064 4.979 4.819 4.69 ] stableRank 28.31 r90 84\nW_o rms 0.06810 topSV [4.19 3.651 3.385 3.352 3.24 ] stableRank 38.95 r90 119\nW_fc rms 0.10451 topSV [20.796 14.143 13.826 13.428 13.122] stableRank 14.90 r90 93\nW_proj rms 0.09781 topSV [7.625 7.359 6.908 6.489 6.344] stableRank 97.06 r90 280\nL1H0 QK: norm 2.471 top [1.116 0.861 0.756 0.719 0.664] sr 4.90 r90 15 symmetry 0.042; OV: norm 27.504 top [9.927 9.271 8.872 7.793 7.506] sr 7.68 r90 16 symmetry 0.087 trace -34.568\nL1H1 QK: norm 4.754 top [2.392 1.846 1.725 1.378 1.068] sr 3.95 r90 14 symmetry 0.082; OV: norm 19.961 top [6.211 5.658 5.403 5.113 4.867] sr 10.33 r90 21 symmetry 0.063 trace -23.812\nL1H2 QK: norm 3.618 top [1.811 1.583 1.277 0.905 0.808] sr 3.99 r90 14 symmetry 0.070; OV: norm 23.710 top [6.799 6.707 6.454 6.192 6. ] sr 12.16 r90 20 symmetry 0.079 trace -31.928\nL1H3 QK: norm 3.776 top [1.998 1.66 1.278 0.901 0.796] sr 3.57 r90 14 symmetry 0.062; OV: norm 20.845 top [7.119 6.39 5.875 5.68 5.417] sr 8.57 r90 18 symmetry 0.053 trace -22.474\nL1H4 QK: norm 3.647 top [1.853 1.468 1.368 0.938 0.865] sr 3.87 r90 14 symmetry 0.052; OV: norm 19.572 top [6.354 5.996 5.642 5.541 5.215] sr 9.49 r90 18 symmetry 0.046 trace -21.931\nL1H5 QK: norm 4.695 top [2.523 2.023 1.535 1.4 1.068] sr 3.46 r90 13 symmetry 0.072; OV: norm 20.352 top [5.781 5.643 5.336 4.976 4.846] sr 12.39 r90 22 symmetry 0.073 trace -27.708\n\nLAYER 2\nln1_w min/mean/max/std: 0.3444815 0.73433226 0.90473896 0.071201816 norm: 14.4574\nln1_b min/mean/max/std: -0.9416654 0.024390584 0.73256344 0.13152973 norm: 2.6213865\nln2_w min/mean/max/std: 0.07753398 1.2749993 1.5018098 0.11256402 norm: 25.081963\nln2_b min/mean/max/std: -0.41570842 0.01880513 0.36354828 0.14735465 norm: 2.9109683\nW_q rms 0.09761 topSV [5.067 4.742 4.456 4.39 4.314] stableRank 54.72 r90 170\nW_k rms 0.09468 topSV [8.293 7.348 5.059 4.467 4.306] stableRank 19.22 r90 169\nW_v rms 0.09049 topSV [3.89 3.824 3.791 3.654 3.644] stableRank 79.78 r90 176\nW_o rms 0.08908 topSV [4.442 4.073 3.768 3.455 3.44 ] stableRank 59.31 r90 190\nW_fc rms 0.10346 topSV [14.829 8.048 7.921 7.76 7.66 ] stableRank 28.71 r90 252\nW_proj rms 0.10088 topSV [10.095 8.985 8.075 7.909 7.791] stableRank 58.91 r90 272\nL2H0 QK: norm 3.752 top [0.935 0.887 0.852 0.809 0.745] sr 16.09 r90 39 symmetry -0.006; OV: norm 25.165 top [5.556 5.132 5.005 4.795 4.773] sr 20.52 r90 41 symmetry 0.080 trace -48.290\nL2H1 QK: norm 3.610 top [0.907 0.818 0.747 0.716 0.675] sr 15.83 r90 39 symmetry 0.040; OV: norm 28.885 top [8.652 6.394 6.333 6.217 6.016] sr 11.15 r90 39 symmetry 0.072 trace -54.375\nL2H2 QK: norm 5.221 top [1.354 1.284 1.219 1.175 1.148] sr 14.87 r90 31 symmetry 0.034; OV: norm 22.619 top [4.742 4.585 4.492 4.275 4.22 ] sr 22.75 r90 42 symmetry 0.069 trace -39.941\nL2H3 QK: norm 3.648 top [1.012 0.878 0.775 0.766 0.712] sr 13.01 r90 39 symmetry -0.015; OV: norm 27.634 top [6.179 5.765 5.693 5.322 5.194] sr 20.00 r90 42 symmetry 0.090 trace -51.741\nL2H4 QK: norm 4.112 top [1.451 1.211 0.897 0.843 0.772] sr 8.03 r90 36 symmetry -0.001; OV: norm 24.785 top [5.445 5.184 5.128 4.988 4.903] sr 20.72 r90 38 symmetry 0.096 trace -52.123\nL2H5 QK: norm 3.041 top [0.788 0.74 0.685 0.588 0.574] sr 14.90 r90 42 symmetry -0.005; OV: norm 31.883 top [8.467 7.339 6.951 6.407 6.182] sr 14.18 r90 40 symmetry 0.053 trace -45.690\n\nLAYER 3\nln1_w min/mean/max/std: 0.45101672 0.7693753 0.8778313 0.04688468 norm: 15.104584\nln1_b min/mean/max/std: -0.5140651 0.010210011 0.62853575 0.09446622 norm: 1.861933\nln2_w min/mean/max/std: 0.26189932 1.2551554 1.4567947 0.08882699 norm: 24.65744\nln2_b min/mean/max/std: -0.5112283 0.013986574 0.3808068 0.12666406 norm: 2.497185\nW_q rms 0.08839 topSV [4.154 3.946 3.875 3.601 3.521] stableRank 66.74 r90 186\nW_k rms 0.08651 topSV [4.888 3.928 3.62 3.437 3.381] stableRank 46.20 r90 190\nW_v rms 0.10140 topSV [4.697 4.356 4.255 4.208 4.171] stableRank 68.73 r90 179\nW_o rms 0.10161 topSV [9.175 6.63 4.442 4.111 4.054] stableRank 18.08 r90 172\nW_fc rms 0.10181 topSV [14.373 8.176 7.973 7.83 7.036] stableRank 29.59 r90 279\nW_proj rms 0.11079 topSV [28.175 20.293 19.292 19.135 18.708] stableRank 9.12 r90 25\nL3H0 QK: norm 3.500 top [0.699 0.673 0.651 0.641 0.626] sr 25.05 r90 48 symmetry 0.019; OV: norm 31.618 top [7.123 6.403 6.09 5.949 5.73 ] sr 19.70 r90 44 symmetry 0.067 trace -62.391\nL3H1 QK: norm 2.865 top [0.55 0.535 0.521 0.511 0.505] sr 27.15 r90 48 symmetry 0.023; OV: norm 33.152 top [9.373 6.819 6.701 6.663 6.486] sr 12.51 r90 37 symmetry 0.081 trace -63.987\nL3H2 QK: norm 3.474 top [0.781 0.687 0.662 0.608 0.6 ] sr 19.80 r90 48 symmetry -0.008; OV: norm 34.637 top [11.577 7.787 6.815 6.5 6.372] sr 8.95 r90 42 symmetry 0.074 trace -65.099\nL3H3 QK: norm 1.784 top [0.461 0.408 0.363 0.34 0.328] sr 14.99 r90 47 symmetry -0.010; OV: norm 48.677 top [18.625 12.287 11.952 11.362 11.111] sr 6.83 r90 32 symmetry 0.065 trace -65.504\nL3H4 QK: norm 2.930 top [0.715 0.63 0.607 0.545 0.526] sr 16.79 r90 47 symmetry 0.004; OV: norm 36.644 top [11.774 10.138 8.487 7.578 7.163] sr 9.69 r90 40 symmetry 0.040 trace -45.155\nL3H5 QK: norm 3.581 top [0.676 0.658 0.643 0.627 0.617] sr 28.07 r90 48 symmetry 0.220; OV: norm 30.624 top [13.046 7.099 6.721 6.521 6.348] sr 5.51 r90 26 symmetry 0.209 trace -73.740\n",
"seconds": 1.334,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "ff0f96a22798aab24541eac1d242643c954645d2a9df919cf77c506ec0deff43",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}WEIGHT KEYS: ['W_fc.0', 'W_fc.1', 'W_fc.2', 'W_fc.3', 'W_k.0', 'W_k.1', 'W_k.2', 'W_k.3', 'W_o.0', 'W_o.1', 'W_o.2', 'W_o.3', 'W_proj.0', 'W_proj.1', 'W_proj.2', 'W_proj.3', 'W_q.0', 'W_q.1', 'W_q.2', 'W_q.3', 'W_v.0', 'W_v.1', 'W_v.2', 'W_v.3', 'ln1_b.0', 'ln1_b.1', 'ln1_b.2', 'ln1_b.3', 'ln1_w.0', 'ln1_w.1', 'ln1_w.2', 'ln1_w.3', 'ln2_b.0', 'ln2_b.1', 'ln2_b.2', 'ln2_b.3', 'ln2_w.0', 'ln2_w.1', 'ln2_w.2', 'ln2_w.3'] LAYER 0 ln1_w min/mean/max/std: 0.01654764 0.26770642 1.0134335 0.21465456 norm: 6.724094 ln1_b min/mean/max/std: -0.12149114 0.00065349764 0.14177816 0.036595043 norm: 0.7172278 ln2_w min/mean/max/std: 0.04681267 0.8168376 1.6411812 0.33119047 norm: 17.27234 ln2_b min/mean/max/std: -0.55371493 0.017958164 0.16627714 0.066667244 norm: 1.3529724 W_q rms 0.05912 topSV [5.039 4.999 4.703 4.377 4.343] stableRank 20.30 r90 61 W_k rms 0.06045 topSV [6.204 5.052 4.572 4.537 4.391] stableRank 14.00 r90 60 W_v rms 0.04739 topSV [4.553 4.024 3.738 3.497 3.465] stableRank 15.97 r90 66 W_o rms 0.05104 topSV [6.678 5.237 2.835 2.679 2.62 ] stableRank 8.61 r90 75 W_fc rms 0.06725 topSV [18.11 12.28 8.473 7.902 7.779] stableRank 8.13 r90 87 W_proj rms 0.05964 topSV [10.932 7.576 6.628 6.293 6.166] stableRank 17.56 r90 135 L0H0 QK: norm 0.807 top [0.649 0.424 0.106 0.082 0.064] sr 1.54 r90 2 symmetry 0.155; OV: norm 8.667 top [6.898 3.897 1.936 1.638 1.469] sr 1.58 r90 4 symmetry 0.049 trace 1.620 L0H1 QK: norm 4.067 top [2.153 1.515 1.437 1.216 1.074] sr 3.57 r90 14 symmetry 0.164; OV: norm 17.067 top [4.703 4.573 4.424 4.248 4.199] sr 13.17 r90 20 symmetry 0.004 trace -0.820 L0H2 QK: norm 1.136 top [0.561 0.447 0.374 0.356 0.324] sr 4.10 r90 11 symmetry 0.176; OV: norm 8.592 top [4.786 3.621 3.514 2.916 2.224] sr 3.22 r90 7 symmetry 0.003 trace -0.743 L0H3 QK: norm 1.434 top [0.959 0.701 0.431 0.294 0.279] sr 2.23 r90 7 symmetry 0.085; OV: norm 5.288 top [2.168 1.997 1.875 1.555 1.484] sr 5.95 r90 12 symmetry 0.007 trace -0.071 L0H4 QK: norm 0.097 top [0.04 0.03 0.024 0.018 0.017] sr 5.86 r90 41 symmetry 0.039; OV: norm 25.795 top [18.815 15.373 5.587 3.989 2.502] sr 1.88 r90 3 symmetry 0.039 trace 6.681 L0H5 QK: norm 5.785 top [1.76 1.72 1.649 1.558 1.543] sr 10.81 r90 18 symmetry -0.228; OV: norm 18.558 top [5.61 5.503 5.123 4.867 4.799] sr 10.94 r90 17 symmetry 0.008 trace -11.398 LAYER 1 ln1_w min/mean/max/std: 0.19793172 0.4753575 1.2095798 0.09638874 norm: 9.504637 ln1_b min/mean/max/std: -1.0504974 0.022033103 0.9102453 0.15318407 norm: 3.0326743 ln2_w min/mean/max/std: 0.0139104035 1.0110891 1.470659 0.15819876 norm: 20.054274 ln2_b min/mean/max/std: -0.37441117 0.02753053 0.36524463 0.14420813 norm: 2.876926 W_q rms 0.08249 topSV [8.984 7.137 6.079 5.874 5.19 ] stableRank 12.43 r90 66 W_k rms 0.07448 topSV [10.967 9.448 5.438 4.842 4.539] stableRank 6.80 r90 68 W_v rms 0.07414 topSV [5.35 5.064 4.979 4.819 4.69 ] stableRank 28.31 r90 84 W_o rms 0.06810 topSV [4.19 3.651 3.385 3.352 3.24 ] stableRank 38.95 r90 119 W_fc rms 0.10451 topSV [20.796 14.143 13.826 13.428 13.122] stableRank 14.90 r90 93 W_proj rms 0.09781 topSV [7.625 7.359 6.908 6.489 6.344] stableRank 97.06 r90 280 L1H0 QK: norm 2.471 top [1.116 0.861 0.756 0.719 0.664] sr 4.90 r90 15 symmetry 0.042; OV: norm 27.504 top [9.927 9.271 8.872 7.793 7.506] sr 7.68 r90 16 symmetry 0.087 trace -34.568 L1H1 QK: norm 4.754 top [2.392 1.846 1.725 1.378 1.068] sr 3.95 r90 14 symmetry 0.082; OV: norm 19.961 top [6.211 5.658 5.403 5.113 4.867] sr 10.33 r90 21 symmetry 0.063 trace -23.812 L1H2 QK: norm 3.618 top [1.811 1.583 1.277 0.905 0.808] sr 3.99 r90 14 symmetry 0.070; OV: norm 23.710 top [6.799 6.707 6.454 6.192 6. ] sr 12.16 r90 20 symmetry 0.079 trace -31.928 L1H3 QK: norm 3.776 top [1.998 1.66 1.278 0.901 0.796] sr 3.57 r90 14 symmetry 0.062; OV: norm 20.845 top [7.119 6.39 5.875 5.68 5.417] sr 8.57 r90 18 symmetry 0.053 trace -22.474 L1H4 QK: norm 3.647 top [1.853 1.468 1.368 0.938 0.865] sr 3.87 r90 14 symmetry 0.052; OV: norm 19.572 top [6.354 5.996 5.642 5.541 5.215] sr 9.49 r90 18 symmetry 0.046 trace -21.931 L1H5 QK: norm 4.695 top [2.523 2.023 1.535 1.4 1.068] sr 3.46 r90 13 symmetry 0.072; OV: norm 20.352 top [5.781 5.643 5.336 4.976 4.846] sr 12.39 r90 22 symmetry 0.073 trace -27.708 LAYER 2 ln1_w min/mean/max/std: 0.3444815 0.73433226 0.90473896 0.071201816 norm: 14.4574 ln1_b min/mean/max/std: -0.9416654 0.024390584 0.73256344 0.13152973 norm: 2.6213865 ln2_w min/mean/max/std: 0.07753398 1.2749993 1.5018098 0.11256402 norm: 25.081963 ln2_b min/mean/max/std: -0.41570842 0.01880513 0.36354828 0.14735465 norm: 2.9109683 W_q rms 0.09761 topSV [5.067 4.742 4.456 4.39 4.314] stableRank 54.72 r90 170 W_k rms 0.09468 topSV [8.293 7.348 5.059 4.467 4.306] stableRank 19.22 r90 169 W_v rms 0.09049 topSV [3.89 3.824 3.791 3.654 3.644] stableRank 79.78 r90 176 W_o rms 0.08908 topSV [4.442 4.073 3.768 3.455 3.44 ] stableRank 59.31 r90 190 W_fc rms 0.10346 topSV [14.829 8.048 7.921 7.76 7.66 ] stableRank 28.71 r90 252 W_proj rms 0.10088 topSV [10.095 8.985 8.075 7.909 7.791] stableRank 58.91 r90 272 L2H0 QK: norm 3.752 top [0.935 0.887 0.852 0.809 0.745] sr 16.09 r90 39 symmetry -0.006; OV: norm 25.165 top [5.556 5.132 5.005 4.795 4.773] sr 20.52 r90 41 symmetry 0.080 trace -48.290 L2H1 QK: norm 3.610 top [0.907 0.818 0.747 0.716 0.675] sr 15.83 r90 39 symmetry 0.040; OV: norm 28.885 top [8.652 6.394 6.333 6.217 6.016] sr 11.15 r90 39 symmetry 0.072 trace -54.375 L2H2 QK: norm 5.221 top [1.354 1.284 1.219 1.175 1.148] sr 14.87 r90 31 symmetry 0.034; OV: norm 22.619 top [4.742 4.585 4.492 4.275 4.22 ] sr 22.75 r90 42 symmetry 0.069 trace -39.941 L2H3 QK: norm 3.648 top [1.012 0.878 0.775 0.766 0.712] sr 13.01 r90 39 symmetry -0.015; OV: norm 27.634 top [6.179 5.765 5.693 5.322 5.194] sr 20.00 r90 42 symmetry 0.090 trace -51.741 L2H4 QK: norm 4.112 top [1.451 1.211 0.897 0.843 0.772] sr 8.03 r90 36 symmetry -0.001; OV: norm 24.785 top [5.445 5.184 5.128 4.988 4.903] sr 20.72 r90 38 symmetry 0.096 trace -52.123 L2H5 QK: norm 3.041 top [0.788 0.74 0.685 0.588 0.574] sr 14.90 r90 42 symmetry -0.005; OV: norm 31.883 top [8.467 7.339 6.951 6.407 6.182] sr 14.18 r90 40 symmetry 0.053 trace -45.690 LAYER 3 ln1_w min/mean/max/std: 0.45101672 0.7693753 0.8778313 0.04688468 norm: 15.104584 ln1_b min/mean/max/std: -0.5140651 0.010210011 0.62853575 0.09446622 norm: 1.861933 ln2_w min/mean/max/std: 0.26189932 1.2551554 1.4567947 0.08882699 norm: 24.65744 ln2_b min/mean/max/std: -0.5112283 0.013986574 0.3808068 0.12666406 norm: 2.497185 W_q rms 0.08839 topSV [4.154 3.946 3.875 3.601 3.521] stableRank 66.74 r90 186 W_k rms 0.08651 topSV [4.888 3.928 3.62 3.437 3.381] stableRank 46.20 r90 190 W_v rms 0.10140 topSV [4.697 4.356 4.255 4.208 4.171] stableRank 68.73 r90 179 W_o rms 0.10161 topSV [9.175 6.63 4.442 4.111 4.054] stableRank 18.08 r90 172 W_fc rms 0.10181 topSV [14.373 8.176 7.973 7.83 7.036] stableRank 29.59 r90 279 W_proj rms 0.11079 topSV [28.175 20.293 19.292 19.135 18.708] stableRank 9.12 r90 25 L3H0 QK: norm 3.500 top [0.699 0.673 0.651 0.641 0.626] sr 25.05 r90 48 symmetry 0.019; OV: norm 31.618 top [7.123 6.403 6.09 5.949 5.73 ] sr 19.70 r90 44 symmetry 0.067 trace -62.391 L3H1 QK: norm 2.865 top [0.55 0.535 0.521 0.511 0.505] sr 27.15 r90 48 symmetry 0.023; OV: norm 33.152 top [9.373 6.819 6.701 6.663 6.486] sr 12.51 r90 37 symmetry 0.081 trace -63.987 L3H2 QK: norm 3.474 top [0.781 0.687 0.662 0.608 0.6 ] sr 19.80 r90 48 symmetry -0.008; OV: norm 34.637 top [11.577 7.787 6.815 6.5 6.372] sr 8.95 r90 42 symmetry 0.074 trace -65.099 L3H3 QK: norm 1.784 top [0.461 0.408 0.363 0.34 0.328] sr 14.99 r90 47 symmetry -0.010; OV: norm 48.677 top [18.625 12.287 11.952 11.362 11.111] sr 6.83 r90 32 symmetry 0.065 trace -65.504 L3H4 QK: norm 2.930 top [0.715 0.63 0.607 0.545 0.526] sr 16.79 r90 47 symmetry 0.004; OV: norm 36.644 top [11.774 10.138 8.487 7.578 7.163] sr 9.69 r90 40 symmetry 0.040 trace -45.155 L3H5 QK: norm 3.581 top [0.676 0.658 0.643 0.627 0.617] sr 28.07 r90 48 symmetry 0.220; OV: norm 30.624 top [13.046 7.099 6.721 6.521 6.348] sr 5.51 r90 26 symmetry 0.209 trace -73.740
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 1 of at most 100: ``` WEIGHT KEYS: ['W_fc.0', 'W_fc.1', 'W_fc.2', 'W_fc.3', 'W_k.0', 'W_k.1', 'W_k.2', 'W_k.3', 'W_o.0', 'W_o.1', 'W_o.2', 'W_o.3', 'W_proj.0', 'W_proj.1', 'W_proj.2', 'W_proj.3', 'W_q.0', 'W_q.1', 'W_q.2', 'W_q.3', 'W_v.0', 'W_v.1', 'W_v.2', 'W_v.3', 'ln1_b.0', 'ln1_b.1', 'ln1_b.2', 'ln1_b.3', 'ln1_w.0', 'ln1_w.1', 'ln1_w.2', 'ln1_w.3', 'ln2_b.0', 'ln2_b.1', 'ln2_b.2', 'ln2_b.3', 'ln2_w.0', 'ln2_w.1', 'ln2_w.2', 'ln2_w.3'] LAYER 0 ln1_w min/mean/max/std: 0.01654764 0.26770642 1.0134335 0.21465456 norm: 6.724094 ln1_b min/mean/max/std: -0.12149114 0.00065349764 0.14177816 0.036595043 norm: 0.7172278 ln2_w min/mean/max/std: 0.04681267 0.8168376 1.6411812 0.33119047 norm: 17.27234 ln2_b min/mean/max/std: -0.55371493 0.017958164 0.16627714 0.066667244 norm: 1.3529724 W_q rms 0.05912 topSV [5.039 4.999 4.703 4.377 4.343] stableRank 20.30 r90 61 W_k rms 0.06045 topSV [6.204 5.052 4.572 4.537 4.391] stableRank 14.00 r90 60 W_v rms 0.04739 topSV [4.553 4.024 3.738 3.497 3.465] stableRank 15.97 r90 66 W_o rms 0.05104 topSV [6.678 5.237 2.835 2.679 2.62 ] stableRank 8.61 r90 75 W_fc rms 0.06725 topSV [18.11 12.28 8.473 7.902 7.779] stableRank 8.13 r90 87 W_proj rms 0.05964 topSV [10.932 7.576 6.628 6.293 6.166] stableRank 17.56 r90 135 L0H0 QK: norm 0.807 top [0.649 0.424 0.106 0.082 0.064] sr 1.54 r90 2 symmetry 0.155; OV: norm 8.667 top [6.898 3.897 1.936 1.638 1.469] sr 1.58 r90 4 symmetry 0.049 trace 1.620 L0H1 QK: norm 4.067 top [2.153 1.515 1.437 1.216 1.074] sr 3.57 r90 14 symmetry 0.164; OV: norm 17.067 top [4.703 4.573 4.424 4.248 4.199] sr 13.17 r90 20 symmetry 0.004 trace -0.820 L0H2 QK: norm 1.136 top [0.561 0.447 0.374 0.356 0.324] sr 4.10 r90 11 symmetry 0.176; OV: norm 8.592 top [4.786 3.621 3.514 2.916 2.224] sr 3.22 r90 7 symmetry 0.003 trace -0.743 L0H3 QK: norm 1.434 top [0.959 0.701 0.431 0.294 0.279] sr 2.23 r90 7 symmetry 0.085; OV: norm 5.288 top [2.168 1.997 1.875 1.555 1.484] sr 5.95 r90 12 symmetry 0.007 trace -0.071 L0H4 QK: norm 0.097 top [0.04 0.03 0.024 0.018 0.017] sr 5.86 r90 41 symmetry 0.039; OV: norm 25.795 top [18.815 15.373 5.587 3.989 2.502] sr 1.88 r90 3 symmetry 0.039 trace 6.681 L0H5 QK: norm 5.785 top [1.76 1.72 1.649 1.558 1.543] sr 10.81 r90 18 symmetry -0.228; OV: norm 18.558 top [5.61 5.503 5.123 4.867 4.799] sr 10.94 r90 17 symmetry 0.008 trace -11.398 LAYER 1 ln1_w min/mean/max/std: 0.19793172 0.4753575 1.2095798 0.09638874 norm: 9.504637 ln1_b min/mean/max/std: -1.0504974 0.022033103 0.9102453 0.15318407 norm: 3.0326743 ln2_w min/mean/max/std: 0.0139104035 1.0110891 1.470659 0.15819876 norm: 20.054274 ln2_b min/mean/max/std: -0.37441117 0.02753053 0.36524463 0.14420813 norm: 2.876926 W_q rms 0.08249 topSV [8.984 7.137 6.079 5.874 5.19 ] stableRank 12.43 r90 66 W_k rms 0.07448 topSV [10.967 9.448 5.438 4.842 4.539] stableRank 6.80 r90 68 W_v rms 0.07414 topSV [5.35 5.064 4.979 4.819 4.69 ] stableRank 28.31 r90 84 W_o rms 0.06810 topSV [4.19 3.651 3.385 3.352 3.24 ] stableRank 38.95 r90 119 W_fc rms 0.10451 topSV [20.796 14.143 13.826 13.428 13.122] stableRank 14.90 r90 93 W_proj rms 0.09781 topSV [7.625 7.359 6.908 6.489 6.344] stableRank 97.06 r90 280 L1H0 QK: norm 2.471 top [1.116 0.861 0.756 0.719 0.664] sr 4.90 r90 15 symmetry 0.042; OV: norm 27.504 top [9.927 9.271 8.872 7.793 7.506] sr 7.68 r90 16 symmetry 0.087 trace -34.568 L1H1 QK: norm 4.754 top [2.392 1.846 1.725 1.378 1.068] sr 3.95 r90 14 symmetry 0.082; OV: norm 19.961 top [6.211 5.658 5.403 5.113 4.867] sr 10.33 r90 21 symmetry 0.063 trace -23.812 L1H2 QK: norm 3.618 top [1.811 1.583 1.277 0.905 0.808] sr 3.99 r90 14 symmetry 0.070; OV: norm 23.710 top [6.799 6.707 6.454 6.192 6. ] sr 12.16 r90 20 symmetry 0.079 trace -31.928 L1H3 QK: norm 3.776 top [1.998 1.66 1.278 0.901 0.796] sr 3.57 r90 14 symmetry 0.062; OV: norm 20.845 top [7.119 6.39 5.875 5.68 5.417] sr 8.57 r90 18 symmetry 0.053 trace -22.474 L1H4 QK: norm 3.647 top [1.853 1.468 1.368 0.938 0.865] sr 3.87 r90 14 symmetry 0.052; OV: norm 19.572 top [6.354 5.996 5.642 5.541 5.215] sr 9.49 r90 18 symmetry 0.046 trace -21.931 L1H5 QK: norm 4.695 top [2.523 2.023 1.535 1.4 1.068] sr 3.46 r90 13 symmetry 0.072; OV: norm 20.352 top [5.781 5.643 5.336 4.976 4.846] sr 12.39 r90 22 symmetry 0.073 trace -27.708 LAYER 2 ln1_w min/mean/max/std: 0.3444815 0.73433226 0.90473896 0.071201816 norm: 14.4574 ln1_b min/mean/max/std: -0.9416654 0.024390584 0.73256344 0.13152973 norm: 2.6213865 ln2_w min/mean/max/std: 0.07753398 1.2749993 1.5018098 0.11256402 norm: 25.081963 ln2_b min/mean/max/std: -0.41570842 0.01880513 0.36354828 0.14735465 norm: 2.9109683 W_q rms 0.09761 topSV [5.067 4.742 4.456 4.39 4.314] stableRank 54.72 r90 170 W_k rms 0.09468 topSV [8.293 7.348 5.059 4.467 4.306] stableRank 19.22 r90 169 W_v rms 0.09049 topSV [3.89 3.824 3.791 3.654 3.644] stableRank 79.78 r90 176 W_o rms 0.08908 topSV [4.442 4.073 3.768 3.455 3.44 ] stableRank 59.31 r90 190 W_fc rms 0.10346 topSV [14.829 8.048 7.921 7.76 7.66 ] stableRank 28.71 r90 252 W_proj rms 0.10088 topSV [10.095 8.985 8.075 7.909 7.791] stableRank 58.91 r90 272 L2H0 QK: norm 3.752 top [0.935 0.887 0.852 0.809 0.745] sr 16.09 r90 39 symmetry -0.006; OV: norm 25.165 top [5.556 5.132 5.005 4.795 4.773] sr 20.52 r90 41 symmetry 0.080 trace -48.290 L2H1 QK: norm 3.610 top [0.907 0.818 0.747 0.716 0.675] sr 15.83 r90 39 symmetry 0.040; OV: norm 28.885 top [8.652 6.394 6.333 6.217 6.016] sr 11.15 r90 39 symmetry 0.072 trace -54.375 L2H2 QK: norm 5.221 top [1.354 1.284 1.219 1.175 1.148] sr 14.87 r90 31 symmetry 0.034; OV: norm 22.619 top [4.742 4.585 4.492 4.275 4.22 ] sr 22.75 r90 42 symmetry 0.069 trace -39.941 L2H3 QK: norm 3.648 top [1.012 0.878 0.775 0.766 0.712] sr 13.01 r90 39 symmetry -0.015; OV: norm 27.634 top [6.179 5.765 5.693 5.322 5.194] sr 20.00 r90 42 symmetry 0.090 trace -51.741 L2H4 QK: norm 4.112 top [1.451 1.211 0.897 0.843 0.772] sr 8.03 r90 36 symmetry -0.001; OV: norm 24.785 top [5.445 5.184 5.128 4.988 4.903] sr 20.72 r90 38 symmetry 0.096 trace -52.123 L2H5 QK: norm 3.041 top [0.788 0.74 0.685 0.588 0.574] sr 14.90 r90 42 symmetry -0.005; OV: norm 31.883 top [8.467 7.339 6.951 6.407 6.182] sr 14.18 r90 40 symmetry 0.053 trace -45.690 LAYER 3 ln1_w min/mean/max/std: 0.45101672 0.7693753 0.8778313 0.04688468 norm: 15.104584 ln1_b min/mean/max/std: -0.5140651 0.010210011 0.62853575 0.09446622 norm: 1.861933 ln2_w min/mean/max/std: 0.26189932 1.2551554 1.4567947 0.08882699 norm: 24.65744 ln2_b min/mean/max/std: -0.5112283 0.013986574 0.3808068 0.12666406 norm: 2.497185 W_q rms 0.08839 topSV [4.154 3.946 3.875 3.601 3.521] stableRank 66.74 r90 186 W_k rms 0.08651 topSV [4.888 3.928 3.62 3.437 3.381] stableRank 46.20 r90 190 W_v rms 0.10140 topSV [4.697 4.356 4.255 4.208 4.171] stableRank 68.73 r90 179 W_o rms 0.10161 topSV [9.175 6.63 4.442 4.111 4.054] stableRank 18.08 r90 172 W_fc rms 0.10181 topSV [14.373 8.176 7.973 7.83 7.036] stableRank 29.59 r90 279 W_proj rms 0.11079 topSV [28.175 20.293 19.292 19.135 18.708] stableRank 9.12 r90 25 L3H0 QK: norm 3.500 top [0.699 0.673 0.651 0.641 0.626] sr 25.05 r90 48 symmetry 0.019; OV: norm 31.618 top [7.123 6.403 6.09 5.949 5.73 ] sr 19.70 r90 44 symmetry 0.067 trace -62.391 L3H1 QK: norm 2.865 top [0.55 0.535 0.521 0.511 0.505] sr 27.15 r90 48 symmetry 0.023; OV: norm 33.152 top [9.373 6.819 6.701 6.663 6.486] sr 12.51 r90 37 symmetry 0.081 trace -63.987 L3H2 QK: norm 3.474 top [0.781 0.687 0.662 0.608 0.6 ] sr 19.80 r90 48 symmetry -0.008; OV: norm 34.637 top [11.577 7.787 6.815 6.5 6.372] sr 8.95 r90 42 symmetry 0.074 trace -65.099 L3H3 QK: norm 1.784 top [0.461 0.408 0.363 0.34 0.328] sr 14.99 r90 47 symmetry -0.010; OV: norm 48.677 top [18.625 12.287 11.952 11.362 11.111] sr 6.83 r90 32 symmetry 0.065 trace -65.504 L3H4 QK: norm 2.930 top [0.715 0.63 0.607 0.545 0.526] sr 16.79 r90 47 symmetry 0.004; OV: norm 36.644 top [11.774 10.138 8.487 7.578 7.163] sr 9.69 r90 40 symmetry 0.040 trace -45.155 L3H5 QK: norm 3.581 top [0.676 0.658 0.643 0.627 0.617] sr 28.07 r90 48 symmetry 0.220; OV: norm 30.624 top [13.046 7.099 6.721 6.521 6.348] sr 5.51 r90 26 symmetry 0.209 trace -73.740 ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svd, eigvals
from threadpoolctl import threadpool_limits
threadpool_limits(limits=2)
np.set_printoptions(precision=3, suppress=True)
# These are weight contractions only, not evaluations on inputs.
# Multiplying columns by gamma and removing their constant-coordinate component
# expresses each matrix on the mean-zero residual subspace.
def read_matrix(name, l, sl=None):
a = W[f"{name}.{l}"].astype(float)
if sl is not None:
a = a[sl]
a = a * W[f"ln1_w.{l}"][None, :]
return a - a.mean(axis=1, keepdims=True)
A, B, Q, K, V = [], [], [], [], []
for l in range(4):
for j in range(6):
sl = slice(64*j, 64*(j+1))
q, k, v = [read_matrix(n, l, sl) for n in ["W_q", "W_k", "W_v"]]
o = W[f"W_o.{l}"][:, sl].astype(float)
o = o - o.mean(axis=0, keepdims=True)
a, b = q.T @ k / 8, o @ v
A.append(a); B.append(b); Q.append(q); K.append(k); V.append(v)
sa = svd(a, compute_uv=False, check_finite=False)
ev = eigvals(a, check_finite=False)
ev = ev[np.argsort(-np.abs(ev))]
beta = W[f"ln1_b.{l}"].astype(float)
qb = W[f"W_q.{l}"][sl] @ beta
kb = W[f"W_k.{l}"][sl] @ beta
key_linear = k.T @ qb / 8
query_linear = q.T @ kb / 8
print("L%dH%d effective QK norm %.4f SV %s; eig %s; "
"key/query bias norms %.3f %.3f; effective OV norm %.3f trace %.3f" %
(l, j, np.linalg.norm(a), str(sa[:4]), str(ev[:6]),
np.linalg.norm(key_linear), np.linalg.norm(query_linear),
np.linalg.norm(b), np.trace(b)))
print("\nNORMALIZED TRACE(QK_l @ OV_earlier), columns earlier heads")
for l in range(1, 4):
for m in range(l):
scores = np.array([
[np.sum(A[6*l+j] * B[6*m+k].T) /
(np.linalg.norm(A[6*l+j])*np.linalg.norm(B[6*m+k]))
for k in range(6)] for j in range(6)])
print("QK layer", l, "OV layer", m)
print(scores)
print("\nL0: overlap of top QK singular subspaces with top value-read subspace")
vall = np.concatenate(V[:6], axis=0)
_, sv, vt = svd(vall, full_matrices=False, check_finite=False)
print("Combined effective V0 singular values:", sv[:45])
for j in range(6):
u, s, vtq = svd(A[j], full_matrices=False, check_finite=False)
for r in [8, 24]:
rv = vt[:r].T
qe = np.linalg.norm(rv.T @ A[j])**2 / np.linalg.norm(A[j])**2
ke = np.linalg.norm(A[j] @ rv)**2 / np.linalg.norm(A[j])**2
print("L0H%d V-rank%d query/key energy %.3f %.3f" % (j, r, qe, ke))
print("\nL0 QK pairwise Frobenius cosines")
print(np.array([[np.sum(a*b)/(np.linalg.norm(a)*np.linalg.norm(b))
for b in A[:6]] for a in A[:6]]))
print("\nL0 V-head top-24 right-subspace overlaps (mean squared cosine)")
bases = [svd(v, full_matrices=False, check_finite=False)[2][:24].T for v in V[:6]]
print(np.array([[np.linalg.norm(a.T@b)**2/24 for b in bases] for a in bases]))
```
{
"run": 2,
"turn": "t002",
"code": "import numpy as np\nfrom scipy.linalg import svd, eigvals\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(limits=2)\nnp.set_printoptions(precision=3, suppress=True)\n\n# These are weight contractions only, not evaluations on inputs.\n# Multiplying columns by gamma and removing their constant-coordinate component\n# expresses each matrix on the mean-zero residual subspace.\ndef read_matrix(name, l, sl=None):\n a = W[f\"{name}.{l}\"].astype(float)\n if sl is not None:\n a = a[sl]\n a = a * W[f\"ln1_w.{l}\"][None, :]\n return a - a.mean(axis=1, keepdims=True)\n\nA, B, Q, K, V = [], [], [], [], []\nfor l in range(4):\n for j in range(6):\n sl = slice(64*j, 64*(j+1))\n q, k, v = [read_matrix(n, l, sl) for n in [\"W_q\", \"W_k\", \"W_v\"]]\n o = W[f\"W_o.{l}\"][:, sl].astype(float)\n o = o - o.mean(axis=0, keepdims=True)\n a, b = q.T @ k / 8, o @ v\n A.append(a); B.append(b); Q.append(q); K.append(k); V.append(v)\n sa = svd(a, compute_uv=False, check_finite=False)\n ev = eigvals(a, check_finite=False)\n ev = ev[np.argsort(-np.abs(ev))]\n beta = W[f\"ln1_b.{l}\"].astype(float)\n qb = W[f\"W_q.{l}\"][sl] @ beta\n kb = W[f\"W_k.{l}\"][sl] @ beta\n key_linear = k.T @ qb / 8\n query_linear = q.T @ kb / 8\n print(\"L%dH%d effective QK norm %.4f SV %s; eig %s; \"\n \"key/query bias norms %.3f %.3f; effective OV norm %.3f trace %.3f\" %\n (l, j, np.linalg.norm(a), str(sa[:4]), str(ev[:6]),\n np.linalg.norm(key_linear), np.linalg.norm(query_linear),\n np.linalg.norm(b), np.trace(b)))\n\nprint(\"\\nNORMALIZED TRACE(QK_l @ OV_earlier), columns earlier heads\")\nfor l in range(1, 4):\n for m in range(l):\n scores = np.array([\n [np.sum(A[6*l+j] * B[6*m+k].T) /\n (np.linalg.norm(A[6*l+j])*np.linalg.norm(B[6*m+k]))\n for k in range(6)] for j in range(6)])\n print(\"QK layer\", l, \"OV layer\", m)\n print(scores)\n\nprint(\"\\nL0: overlap of top QK singular subspaces with top value-read subspace\")\nvall = np.concatenate(V[:6], axis=0)\n_, sv, vt = svd(vall, full_matrices=False, check_finite=False)\nprint(\"Combined effective V0 singular values:\", sv[:45])\nfor j in range(6):\n u, s, vtq = svd(A[j], full_matrices=False, check_finite=False)\n for r in [8, 24]:\n rv = vt[:r].T\n qe = np.linalg.norm(rv.T @ A[j])**2 / np.linalg.norm(A[j])**2\n ke = np.linalg.norm(A[j] @ rv)**2 / np.linalg.norm(A[j])**2\n print(\"L0H%d V-rank%d query/key energy %.3f %.3f\" % (j, r, qe, ke))\n\nprint(\"\\nL0 QK pairwise Frobenius cosines\")\nprint(np.array([[np.sum(a*b)/(np.linalg.norm(a)*np.linalg.norm(b))\n for b in A[:6]] for a in A[:6]]))\nprint(\"\\nL0 V-head top-24 right-subspace overlaps (mean squared cosine)\")\nbases = [svd(v, full_matrices=False, check_finite=False)[2][:24].T for v in V[:6]]\nprint(np.array([[np.linalg.norm(a.T@b)**2/24 for b in bases] for a in bases]))",
"out": "L0H0 effective QK norm 0.1137 SV [0.089 0.068 0.013 0.008]; eig [-0.049+0.j 0.006+0.009j 0.006-0.009j -0.003+0.j -0.001+0.002j\n -0.001-0.002j]; key/query bias norms 0.018 0.016; effective OV norm 4.321 trace 1.666\nL0H1 effective QK norm 0.8775 SV [0.601 0.373 0.297 0.211]; eig [ 0.218+0.345j 0.218-0.345j -0.284+0.j -0.123+0.15j -0.123-0.15j\n -0.097+0.098j]; key/query bias norms 0.074 0.106; effective OV norm 3.488 trace 0.164\nL0H2 effective QK norm 0.1745 SV [0.132 0.08 0.044 0.031]; eig [ 0.091+0.j 0.053+0.j -0.022+0.j -0.013+0.001j -0.013-0.001j\n 0.011+0.j ]; key/query bias norms 0.023 0.079; effective OV norm 2.023 trace 0.015\nL0H3 effective QK norm 0.2395 SV [0.165 0.14 0.081 0.031]; eig [-0.078+0.005j -0.078-0.005j -0.02 +0.j 0.006+0.017j 0.006-0.017j\n 0.007+0.j ]; key/query bias norms 0.057 0.149; effective OV norm 1.621 trace 0.242\nL0H4 effective QK norm 0.0093 SV [0.004 0.003 0.002 0.002]; eig [-0.001+0.001j -0.001-0.001j -0.001+0.j -0.001-0.j 0. +0.001j\n 0. -0.001j]; key/query bias norms 0.001 0.001; effective OV norm 17.158 trace 6.049\nL0H5 effective QK norm 0.9704 SV [0.41 0.362 0.273 0.254]; eig [0.207+0.275j 0.207-0.275j 0.114+0.172j 0.114-0.172j 0.178+0.j\n 0.155+0.084j]; key/query bias norms 0.051 0.066; effective OV norm 3.707 trace -2.463\nL1H0 effective QK norm 0.6440 SV [0.354 0.252 0.218 0.167]; eig [ 0.028+0.051j 0.028-0.051j -0.051+0.023j -0.051-0.023j -0.051+0.j\n -0.041+0.013j]; key/query bias norms 0.839 0.841; effective OV norm 13.436 trace -16.299\nL1H1 effective QK norm 1.5843 SV [1.146 0.661 0.459 0.326]; eig [ 0.08 +0.26j 0.08 -0.26j -0.144+0.009j -0.144-0.009j -0.092+0.079j\n -0.092-0.079j]; key/query bias norms 3.150 2.183; effective OV norm 9.592 trace -11.205\nL1H2 effective QK norm 1.2358 SV [0.807 0.693 0.332 0.205]; eig [ 0.057+0.202j 0.057-0.202j -0.108+0.j -0.07 +0.05j -0.07 -0.05j\n -0.079+0.004j]; key/query bias norms 2.200 1.897; effective OV norm 11.496 trace -14.892\nL1H3 effective QK norm 1.3094 SV [0.913 0.699 0.309 0.202]; eig [ 0.076+0.272j 0.076-0.272j -0.094+0.j 0.092+0.j -0.08 +0.024j\n -0.08 -0.024j]; key/query bias norms 2.719 2.207; effective OV norm 10.043 trace -10.571\nL1H4 effective QK norm 1.2492 SV [0.848 0.654 0.34 0.213]; eig [ 0.071+0.239j 0.071-0.239j -0.085+0.014j -0.085-0.014j -0.078+0.j\n -0.065+0.028j]; key/query bias norms 2.518 1.942; effective OV norm 9.557 trace -10.446\nL1H5 effective QK norm 1.4614 SV [1.009 0.555 0.482 0.45 ]; eig [ 0.072+0.204j 0.072-0.204j -0.156+0.j -0.124+0.j -0.083+0.053j\n -0.083-0.053j]; key/query bias norms 2.460 1.810; effective OV norm 9.826 trace -13.045\nL2H0 effective QK norm 2.0456 SV [0.518 0.485 0.459 0.416]; eig [ 0.033+0.132j 0.033-0.132j -0.117+0.j 0.091+0.074j 0.091-0.074j\n 0.037+0.104j]; key/query bias norms 0.895 0.774; effective OV norm 18.698 trace -35.970\nL2H1 effective QK norm 1.9454 SV [0.458 0.421 0.396 0.393]; eig [0.142+0.036j 0.142-0.036j 0.142+0.j 0.141+0.011j 0.141-0.011j\n 0.131+0.022j]; key/query bias norms 0.638 0.333; effective OV norm 21.448 trace -40.225\nL2H2 effective QK norm 2.8040 SV [0.709 0.693 0.676 0.634]; eig [0.259+0.j 0.08 +0.216j 0.08 -0.216j 0.193+0.058j 0.193-0.058j\n 0.195-0.026j]; key/query bias norms 1.584 1.581; effective OV norm 16.782 trace -29.462\nL2H3 effective QK norm 1.9759 SV [0.571 0.423 0.412 0.392]; eig [ 0.058+0.185j 0.058-0.185j 0.002+0.106j 0.002-0.106j -0.055+0.085j\n -0.055-0.085j]; key/query bias norms 1.037 1.265; effective OV norm 20.617 trace -38.568\nL2H4 effective QK norm 2.2016 SV [0.783 0.587 0.484 0.429]; eig [ 0.08 +0.22j 0.08 -0.22j 0.154+0.j -0.121+0.j 0.086+0.077j\n 0.086-0.077j]; key/query bias norms 1.199 1.340; effective OV norm 18.415 trace -38.753\nL2H5 effective QK norm 1.6541 SV [0.419 0.414 0.359 0.32 ]; eig [-0.107+0.j -0.007+0.101j -0.007-0.101j -0.079+0.045j -0.079-0.045j\n 0.09 +0.j ]; key/query bias norms 0.636 0.492; effective OV norm 23.766 trace -33.940\nL3H0 effective QK norm 2.0777 SV [0.405 0.399 0.385 0.377]; eig [0.13 +0.j 0.119-0.04j 0.119+0.04j 0.119-0.018j 0.119+0.018j\n 0.109+0.j ]; key/query bias norms 0.299 0.302; effective OV norm 24.502 trace -48.319\nL3H1 effective QK norm 1.6948 SV [0.322 0.315 0.309 0.304]; eig [0.11 +0.027j 0.11 -0.027j 0.112+0.j 0.075+0.071j 0.075-0.071j\n 0.102+0.j ]; key/query bias norms 0.263 0.253; effective OV norm 25.724 trace -49.613\nL3H2 effective QK norm 2.0608 SV [0.434 0.411 0.392 0.362]; eig [0.036+0.114j 0.036-0.114j 0.114-0.026j 0.114+0.026j 0.003+0.11j\n 0.003-0.11j ]; key/query bias norms 0.342 0.286; effective OV norm 26.860 trace -50.565\nL3H3 effective QK norm 1.0530 SV [0.262 0.237 0.208 0.202]; eig [ 0.027+0.061j 0.027-0.061j 0.06 +0.j 0.026+0.049j 0.026-0.049j\n -0.002-0.051j]; key/query bias norms 0.258 0.160; effective OV norm 37.938 trace -51.153\nL3H4 effective QK norm 1.7344 SV [0.401 0.369 0.357 0.323]; eig [-0.105+0.j 0.093+0.j 0.066+0.059j 0.066-0.059j 0.029+0.079j\n 0.029-0.079j]; key/query bias norms 0.307 0.239; effective OV norm 28.440 trace -35.013\nL3H5 effective QK norm 2.1169 SV [0.404 0.39 0.384 0.37 ]; eig [0.195+0.117j 0.195-0.117j 0.213-0.074j 0.213+0.074j 0.2 +0.104j\n 0.2 -0.104j]; key/query bias norms 0.310 0.307; effective OV norm 23.687 trace -56.843\n\nNORMALIZED TRACE(QK_l @ OV_earlier), columns earlier heads\nQK layer 1 OV layer 0\n[[ 0.007 -0. -0.008 0.005 0.021 -0.003]\n [-0.005 0.007 -0.016 0.006 0.019 0.015]\n [ 0.004 0.001 -0.017 0.009 0.033 -0.003]\n [ 0.008 -0.003 -0.013 0.006 0.042 0.012]\n [ 0.009 0.002 -0.018 0.011 0.044 0.009]\n [-0.019 0.005 -0.013 -0. -0.004 0.013]]\nQK layer 2 OV layer 0\n[[ 0.005 -0.001 -0.003 0.008 0.01 -0.005]\n [ 0.007 0.001 0.011 0.003 0.004 0.078]\n [ 0.011 -0.017 -0.005 0.001 0.02 -0.039]\n [ 0.018 -0.001 -0.007 0.007 0.04 -0.003]\n [ 0.008 -0.006 -0.004 0.01 0.025 0.002]\n [ 0.009 0.001 0. 0.005 0.019 0.007]]\nQK layer 2 OV layer 1\n[[-0.013 0.004 -0. 0.002 -0. 0.006]\n [ 0.003 0.048 0.048 0.057 0.048 0.035]\n [-0.042 -0.016 -0.043 -0.007 -0.023 -0.007]\n [-0.02 0.005 -0.016 0.004 -0.005 0.004]\n [-0.014 -0.008 -0.011 -0.002 -0.004 -0.006]\n [ 0. 0.008 0.009 0.006 0.007 0.007]]\nQK layer 3 OV layer 0\n[[ 0.01 -0.004 0.007 0.008 0.01 0.019]\n [ 0.008 0.007 0.006 0.001 0.004 0.061]\n [ 0.006 -0.003 0.002 -0.001 0.013 -0.001]\n [ 0.004 0.003 0.005 0.003 0.007 0.001]\n [ 0.009 -0.006 0.006 0.003 0.012 0.01 ]\n [ 0.006 -0.009 0. 0.003 0.003 0.071]]\nQK layer 3 OV layer 1\n[[ 0.008 0.008 0.01 0.006 0.006 0.001]\n [ 0.013 0.033 0.038 0.045 0.046 0.027]\n [-0.012 0.001 -0.009 0. -0.009 -0.004]\n [ 0.005 0.002 0.008 -0. 0.002 0.004]\n [ 0. 0.007 0.003 0.005 0.005 0.008]\n [ 0.004 0.042 0.048 0.068 0.057 0.02 ]]\nQK layer 3 OV layer 2\n[[-0.003 -0.013 0.024 -0.001 0.004 -0.009]\n [ 0.015 -0.018 0.11 0.021 0.005 -0. ]\n [-0.005 -0.007 -0. -0.003 -0.001 -0.003]\n [ 0.006 -0.005 0.009 0.01 0.015 0.005]\n [ 0.001 -0.008 0.008 0.004 0.01 -0.004]\n [-0.008 -0.06 0.184 0.055 -0.022 -0.022]]\n\nL0: overlap of top QK singular subspaces with top value-read subspace\nCombined effective V0 singular values: [3.195 2.304 1.316 0.88 0.809 0.771 0.73 0.704 0.685 0.662 0.638 0.629\n 0.623 0.621 0.604 0.595 0.582 0.567 0.554 0.55 0.542 0.541 0.522 0.513\n 0.496 0.487 0.479 0.475 0.464 0.457 0.447 0.431 0.428 0.424 0.41 0.399\n 0.389 0.383 0.373 0.369 0.358 0.345 0.334 0.333 0.319]\nL0H0 V-rank8 query/key energy 0.744 0.374\nL0H0 V-rank24 query/key energy 0.794 0.603\nL0H1 V-rank8 query/key energy 0.590 0.611\nL0H1 V-rank24 query/key energy 0.630 0.652\nL0H2 V-rank8 query/key energy 0.712 0.559\nL0H2 V-rank24 query/key energy 0.778 0.732\nL0H3 V-rank8 query/key energy 0.632 0.587\nL0H3 V-rank24 query/key energy 0.691 0.746\nL0H4 V-rank8 query/key energy 0.050 0.099\nL0H4 V-rank24 query/key energy 0.116 0.193\nL0H5 V-rank8 query/key energy 0.276 0.288\nL0H5 V-rank24 query/key energy 0.320 0.336\n\nL0 QK pairwise Frobenius cosines\n[[ 1. 0.179 0.112 0.206 0.063 0.093]\n [ 0.179 1. 0.21 -0.15 0.005 0.414]\n [ 0.112 0.21 1. -0.247 -0.006 0.114]\n [ 0.206 -0.15 -0.247 1. -0.027 -0.098]\n [ 0.063 0.005 -0.006 -0.027 1. -0.004]\n [ 0.093 0.414 0.114 -0.098 -0.004 1. ]]\n\nL0 V-head top-24 right-subspace overlaps (mean squared cosine)\n[[1. 0.219 0.254 0.277 0.347 0.217]\n [0.219 1. 0.325 0.314 0.324 0.619]\n [0.254 0.325 1. 0.296 0.25 0.291]\n [0.277 0.314 0.296 1. 0.28 0.264]\n [0.347 0.324 0.25 0.28 1. 0.35 ]\n [0.217 0.619 0.291 0.264 0.35 1. ]]\n",
"seconds": 1.507,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "8871359573c6d2bae904a7e9cbcb7f81dfc5fee3ce43b1cb6e3cba7aa3c9e6a0",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}L0H0 effective QK norm 0.1137 SV [0.089 0.068 0.013 0.008]; eig [-0.049+0.j 0.006+0.009j 0.006-0.009j -0.003+0.j -0.001+0.002j -0.001-0.002j]; key/query bias norms 0.018 0.016; effective OV norm 4.321 trace 1.666 L0H1 effective QK norm 0.8775 SV [0.601 0.373 0.297 0.211]; eig [ 0.218+0.345j 0.218-0.345j -0.284+0.j -0.123+0.15j -0.123-0.15j -0.097+0.098j]; key/query bias norms 0.074 0.106; effective OV norm 3.488 trace 0.164 L0H2 effective QK norm 0.1745 SV [0.132 0.08 0.044 0.031]; eig [ 0.091+0.j 0.053+0.j -0.022+0.j -0.013+0.001j -0.013-0.001j 0.011+0.j ]; key/query bias norms 0.023 0.079; effective OV norm 2.023 trace 0.015 L0H3 effective QK norm 0.2395 SV [0.165 0.14 0.081 0.031]; eig [-0.078+0.005j -0.078-0.005j -0.02 +0.j 0.006+0.017j 0.006-0.017j 0.007+0.j ]; key/query bias norms 0.057 0.149; effective OV norm 1.621 trace 0.242 L0H4 effective QK norm 0.0093 SV [0.004 0.003 0.002 0.002]; eig [-0.001+0.001j -0.001-0.001j -0.001+0.j -0.001-0.j 0. +0.001j 0. -0.001j]; key/query bias norms 0.001 0.001; effective OV norm 17.158 trace 6.049 L0H5 effective QK norm 0.9704 SV [0.41 0.362 0.273 0.254]; eig [0.207+0.275j 0.207-0.275j 0.114+0.172j 0.114-0.172j 0.178+0.j 0.155+0.084j]; key/query bias norms 0.051 0.066; effective OV norm 3.707 trace -2.463 L1H0 effective QK norm 0.6440 SV [0.354 0.252 0.218 0.167]; eig [ 0.028+0.051j 0.028-0.051j -0.051+0.023j -0.051-0.023j -0.051+0.j -0.041+0.013j]; key/query bias norms 0.839 0.841; effective OV norm 13.436 trace -16.299 L1H1 effective QK norm 1.5843 SV [1.146 0.661 0.459 0.326]; eig [ 0.08 +0.26j 0.08 -0.26j -0.144+0.009j -0.144-0.009j -0.092+0.079j -0.092-0.079j]; key/query bias norms 3.150 2.183; effective OV norm 9.592 trace -11.205 L1H2 effective QK norm 1.2358 SV [0.807 0.693 0.332 0.205]; eig [ 0.057+0.202j 0.057-0.202j -0.108+0.j -0.07 +0.05j -0.07 -0.05j -0.079+0.004j]; key/query bias norms 2.200 1.897; effective OV norm 11.496 trace -14.892 L1H3 effective QK norm 1.3094 SV [0.913 0.699 0.309 0.202]; eig [ 0.076+0.272j 0.076-0.272j -0.094+0.j 0.092+0.j -0.08 +0.024j -0.08 -0.024j]; key/query bias norms 2.719 2.207; effective OV norm 10.043 trace -10.571 L1H4 effective QK norm 1.2492 SV [0.848 0.654 0.34 0.213]; eig [ 0.071+0.239j 0.071-0.239j -0.085+0.014j -0.085-0.014j -0.078+0.j -0.065+0.028j]; key/query bias norms 2.518 1.942; effective OV norm 9.557 trace -10.446 L1H5 effective QK norm 1.4614 SV [1.009 0.555 0.482 0.45 ]; eig [ 0.072+0.204j 0.072-0.204j -0.156+0.j -0.124+0.j -0.083+0.053j -0.083-0.053j]; key/query bias norms 2.460 1.810; effective OV norm 9.826 trace -13.045 L2H0 effective QK norm 2.0456 SV [0.518 0.485 0.459 0.416]; eig [ 0.033+0.132j 0.033-0.132j -0.117+0.j 0.091+0.074j 0.091-0.074j 0.037+0.104j]; key/query bias norms 0.895 0.774; effective OV norm 18.698 trace -35.970 L2H1 effective QK norm 1.9454 SV [0.458 0.421 0.396 0.393]; eig [0.142+0.036j 0.142-0.036j 0.142+0.j 0.141+0.011j 0.141-0.011j 0.131+0.022j]; key/query bias norms 0.638 0.333; effective OV norm 21.448 trace -40.225 L2H2 effective QK norm 2.8040 SV [0.709 0.693 0.676 0.634]; eig [0.259+0.j 0.08 +0.216j 0.08 -0.216j 0.193+0.058j 0.193-0.058j 0.195-0.026j]; key/query bias norms 1.584 1.581; effective OV norm 16.782 trace -29.462 L2H3 effective QK norm 1.9759 SV [0.571 0.423 0.412 0.392]; eig [ 0.058+0.185j 0.058-0.185j 0.002+0.106j 0.002-0.106j -0.055+0.085j -0.055-0.085j]; key/query bias norms 1.037 1.265; effective OV norm 20.617 trace -38.568 L2H4 effective QK norm 2.2016 SV [0.783 0.587 0.484 0.429]; eig [ 0.08 +0.22j 0.08 -0.22j 0.154+0.j -0.121+0.j 0.086+0.077j 0.086-0.077j]; key/query bias norms 1.199 1.340; effective OV norm 18.415 trace -38.753 L2H5 effective QK norm 1.6541 SV [0.419 0.414 0.359 0.32 ]; eig [-0.107+0.j -0.007+0.101j -0.007-0.101j -0.079+0.045j -0.079-0.045j 0.09 +0.j ]; key/query bias norms 0.636 0.492; effective OV norm 23.766 trace -33.940 L3H0 effective QK norm 2.0777 SV [0.405 0.399 0.385 0.377]; eig [0.13 +0.j 0.119-0.04j 0.119+0.04j 0.119-0.018j 0.119+0.018j 0.109+0.j ]; key/query bias norms 0.299 0.302; effective OV norm 24.502 trace -48.319 L3H1 effective QK norm 1.6948 SV [0.322 0.315 0.309 0.304]; eig [0.11 +0.027j 0.11 -0.027j 0.112+0.j 0.075+0.071j 0.075-0.071j 0.102+0.j ]; key/query bias norms 0.263 0.253; effective OV norm 25.724 trace -49.613 L3H2 effective QK norm 2.0608 SV [0.434 0.411 0.392 0.362]; eig [0.036+0.114j 0.036-0.114j 0.114-0.026j 0.114+0.026j 0.003+0.11j 0.003-0.11j ]; key/query bias norms 0.342 0.286; effective OV norm 26.860 trace -50.565 L3H3 effective QK norm 1.0530 SV [0.262 0.237 0.208 0.202]; eig [ 0.027+0.061j 0.027-0.061j 0.06 +0.j 0.026+0.049j 0.026-0.049j -0.002-0.051j]; key/query bias norms 0.258 0.160; effective OV norm 37.938 trace -51.153 L3H4 effective QK norm 1.7344 SV [0.401 0.369 0.357 0.323]; eig [-0.105+0.j 0.093+0.j 0.066+0.059j 0.066-0.059j 0.029+0.079j 0.029-0.079j]; key/query bias norms 0.307 0.239; effective OV norm 28.440 trace -35.013 L3H5 effective QK norm 2.1169 SV [0.404 0.39 0.384 0.37 ]; eig [0.195+0.117j 0.195-0.117j 0.213-0.074j 0.213+0.074j 0.2 +0.104j 0.2 -0.104j]; key/query bias norms 0.310 0.307; effective OV norm 23.687 trace -56.843 NORMALIZED TRACE(QK_l @ OV_earlier), columns earlier heads QK layer 1 OV layer 0 [[ 0.007 -0. -0.008 0.005 0.021 -0.003] [-0.005 0.007 -0.016 0.006 0.019 0.015] [ 0.004 0.001 -0.017 0.009 0.033 -0.003] [ 0.008 -0.003 -0.013 0.006 0.042 0.012] [ 0.009 0.002 -0.018 0.011 0.044 0.009] [-0.019 0.005 -0.013 -0. -0.004 0.013]] QK layer 2 OV layer 0 [[ 0.005 -0.001 -0.003 0.008 0.01 -0.005] [ 0.007 0.001 0.011 0.003 0.004 0.078] [ 0.011 -0.017 -0.005 0.001 0.02 -0.039] [ 0.018 -0.001 -0.007 0.007 0.04 -0.003] [ 0.008 -0.006 -0.004 0.01 0.025 0.002] [ 0.009 0.001 0. 0.005 0.019 0.007]] QK layer 2 OV layer 1 [[-0.013 0.004 -0. 0.002 -0. 0.006] [ 0.003 0.048 0.048 0.057 0.048 0.035] [-0.042 -0.016 -0.043 -0.007 -0.023 -0.007] [-0.02 0.005 -0.016 0.004 -0.005 0.004] [-0.014 -0.008 -0.011 -0.002 -0.004 -0.006] [ 0. 0.008 0.009 0.006 0.007 0.007]] QK layer 3 OV layer 0 [[ 0.01 -0.004 0.007 0.008 0.01 0.019] [ 0.008 0.007 0.006 0.001 0.004 0.061] [ 0.006 -0.003 0.002 -0.001 0.013 -0.001] [ 0.004 0.003 0.005 0.003 0.007 0.001] [ 0.009 -0.006 0.006 0.003 0.012 0.01 ] [ 0.006 -0.009 0. 0.003 0.003 0.071]] QK layer 3 OV layer 1 [[ 0.008 0.008 0.01 0.006 0.006 0.001] [ 0.013 0.033 0.038 0.045 0.046 0.027] [-0.012 0.001 -0.009 0. -0.009 -0.004] [ 0.005 0.002 0.008 -0. 0.002 0.004] [ 0. 0.007 0.003 0.005 0.005 0.008] [ 0.004 0.042 0.048 0.068 0.057 0.02 ]] QK layer 3 OV layer 2 [[-0.003 -0.013 0.024 -0.001 0.004 -0.009] [ 0.015 -0.018 0.11 0.021 0.005 -0. ] [-0.005 -0.007 -0. -0.003 -0.001 -0.003] [ 0.006 -0.005 0.009 0.01 0.015 0.005] [ 0.001 -0.008 0.008 0.004 0.01 -0.004] [-0.008 -0.06 0.184 0.055 -0.022 -0.022]] L0: overlap of top QK singular subspaces with top value-read subspace Combined effective V0 singular values: [3.195 2.304 1.316 0.88 0.809 0.771 0.73 0.704 0.685 0.662 0.638 0.629 0.623 0.621 0.604 0.595 0.582 0.567 0.554 0.55 0.542 0.541 0.522 0.513 0.496 0.487 0.479 0.475 0.464 0.457 0.447 0.431 0.428 0.424 0.41 0.399 0.389 0.383 0.373 0.369 0.358 0.345 0.334 0.333 0.319] L0H0 V-rank8 query/key energy 0.744 0.374 L0H0 V-rank24 query/key energy 0.794 0.603 L0H1 V-rank8 query/key energy 0.590 0.611 L0H1 V-rank24 query/key energy 0.630 0.652 L0H2 V-rank8 query/key energy 0.712 0.559 L0H2 V-rank24 query/key energy 0.778 0.732 L0H3 V-rank8 query/key energy 0.632 0.587 L0H3 V-rank24 query/key energy 0.691 0.746 L0H4 V-rank8 query/key energy 0.050 0.099 L0H4 V-rank24 query/key energy 0.116 0.193 L0H5 V-rank8 query/key energy 0.276 0.288 L0H5 V-rank24 query/key energy 0.320 0.336 L0 QK pairwise Frobenius cosines [[ 1. 0.179 0.112 0.206 0.063 0.093] [ 0.179 1. 0.21 -0.15 0.005 0.414] [ 0.112 0.21 1. -0.247 -0.006 0.114] [ 0.206 -0.15 -0.247 1. -0.027 -0.098] [ 0.063 0.005 -0.006 -0.027 1. -0.004] [ 0.093 0.414 0.114 -0.098 -0.004 1. ]] L0 V-head top-24 right-subspace overlaps (mean squared cosine) [[1. 0.219 0.254 0.277 0.347 0.217] [0.219 1. 0.325 0.314 0.324 0.619] [0.254 0.325 1. 0.296 0.25 0.291] [0.277 0.314 0.296 1. 0.28 0.264] [0.347 0.324 0.25 0.28 1. 0.35 ] [0.217 0.619 0.291 0.264 0.35 1. ]]
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 2 of at most 100: ``` L0H0 effective QK norm 0.1137 SV [0.089 0.068 0.013 0.008]; eig [-0.049+0.j 0.006+0.009j 0.006-0.009j -0.003+0.j -0.001+0.002j -0.001-0.002j]; key/query bias norms 0.018 0.016; effective OV norm 4.321 trace 1.666 L0H1 effective QK norm 0.8775 SV [0.601 0.373 0.297 0.211]; eig [ 0.218+0.345j 0.218-0.345j -0.284+0.j -0.123+0.15j -0.123-0.15j -0.097+0.098j]; key/query bias norms 0.074 0.106; effective OV norm 3.488 trace 0.164 L0H2 effective QK norm 0.1745 SV [0.132 0.08 0.044 0.031]; eig [ 0.091+0.j 0.053+0.j -0.022+0.j -0.013+0.001j -0.013-0.001j 0.011+0.j ]; key/query bias norms 0.023 0.079; effective OV norm 2.023 trace 0.015 L0H3 effective QK norm 0.2395 SV [0.165 0.14 0.081 0.031]; eig [-0.078+0.005j -0.078-0.005j -0.02 +0.j 0.006+0.017j 0.006-0.017j 0.007+0.j ]; key/query bias norms 0.057 0.149; effective OV norm 1.621 trace 0.242 L0H4 effective QK norm 0.0093 SV [0.004 0.003 0.002 0.002]; eig [-0.001+0.001j -0.001-0.001j -0.001+0.j -0.001-0.j 0. +0.001j 0. -0.001j]; key/query bias norms 0.001 0.001; effective OV norm 17.158 trace 6.049 L0H5 effective QK norm 0.9704 SV [0.41 0.362 0.273 0.254]; eig [0.207+0.275j 0.207-0.275j 0.114+0.172j 0.114-0.172j 0.178+0.j 0.155+0.084j]; key/query bias norms 0.051 0.066; effective OV norm 3.707 trace -2.463 L1H0 effective QK norm 0.6440 SV [0.354 0.252 0.218 0.167]; eig [ 0.028+0.051j 0.028-0.051j -0.051+0.023j -0.051-0.023j -0.051+0.j -0.041+0.013j]; key/query bias norms 0.839 0.841; effective OV norm 13.436 trace -16.299 L1H1 effective QK norm 1.5843 SV [1.146 0.661 0.459 0.326]; eig [ 0.08 +0.26j 0.08 -0.26j -0.144+0.009j -0.144-0.009j -0.092+0.079j -0.092-0.079j]; key/query bias norms 3.150 2.183; effective OV norm 9.592 trace -11.205 L1H2 effective QK norm 1.2358 SV [0.807 0.693 0.332 0.205]; eig [ 0.057+0.202j 0.057-0.202j -0.108+0.j -0.07 +0.05j -0.07 -0.05j -0.079+0.004j]; key/query bias norms 2.200 1.897; effective OV norm 11.496 trace -14.892 L1H3 effective QK norm 1.3094 SV [0.913 0.699 0.309 0.202]; eig [ 0.076+0.272j 0.076-0.272j -0.094+0.j 0.092+0.j -0.08 +0.024j -0.08 -0.024j]; key/query bias norms 2.719 2.207; effective OV norm 10.043 trace -10.571 L1H4 effective QK norm 1.2492 SV [0.848 0.654 0.34 0.213]; eig [ 0.071+0.239j 0.071-0.239j -0.085+0.014j -0.085-0.014j -0.078+0.j -0.065+0.028j]; key/query bias norms 2.518 1.942; effective OV norm 9.557 trace -10.446 L1H5 effective QK norm 1.4614 SV [1.009 0.555 0.482 0.45 ]; eig [ 0.072+0.204j 0.072-0.204j -0.156+0.j -0.124+0.j -0.083+0.053j -0.083-0.053j]; key/query bias norms 2.460 1.810; effective OV norm 9.826 trace -13.045 L2H0 effective QK norm 2.0456 SV [0.518 0.485 0.459 0.416]; eig [ 0.033+0.132j 0.033-0.132j -0.117+0.j 0.091+0.074j 0.091-0.074j 0.037+0.104j]; key/query bias norms 0.895 0.774; effective OV norm 18.698 trace -35.970 L2H1 effective QK norm 1.9454 SV [0.458 0.421 0.396 0.393]; eig [0.142+0.036j 0.142-0.036j 0.142+0.j 0.141+0.011j 0.141-0.011j 0.131+0.022j]; key/query bias norms 0.638 0.333; effective OV norm 21.448 trace -40.225 L2H2 effective QK norm 2.8040 SV [0.709 0.693 0.676 0.634]; eig [0.259+0.j 0.08 +0.216j 0.08 -0.216j 0.193+0.058j 0.193-0.058j 0.195-0.026j]; key/query bias norms 1.584 1.581; effective OV norm 16.782 trace -29.462 L2H3 effective QK norm 1.9759 SV [0.571 0.423 0.412 0.392]; eig [ 0.058+0.185j 0.058-0.185j 0.002+0.106j 0.002-0.106j -0.055+0.085j -0.055-0.085j]; key/query bias norms 1.037 1.265; effective OV norm 20.617 trace -38.568 L2H4 effective QK norm 2.2016 SV [0.783 0.587 0.484 0.429]; eig [ 0.08 +0.22j 0.08 -0.22j 0.154+0.j -0.121+0.j 0.086+0.077j 0.086-0.077j]; key/query bias norms 1.199 1.340; effective OV norm 18.415 trace -38.753 L2H5 effective QK norm 1.6541 SV [0.419 0.414 0.359 0.32 ]; eig [-0.107+0.j -0.007+0.101j -0.007-0.101j -0.079+0.045j -0.079-0.045j 0.09 +0.j ]; key/query bias norms 0.636 0.492; effective OV norm 23.766 trace -33.940 L3H0 effective QK norm 2.0777 SV [0.405 0.399 0.385 0.377]; eig [0.13 +0.j 0.119-0.04j 0.119+0.04j 0.119-0.018j 0.119+0.018j 0.109+0.j ]; key/query bias norms 0.299 0.302; effective OV norm 24.502 trace -48.319 L3H1 effective QK norm 1.6948 SV [0.322 0.315 0.309 0.304]; eig [0.11 +0.027j 0.11 -0.027j 0.112+0.j 0.075+0.071j 0.075-0.071j 0.102+0.j ]; key/query bias norms 0.263 0.253; effective OV norm 25.724 trace -49.613 L3H2 effective QK norm 2.0608 SV [0.434 0.411 0.392 0.362]; eig [0.036+0.114j 0.036-0.114j 0.114-0.026j 0.114+0.026j 0.003+0.11j 0.003-0.11j ]; key/query bias norms 0.342 0.286; effective OV norm 26.860 trace -50.565 L3H3 effective QK norm 1.0530 SV [0.262 0.237 0.208 0.202]; eig [ 0.027+0.061j 0.027-0.061j 0.06 +0.j 0.026+0.049j 0.026-0.049j -0.002-0.051j]; key/query bias norms 0.258 0.160; effective OV norm 37.938 trace -51.153 L3H4 effective QK norm 1.7344 SV [0.401 0.369 0.357 0.323]; eig [-0.105+0.j 0.093+0.j 0.066+0.059j 0.066-0.059j 0.029+0.079j 0.029-0.079j]; key/query bias norms 0.307 0.239; effective OV norm 28.440 trace -35.013 L3H5 effective QK norm 2.1169 SV [0.404 0.39 0.384 0.37 ]; eig [0.195+0.117j 0.195-0.117j 0.213-0.074j 0.213+0.074j 0.2 +0.104j 0.2 -0.104j]; key/query bias norms 0.310 0.307; effective OV norm 23.687 trace -56.843 NORMALIZED TRACE(QK_l @ OV_earlier), columns earlier heads QK layer 1 OV layer 0 [[ 0.007 -0. -0.008 0.005 0.021 -0.003] [-0.005 0.007 -0.016 0.006 0.019 0.015] [ 0.004 0.001 -0.017 0.009 0.033 -0.003] [ 0.008 -0.003 -0.013 0.006 0.042 0.012] [ 0.009 0.002 -0.018 0.011 0.044 0.009] [-0.019 0.005 -0.013 -0. -0.004 0.013]] QK layer 2 OV layer 0 [[ 0.005 -0.001 -0.003 0.008 0.01 -0.005] [ 0.007 0.001 0.011 0.003 0.004 0.078] [ 0.011 -0.017 -0.005 0.001 0.02 -0.039] [ 0.018 -0.001 -0.007 0.007 0.04 -0.003] [ 0.008 -0.006 -0.004 0.01 0.025 0.002] [ 0.009 0.001 0. 0.005 0.019 0.007]] QK layer 2 OV layer 1 [[-0.013 0.004 -0. 0.002 -0. 0.006] [ 0.003 0.048 0.048 0.057 0.048 0.035] [-0.042 -0.016 -0.043 -0.007 -0.023 -0.007] [-0.02 0.005 -0.016 0.004 -0.005 0.004] [-0.014 -0.008 -0.011 -0.002 -0.004 -0.006] [ 0. 0.008 0.009 0.006 0.007 0.007]] QK layer 3 OV layer 0 [[ 0.01 -0.004 0.007 0.008 0.01 0.019] [ 0.008 0.007 0.006 0.001 0.004 0.061] [ 0.006 -0.003 0.002 -0.001 0.013 -0.001] [ 0.004 0.003 0.005 0.003 0.007 0.001] [ 0.009 -0.006 0.006 0.003 0.012 0.01 ] [ 0.006 -0.009 0. 0.003 0.003 0.071]] QK layer 3 OV layer 1 [[ 0.008 0.008 0.01 0.006 0.006 0.001] [ 0.013 0.033 0.038 0.045 0.046 0.027] [-0.012 0.001 -0.009 0. -0.009 -0.004] [ 0.005 0.002 0.008 -0. 0.002 0.004] [ 0. 0.007 0.003 0.005 0.005 0.008] [ 0.004 0.042 0.048 0.068 0.057 0.02 ]] QK layer 3 OV layer 2 [[-0.003 -0.013 0.024 -0.001 0.004 -0.009] [ 0.015 -0.018 0.11 0.021 0.005 -0. ] [-0.005 -0.007 -0. -0.003 -0.001 -0.003] [ 0.006 -0.005 0.009 0.01 0.015 0.005] [ 0.001 -0.008 0.008 0.004 0.01 -0.004] [-0.008 -0.06 0.184 0.055 -0.022 -0.022]] L0: overlap of top QK singular subspaces with top value-read subspace Combined effective V0 singular values: [3.195 2.304 1.316 0.88 0.809 0.771 0.73 0.704 0.685 0.662 0.638 0.629 0.623 0.621 0.604 0.595 0.582 0.567 0.554 0.55 0.542 0.541 0.522 0.513 0.496 0.487 0.479 0.475 0.464 0.457 0.447 0.431 0.428 0.424 0.41 0.399 0.389 0.383 0.373 0.369 0.358 0.345 0.334 0.333 0.319] L0H0 V-rank8 query/key energy 0.744 0.374 L0H0 V-rank24 query/key energy 0.794 0.603 L0H1 V-rank8 query/key energy 0.590 0.611 L0H1 V-rank24 query/key energy 0.630 0.652 L0H2 V-rank8 query/key energy 0.712 0.559 L0H2 V-rank24 query/key energy 0.778 0.732 L0H3 V-rank8 query/key energy 0.632 0.587 L0H3 V-rank24 query/key energy 0.691 0.746 L0H4 V-rank8 query/key energy 0.050 0.099 L0H4 V-rank24 query/key energy 0.116 0.193 L0H5 V-rank8 query/key energy 0.276 0.288 L0H5 V-rank24 query/key energy 0.320 0.336 L0 QK pairwise Frobenius cosines [[ 1. 0.179 0.112 0.206 0.063 0.093] [ 0.179 1. 0.21 -0.15 0.005 0.414] [ 0.112 0.21 1. -0.247 -0.006 0.114] [ 0.206 -0.15 -0.247 1. -0.027 -0.098] [ 0.063 0.005 -0.006 -0.027 1. -0.004] [ 0.093 0.414 0.114 -0.098 -0.004 1. ]] L0 V-head top-24 right-subspace overlaps (mean squared cosine) [[1. 0.219 0.254 0.277 0.347 0.217] [0.219 1. 0.325 0.314 0.324 0.619] [0.254 0.325 1. 0.296 0.25 0.291] [0.277 0.314 0.296 1. 0.28 0.264] [0.347 0.324 0.25 0.28 1. 0.35 ] [0.217 0.619 0.291 0.264 0.35 1. ]] ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svd
from threadpoolctl import threadpool_limits
threadpool_limits(limits=2)
np.set_printoptions(precision=4, suppress=True, linewidth=150)
for name in ["W_fc.0", "W_proj.0", "W_fc.1", "W_proj.1", "W_fc.2", "W_proj.2", "W_fc.3", "W_proj.3"]:
a = W[name].astype(float)
s = svd(a, compute_uv=False, check_finite=False)
print("\n", name, "first 48 singular values:\n", s[:48])
print("Energy fractions at ranks 16,20,22,24,26,28,32,64:",
[round(float(np.sum(s[:r]**2)/np.sum(s**2)), 4) for r in [16,20,22,24,26,28,32,64]])
# First-layer MLP row geometry: no neuron is evaluated.
f = W["W_fc.0"].astype(float) * W["ln2_w.0"][None, :]
f -= f.mean(axis=1, keepdims=True)
norms = np.linalg.norm(f, axis=1)
unit = f / norms[:, None]
cos = unit @ unit.T
np.fill_diagonal(cos, -1)
print("\nMLP0 effective row norm quantiles:", np.quantile(norms, [0,.1,.25,.5,.75,.9,1]))
print("MLP0 largest off-diagonal row cosine quantiles:",
np.quantile(cos.max(axis=1), [0,.1,.25,.5,.75,.9,1]))
# Shared affine coefficient geometry in layer 1.
As, ks, qs = [], [], []
for j in range(6):
sl = slice(64*j, 64*(j+1))
q0, k0 = W["W_q.1"][sl].astype(float), W["W_k.1"][sl].astype(float)
q = q0 * W["ln1_w.1"][None, :]
k = k0 * W["ln1_w.1"][None, :]
q -= q.mean(axis=1, keepdims=True)
k -= k.mean(axis=1, keepdims=True)
As.append(q.T @ k / 8)
ks.append(k.T @ (q0 @ W["ln1_b.1"]) / 8)
qs.append(q.T @ (k0 @ W["ln1_b.1"]) / 8)
def cosines(xs):
z = np.stack([x.ravel() for x in xs])
z /= np.linalg.norm(z, axis=1, keepdims=True)
return z @ z.T
print("\nL1 QK coefficient cosines:\n", cosines(As))
print("L1 key-linear coefficient cosines:\n", cosines(ks))
print("L1 query-linear coefficient cosines:\n", cosines(qs))
print("L1 key-linear versus query-linear cosines:\n",
np.array([[np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) for b in qs] for a in ks]))
# For each layer, singular spectrum of the six-head stack of bias-derived
# key-only coefficient vectors; strong shared directions suggest common routing.
for l in range(4):
coeff = []
for j in range(6):
sl = slice(64*j,64*(j+1))
q0, k0 = W[f"W_q.{l}"][sl].astype(float), W[f"W_k.{l}"][sl].astype(float)
k = k0 * W[f"ln1_w.{l}"][None,:]
k -= k.mean(axis=1, keepdims=True)
coeff.append(k.T @ (q0 @ W[f"ln1_b.{l}"]) / 8)
print("Layer", l, "key-only coefficient stack SV:",
svd(np.stack(coeff), compute_uv=False, check_finite=False))
```
{
"run": 3,
"turn": "t003",
"code": "import numpy as np\nfrom scipy.linalg import svd\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(limits=2)\nnp.set_printoptions(precision=4, suppress=True, linewidth=150)\n\nfor name in [\"W_fc.0\", \"W_proj.0\", \"W_fc.1\", \"W_proj.1\", \"W_fc.2\", \"W_proj.2\", \"W_fc.3\", \"W_proj.3\"]:\n a = W[name].astype(float)\n s = svd(a, compute_uv=False, check_finite=False)\n print(\"\\n\", name, \"first 48 singular values:\\n\", s[:48])\n print(\"Energy fractions at ranks 16,20,22,24,26,28,32,64:\",\n [round(float(np.sum(s[:r]**2)/np.sum(s**2)), 4) for r in [16,20,22,24,26,28,32,64]])\n\n# First-layer MLP row geometry: no neuron is evaluated.\nf = W[\"W_fc.0\"].astype(float) * W[\"ln2_w.0\"][None, :]\nf -= f.mean(axis=1, keepdims=True)\nnorms = np.linalg.norm(f, axis=1)\nunit = f / norms[:, None]\ncos = unit @ unit.T\nnp.fill_diagonal(cos, -1)\nprint(\"\\nMLP0 effective row norm quantiles:\", np.quantile(norms, [0,.1,.25,.5,.75,.9,1]))\nprint(\"MLP0 largest off-diagonal row cosine quantiles:\",\n np.quantile(cos.max(axis=1), [0,.1,.25,.5,.75,.9,1]))\n\n# Shared affine coefficient geometry in layer 1.\nAs, ks, qs = [], [], []\nfor j in range(6):\n sl = slice(64*j, 64*(j+1))\n q0, k0 = W[\"W_q.1\"][sl].astype(float), W[\"W_k.1\"][sl].astype(float)\n q = q0 * W[\"ln1_w.1\"][None, :]\n k = k0 * W[\"ln1_w.1\"][None, :]\n q -= q.mean(axis=1, keepdims=True)\n k -= k.mean(axis=1, keepdims=True)\n As.append(q.T @ k / 8)\n ks.append(k.T @ (q0 @ W[\"ln1_b.1\"]) / 8)\n qs.append(q.T @ (k0 @ W[\"ln1_b.1\"]) / 8)\n\ndef cosines(xs):\n z = np.stack([x.ravel() for x in xs])\n z /= np.linalg.norm(z, axis=1, keepdims=True)\n return z @ z.T\n\nprint(\"\\nL1 QK coefficient cosines:\\n\", cosines(As))\nprint(\"L1 key-linear coefficient cosines:\\n\", cosines(ks))\nprint(\"L1 query-linear coefficient cosines:\\n\", cosines(qs))\nprint(\"L1 key-linear versus query-linear cosines:\\n\",\n np.array([[np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) for b in qs] for a in ks]))\n\n# For each layer, singular spectrum of the six-head stack of bias-derived\n# key-only coefficient vectors; strong shared directions suggest common routing.\nfor l in range(4):\n coeff = []\n for j in range(6):\n sl = slice(64*j,64*(j+1))\n q0, k0 = W[f\"W_q.{l}\"][sl].astype(float), W[f\"W_k.{l}\"][sl].astype(float)\n k = k0 * W[f\"ln1_w.{l}\"][None,:]\n k -= k.mean(axis=1, keepdims=True)\n coeff.append(k.T @ (q0 @ W[f\"ln1_b.{l}\"]) / 8)\n print(\"Layer\", l, \"key-only coefficient stack SV:\",\n svd(np.stack(coeff), compute_uv=False, check_finite=False))",
"out": "\n W_fc.0 first 48 singular values:\n [18.1104 12.2798 8.4726 7.9016 7.7793 7.442 7.1458 7.1094 7.0083 6.8053 6.7211 6.5398 6.3779 6.3286 6.2698 6.2168 6.2037 6.0345\n 6.0235 5.9437 5.8446 5.7365 5.7016 5.6681 5.6093 5.5798 5.498 5.4472 5.3773 5.3341 5.2437 5.203 5.1406 5.1261 4.9896 4.9434\n 4.9202 4.8457 4.7874 4.7161 4.6694 4.6008 4.5188 4.4395 4.4082 4.3845 4.3074 4.2416]\nEnergy fractions at ranks 16,20,22,24,26,28,32,64: [0.4395, 0.4945, 0.5196, 0.5438, 0.5673, 0.5898, 0.6317, 0.8472]\n\n W_proj.0 first 48 singular values:\n [10.9319 7.576 6.6285 6.2932 6.166 5.9239 5.886 5.7686 5.7059 5.6097 5.4949 5.4626 5.4044 5.3131 5.1933 5.1607 5.1191 5.0699\n 5.0033 4.9374 4.8767 4.7926 4.6927 4.6637 4.6514 4.6368 4.5477 4.5258 4.4869 4.4042 4.329 4.2833 4.2468 4.2063 4.1781 4.1409\n 4.1201 4.1053 4.0623 4.0225 3.9971 3.9583 3.9344 3.9041 3.8574 3.8499 3.7981 3.7584]\nEnergy fractions at ranks 16,20,22,24,26,28,32,64: [0.3034, 0.3517, 0.374, 0.3948, 0.4154, 0.435, 0.4715, 0.6879]\n\n W_fc.1 first 48 singular values:\n [20.7957 14.1429 13.8258 13.4276 13.1222 12.8832 12.7635 12.474 12.289 11.8837 11.7528 11.4702 11.3766 11.2499 10.9086 10.5664 10.4358 10.1111\n 9.9044 9.7303 9.4785 9.3223 9.3181 9.0878 8.9269 8.89 8.8311 8.5779 8.5148 8.451 8.3159 8.1094 7.9966 7.8812 7.7992 7.7515\n 7.6666 7.492 7.3091 7.2354 7.1804 7.0421 6.9016 6.6692 6.5361 6.4514 6.2559 6.2155]\nEnergy fractions at ranks 16,20,22,24,26,28,32,64: [0.4206, 0.4833, 0.5107, 0.537, 0.5616, 0.5852, 0.6285, 0.8294]\n\n W_proj.1 first 48 singular values:\n [7.6249 7.3593 6.9075 6.4887 6.3442 6.2228 6.1924 6.0202 5.9493 5.902 5.8199 5.7562 5.7277 5.6916 5.6739 5.6502 5.633 5.6084 5.5811 5.5672 5.5442\n 5.5233 5.479 5.4574 5.4498 5.4081 5.4044 5.3899 5.3745 5.3414 5.3376 5.3089 5.2848 5.2674 5.2573 5.2226 5.1969 5.1933 5.1791 5.1592 5.1544 5.127\n 5.1248 5.112 5.0938 5.0829 5.0821 5.0715]\nEnergy fractions at ranks 16,20,22,24,26,28,32,64: [0.1103, 0.1325, 0.1433, 0.1539, 0.1644, 0.1747, 0.1949, 0.3401]\n\n W_fc.2 first 48 singular values:\n [14.8285 8.0483 7.9211 7.76 7.6596 7.5865 7.5217 7.5088 7.367 7.3642 7.1778 7.1535 7.0887 7.0697 6.9674 6.9088 6.8907 6.8028\n 6.7443 6.739 6.6891 6.6337 6.6059 6.6018 6.5364 6.5158 6.4202 6.3948 6.3714 6.3508 6.3344 6.2827 6.2544 6.2211 6.163 6.1233\n 6.0878 6.0704 6.0281 5.9836 5.9701 5.9319 5.8981 5.8675 5.8219 5.8084 5.8002 5.7257]\nEnergy fractions at ranks 16,20,22,24,26,28,32,64: [0.1655, 0.1947, 0.2088, 0.2226, 0.2361, 0.2491, 0.2745, 0.4416]\n\n W_proj.2 first 48 singular values:\n [10.0946 8.9847 8.0747 7.9088 7.7913 7.6557 7.588 7.3746 7.2122 7.1441 7.0892 7.0344 7.003 6.9509 6.7302 6.7086 6.6299 6.5722\n 6.4523 6.3853 6.3412 6.3213 6.2519 6.2074 6.0874 6.0002 5.9051 5.8421 5.789 5.7646 5.7176 5.7159 5.6793 5.6364 5.5986 5.562\n 5.5049 5.4579 5.415 5.3935 5.3704 5.3605 5.3052 5.2815 5.2674 5.2524 5.238 5.213 ]\nEnergy fractions at ranks 16,20,22,24,26,28,32,64: [0.1553, 0.1835, 0.1969, 0.2098, 0.222, 0.2335, 0.2555, 0.4011]\n\n W_fc.3 first 48 singular values:\n [14.3731 8.1763 7.9733 7.8303 7.0359 6.7574 6.7114 6.5939 6.4676 6.3875 6.3466 6.2805 6.2285 6.1792 6.1624 6.0675 5.9953 5.9874\n 5.9467 5.895 5.8543 5.816 5.7909 5.7644 5.7533 5.6992 5.6811 5.6518 5.6413 5.6405 5.5883 5.5639 5.5377 5.5088 5.4604 5.4427\n 5.4106 5.3865 5.3686 5.3565 5.3281 5.3013 5.2929 5.2783 5.2729 5.2493 5.2308 5.2172]\nEnergy fractions at ranks 16,20,22,24,26,28,32,64: [0.1466, 0.1698, 0.1809, 0.1919, 0.2026, 0.2131, 0.2337, 0.3757]\n\n W_proj.3 first 48 singular values:\n [28.1752 20.2934 19.2921 19.1349 18.7081 18.5697 18.1393 17.7759 17.4845 16.5396 15.9131 15.6096 14.9752 14.8725 14.4167 14.3133 13.3269 12.9707\n 12.5934 12.3698 12.2251 12.0191 11.3225 11.115 10.7693 9.2267 8.5731 7.929 6.5001 5.2712 3.9657 3.1843 2.7917 2.6972 2.6532 2.5552\n 2.511 2.4709 2.3788 2.29 2.25 2.2147 2.1167 2.0811 2.0364 2.0107 1.9843 1.9692]\nEnergy fractions at ranks 16,20,22,24,26,28,32,64: [0.7207, 0.8115, 0.8521, 0.8869, 0.9147, 0.9335, 0.9468, 0.9659]\n\nMLP0 effective row norm quantiles: [0.2907 0.3287 0.3511 0.3939 1.7096 2.2242 4.3251]\nMLP0 largest off-diagonal row cosine quantiles: [0.2331 0.3941 0.4231 0.4629 0.5541 0.8192 0.9855]\n\nL1 QK coefficient cosines:\n [[1. 0.3023 0.3935 0.3265 0.3375 0.2656]\n [0.3023 1. 0.5243 0.5908 0.5677 0.5659]\n [0.3935 0.5243 1. 0.538 0.5781 0.4498]\n [0.3265 0.5908 0.538 1. 0.6072 0.5237]\n [0.3375 0.5677 0.5781 0.6072 1. 0.4791]\n [0.2656 0.5659 0.4498 0.5237 0.4791 1. ]]\nL1 key-linear coefficient cosines:\n [[1. 0.7977 0.8084 0.8013 0.8098 0.7573]\n [0.7977 1. 0.9231 0.9236 0.9207 0.8974]\n [0.8084 0.9231 1. 0.8985 0.9 0.8683]\n [0.8013 0.9236 0.8985 1. 0.9264 0.8987]\n [0.8098 0.9207 0.9 0.9264 1. 0.8831]\n [0.7573 0.8974 0.8683 0.8987 0.8831 1. ]]\nL1 query-linear coefficient cosines:\n [[1. 0.6085 0.7159 0.6376 0.7005 0.5869]\n [0.6085 1. 0.7067 0.872 0.8576 0.8649]\n [0.7159 0.7067 1. 0.784 0.8399 0.7013]\n [0.6376 0.872 0.784 1. 0.8987 0.864 ]\n [0.7005 0.8576 0.8399 0.8987 1. 0.8378]\n [0.5869 0.8649 0.7013 0.864 0.8378 1. ]]\nL1 key-linear versus query-linear cosines:\n [[-0.119 -0.2537 -0.1627 -0.1966 -0.1842 -0.2325]\n [-0.1606 -0.3366 -0.2368 -0.2599 -0.257 -0.3065]\n [-0.1822 -0.3153 -0.245 -0.2725 -0.2535 -0.3097]\n [-0.1535 -0.3143 -0.2374 -0.2855 -0.2597 -0.3162]\n [-0.1668 -0.3182 -0.2516 -0.2776 -0.2824 -0.3267]\n [-0.1599 -0.3014 -0.2403 -0.2662 -0.2612 -0.317 ]]\nLayer 0 key-only coefficient stack SV: [0.0796 0.0542 0.047 0.0193 0.0171 0.0009]\nLayer 1 key-only coefficient stack SV: [5.7026 0.9101 0.8376 0.7153 0.6953 0.4556]\nLayer 2 key-only coefficient stack SV: [2.1896 0.7759 0.6682 0.5513 0.5253 0.4585]\nLayer 3 key-only coefficient stack SV: [0.4312 0.3295 0.2685 0.2541 0.2394 0.2101]\n",
"seconds": 0.835,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "04fa1ff1719276a31f85f24ddf1d38b50a02ae7ea143a679213df5bb490c4363",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}W_fc.0 first 48 singular values: [18.1104 12.2798 8.4726 7.9016 7.7793 7.442 7.1458 7.1094 7.0083 6.8053 6.7211 6.5398 6.3779 6.3286 6.2698 6.2168 6.2037 6.0345 6.0235 5.9437 5.8446 5.7365 5.7016 5.6681 5.6093 5.5798 5.498 5.4472 5.3773 5.3341 5.2437 5.203 5.1406 5.1261 4.9896 4.9434 4.9202 4.8457 4.7874 4.7161 4.6694 4.6008 4.5188 4.4395 4.4082 4.3845 4.3074 4.2416] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.4395, 0.4945, 0.5196, 0.5438, 0.5673, 0.5898, 0.6317, 0.8472] W_proj.0 first 48 singular values: [10.9319 7.576 6.6285 6.2932 6.166 5.9239 5.886 5.7686 5.7059 5.6097 5.4949 5.4626 5.4044 5.3131 5.1933 5.1607 5.1191 5.0699 5.0033 4.9374 4.8767 4.7926 4.6927 4.6637 4.6514 4.6368 4.5477 4.5258 4.4869 4.4042 4.329 4.2833 4.2468 4.2063 4.1781 4.1409 4.1201 4.1053 4.0623 4.0225 3.9971 3.9583 3.9344 3.9041 3.8574 3.8499 3.7981 3.7584] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.3034, 0.3517, 0.374, 0.3948, 0.4154, 0.435, 0.4715, 0.6879] W_fc.1 first 48 singular values: [20.7957 14.1429 13.8258 13.4276 13.1222 12.8832 12.7635 12.474 12.289 11.8837 11.7528 11.4702 11.3766 11.2499 10.9086 10.5664 10.4358 10.1111 9.9044 9.7303 9.4785 9.3223 9.3181 9.0878 8.9269 8.89 8.8311 8.5779 8.5148 8.451 8.3159 8.1094 7.9966 7.8812 7.7992 7.7515 7.6666 7.492 7.3091 7.2354 7.1804 7.0421 6.9016 6.6692 6.5361 6.4514 6.2559 6.2155] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.4206, 0.4833, 0.5107, 0.537, 0.5616, 0.5852, 0.6285, 0.8294] W_proj.1 first 48 singular values: [7.6249 7.3593 6.9075 6.4887 6.3442 6.2228 6.1924 6.0202 5.9493 5.902 5.8199 5.7562 5.7277 5.6916 5.6739 5.6502 5.633 5.6084 5.5811 5.5672 5.5442 5.5233 5.479 5.4574 5.4498 5.4081 5.4044 5.3899 5.3745 5.3414 5.3376 5.3089 5.2848 5.2674 5.2573 5.2226 5.1969 5.1933 5.1791 5.1592 5.1544 5.127 5.1248 5.112 5.0938 5.0829 5.0821 5.0715] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.1103, 0.1325, 0.1433, 0.1539, 0.1644, 0.1747, 0.1949, 0.3401] W_fc.2 first 48 singular values: [14.8285 8.0483 7.9211 7.76 7.6596 7.5865 7.5217 7.5088 7.367 7.3642 7.1778 7.1535 7.0887 7.0697 6.9674 6.9088 6.8907 6.8028 6.7443 6.739 6.6891 6.6337 6.6059 6.6018 6.5364 6.5158 6.4202 6.3948 6.3714 6.3508 6.3344 6.2827 6.2544 6.2211 6.163 6.1233 6.0878 6.0704 6.0281 5.9836 5.9701 5.9319 5.8981 5.8675 5.8219 5.8084 5.8002 5.7257] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.1655, 0.1947, 0.2088, 0.2226, 0.2361, 0.2491, 0.2745, 0.4416] W_proj.2 first 48 singular values: [10.0946 8.9847 8.0747 7.9088 7.7913 7.6557 7.588 7.3746 7.2122 7.1441 7.0892 7.0344 7.003 6.9509 6.7302 6.7086 6.6299 6.5722 6.4523 6.3853 6.3412 6.3213 6.2519 6.2074 6.0874 6.0002 5.9051 5.8421 5.789 5.7646 5.7176 5.7159 5.6793 5.6364 5.5986 5.562 5.5049 5.4579 5.415 5.3935 5.3704 5.3605 5.3052 5.2815 5.2674 5.2524 5.238 5.213 ] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.1553, 0.1835, 0.1969, 0.2098, 0.222, 0.2335, 0.2555, 0.4011] W_fc.3 first 48 singular values: [14.3731 8.1763 7.9733 7.8303 7.0359 6.7574 6.7114 6.5939 6.4676 6.3875 6.3466 6.2805 6.2285 6.1792 6.1624 6.0675 5.9953 5.9874 5.9467 5.895 5.8543 5.816 5.7909 5.7644 5.7533 5.6992 5.6811 5.6518 5.6413 5.6405 5.5883 5.5639 5.5377 5.5088 5.4604 5.4427 5.4106 5.3865 5.3686 5.3565 5.3281 5.3013 5.2929 5.2783 5.2729 5.2493 5.2308 5.2172] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.1466, 0.1698, 0.1809, 0.1919, 0.2026, 0.2131, 0.2337, 0.3757] W_proj.3 first 48 singular values: [28.1752 20.2934 19.2921 19.1349 18.7081 18.5697 18.1393 17.7759 17.4845 16.5396 15.9131 15.6096 14.9752 14.8725 14.4167 14.3133 13.3269 12.9707 12.5934 12.3698 12.2251 12.0191 11.3225 11.115 10.7693 9.2267 8.5731 7.929 6.5001 5.2712 3.9657 3.1843 2.7917 2.6972 2.6532 2.5552 2.511 2.4709 2.3788 2.29 2.25 2.2147 2.1167 2.0811 2.0364 2.0107 1.9843 1.9692] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.7207, 0.8115, 0.8521, 0.8869, 0.9147, 0.9335, 0.9468, 0.9659] MLP0 effective row norm quantiles: [0.2907 0.3287 0.3511 0.3939 1.7096 2.2242 4.3251] MLP0 largest off-diagonal row cosine quantiles: [0.2331 0.3941 0.4231 0.4629 0.5541 0.8192 0.9855] L1 QK coefficient cosines: [[1. 0.3023 0.3935 0.3265 0.3375 0.2656] [0.3023 1. 0.5243 0.5908 0.5677 0.5659] [0.3935 0.5243 1. 0.538 0.5781 0.4498] [0.3265 0.5908 0.538 1. 0.6072 0.5237] [0.3375 0.5677 0.5781 0.6072 1. 0.4791] [0.2656 0.5659 0.4498 0.5237 0.4791 1. ]] L1 key-linear coefficient cosines: [[1. 0.7977 0.8084 0.8013 0.8098 0.7573] [0.7977 1. 0.9231 0.9236 0.9207 0.8974] [0.8084 0.9231 1. 0.8985 0.9 0.8683] [0.8013 0.9236 0.8985 1. 0.9264 0.8987] [0.8098 0.9207 0.9 0.9264 1. 0.8831] [0.7573 0.8974 0.8683 0.8987 0.8831 1. ]] L1 query-linear coefficient cosines: [[1. 0.6085 0.7159 0.6376 0.7005 0.5869] [0.6085 1. 0.7067 0.872 0.8576 0.8649] [0.7159 0.7067 1. 0.784 0.8399 0.7013] [0.6376 0.872 0.784 1. 0.8987 0.864 ] [0.7005 0.8576 0.8399 0.8987 1. 0.8378] [0.5869 0.8649 0.7013 0.864 0.8378 1. ]] L1 key-linear versus query-linear cosines: [[-0.119 -0.2537 -0.1627 -0.1966 -0.1842 -0.2325] [-0.1606 -0.3366 -0.2368 -0.2599 -0.257 -0.3065] [-0.1822 -0.3153 -0.245 -0.2725 -0.2535 -0.3097] [-0.1535 -0.3143 -0.2374 -0.2855 -0.2597 -0.3162] [-0.1668 -0.3182 -0.2516 -0.2776 -0.2824 -0.3267] [-0.1599 -0.3014 -0.2403 -0.2662 -0.2612 -0.317 ]] Layer 0 key-only coefficient stack SV: [0.0796 0.0542 0.047 0.0193 0.0171 0.0009] Layer 1 key-only coefficient stack SV: [5.7026 0.9101 0.8376 0.7153 0.6953 0.4556] Layer 2 key-only coefficient stack SV: [2.1896 0.7759 0.6682 0.5513 0.5253 0.4585] Layer 3 key-only coefficient stack SV: [0.4312 0.3295 0.2685 0.2541 0.2394 0.2101]
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 3 of at most 100: ``` W_fc.0 first 48 singular values: [18.1104 12.2798 8.4726 7.9016 7.7793 7.442 7.1458 7.1094 7.0083 6.8053 6.7211 6.5398 6.3779 6.3286 6.2698 6.2168 6.2037 6.0345 6.0235 5.9437 5.8446 5.7365 5.7016 5.6681 5.6093 5.5798 5.498 5.4472 5.3773 5.3341 5.2437 5.203 5.1406 5.1261 4.9896 4.9434 4.9202 4.8457 4.7874 4.7161 4.6694 4.6008 4.5188 4.4395 4.4082 4.3845 4.3074 4.2416] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.4395, 0.4945, 0.5196, 0.5438, 0.5673, 0.5898, 0.6317, 0.8472] W_proj.0 first 48 singular values: [10.9319 7.576 6.6285 6.2932 6.166 5.9239 5.886 5.7686 5.7059 5.6097 5.4949 5.4626 5.4044 5.3131 5.1933 5.1607 5.1191 5.0699 5.0033 4.9374 4.8767 4.7926 4.6927 4.6637 4.6514 4.6368 4.5477 4.5258 4.4869 4.4042 4.329 4.2833 4.2468 4.2063 4.1781 4.1409 4.1201 4.1053 4.0623 4.0225 3.9971 3.9583 3.9344 3.9041 3.8574 3.8499 3.7981 3.7584] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.3034, 0.3517, 0.374, 0.3948, 0.4154, 0.435, 0.4715, 0.6879] W_fc.1 first 48 singular values: [20.7957 14.1429 13.8258 13.4276 13.1222 12.8832 12.7635 12.474 12.289 11.8837 11.7528 11.4702 11.3766 11.2499 10.9086 10.5664 10.4358 10.1111 9.9044 9.7303 9.4785 9.3223 9.3181 9.0878 8.9269 8.89 8.8311 8.5779 8.5148 8.451 8.3159 8.1094 7.9966 7.8812 7.7992 7.7515 7.6666 7.492 7.3091 7.2354 7.1804 7.0421 6.9016 6.6692 6.5361 6.4514 6.2559 6.2155] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.4206, 0.4833, 0.5107, 0.537, 0.5616, 0.5852, 0.6285, 0.8294] W_proj.1 first 48 singular values: [7.6249 7.3593 6.9075 6.4887 6.3442 6.2228 6.1924 6.0202 5.9493 5.902 5.8199 5.7562 5.7277 5.6916 5.6739 5.6502 5.633 5.6084 5.5811 5.5672 5.5442 5.5233 5.479 5.4574 5.4498 5.4081 5.4044 5.3899 5.3745 5.3414 5.3376 5.3089 5.2848 5.2674 5.2573 5.2226 5.1969 5.1933 5.1791 5.1592 5.1544 5.127 5.1248 5.112 5.0938 5.0829 5.0821 5.0715] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.1103, 0.1325, 0.1433, 0.1539, 0.1644, 0.1747, 0.1949, 0.3401] W_fc.2 first 48 singular values: [14.8285 8.0483 7.9211 7.76 7.6596 7.5865 7.5217 7.5088 7.367 7.3642 7.1778 7.1535 7.0887 7.0697 6.9674 6.9088 6.8907 6.8028 6.7443 6.739 6.6891 6.6337 6.6059 6.6018 6.5364 6.5158 6.4202 6.3948 6.3714 6.3508 6.3344 6.2827 6.2544 6.2211 6.163 6.1233 6.0878 6.0704 6.0281 5.9836 5.9701 5.9319 5.8981 5.8675 5.8219 5.8084 5.8002 5.7257] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.1655, 0.1947, 0.2088, 0.2226, 0.2361, 0.2491, 0.2745, 0.4416] W_proj.2 first 48 singular values: [10.0946 8.9847 8.0747 7.9088 7.7913 7.6557 7.588 7.3746 7.2122 7.1441 7.0892 7.0344 7.003 6.9509 6.7302 6.7086 6.6299 6.5722 6.4523 6.3853 6.3412 6.3213 6.2519 6.2074 6.0874 6.0002 5.9051 5.8421 5.789 5.7646 5.7176 5.7159 5.6793 5.6364 5.5986 5.562 5.5049 5.4579 5.415 5.3935 5.3704 5.3605 5.3052 5.2815 5.2674 5.2524 5.238 5.213 ] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.1553, 0.1835, 0.1969, 0.2098, 0.222, 0.2335, 0.2555, 0.4011] W_fc.3 first 48 singular values: [14.3731 8.1763 7.9733 7.8303 7.0359 6.7574 6.7114 6.5939 6.4676 6.3875 6.3466 6.2805 6.2285 6.1792 6.1624 6.0675 5.9953 5.9874 5.9467 5.895 5.8543 5.816 5.7909 5.7644 5.7533 5.6992 5.6811 5.6518 5.6413 5.6405 5.5883 5.5639 5.5377 5.5088 5.4604 5.4427 5.4106 5.3865 5.3686 5.3565 5.3281 5.3013 5.2929 5.2783 5.2729 5.2493 5.2308 5.2172] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.1466, 0.1698, 0.1809, 0.1919, 0.2026, 0.2131, 0.2337, 0.3757] W_proj.3 first 48 singular values: [28.1752 20.2934 19.2921 19.1349 18.7081 18.5697 18.1393 17.7759 17.4845 16.5396 15.9131 15.6096 14.9752 14.8725 14.4167 14.3133 13.3269 12.9707 12.5934 12.3698 12.2251 12.0191 11.3225 11.115 10.7693 9.2267 8.5731 7.929 6.5001 5.2712 3.9657 3.1843 2.7917 2.6972 2.6532 2.5552 2.511 2.4709 2.3788 2.29 2.25 2.2147 2.1167 2.0811 2.0364 2.0107 1.9843 1.9692] Energy fractions at ranks 16,20,22,24,26,28,32,64: [0.7207, 0.8115, 0.8521, 0.8869, 0.9147, 0.9335, 0.9468, 0.9659] MLP0 effective row norm quantiles: [0.2907 0.3287 0.3511 0.3939 1.7096 2.2242 4.3251] MLP0 largest off-diagonal row cosine quantiles: [0.2331 0.3941 0.4231 0.4629 0.5541 0.8192 0.9855] L1 QK coefficient cosines: [[1. 0.3023 0.3935 0.3265 0.3375 0.2656] [0.3023 1. 0.5243 0.5908 0.5677 0.5659] [0.3935 0.5243 1. 0.538 0.5781 0.4498] [0.3265 0.5908 0.538 1. 0.6072 0.5237] [0.3375 0.5677 0.5781 0.6072 1. 0.4791] [0.2656 0.5659 0.4498 0.5237 0.4791 1. ]] L1 key-linear coefficient cosines: [[1. 0.7977 0.8084 0.8013 0.8098 0.7573] [0.7977 1. 0.9231 0.9236 0.9207 0.8974] [0.8084 0.9231 1. 0.8985 0.9 0.8683] [0.8013 0.9236 0.8985 1. 0.9264 0.8987] [0.8098 0.9207 0.9 0.9264 1. 0.8831] [0.7573 0.8974 0.8683 0.8987 0.8831 1. ]] L1 query-linear coefficient cosines: [[1. 0.6085 0.7159 0.6376 0.7005 0.5869] [0.6085 1. 0.7067 0.872 0.8576 0.8649] [0.7159 0.7067 1. 0.784 0.8399 0.7013] [0.6376 0.872 0.784 1. 0.8987 0.864 ] [0.7005 0.8576 0.8399 0.8987 1. 0.8378] [0.5869 0.8649 0.7013 0.864 0.8378 1. ]] L1 key-linear versus query-linear cosines: [[-0.119 -0.2537 -0.1627 -0.1966 -0.1842 -0.2325] [-0.1606 -0.3366 -0.2368 -0.2599 -0.257 -0.3065] [-0.1822 -0.3153 -0.245 -0.2725 -0.2535 -0.3097] [-0.1535 -0.3143 -0.2374 -0.2855 -0.2597 -0.3162] [-0.1668 -0.3182 -0.2516 -0.2776 -0.2824 -0.3267] [-0.1599 -0.3014 -0.2403 -0.2662 -0.2612 -0.317 ]] Layer 0 key-only coefficient stack SV: [0.0796 0.0542 0.047 0.0193 0.0171 0.0009] Layer 1 key-only coefficient stack SV: [5.7026 0.9101 0.8376 0.7153 0.6953 0.4556] Layer 2 key-only coefficient stack SV: [2.1896 0.7759 0.6682 0.5513 0.5253 0.4585] Layer 3 key-only coefficient stack SV: [0.4312 0.3295 0.2685 0.2541 0.2394 0.2101] ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svd, eigvals
from threadpoolctl import threadpool_limits
threadpool_limits(limits=2)
np.set_printoptions(precision=3, suppress=True, linewidth=160)
def scaled_read(name, l, sl=None):
a = W[f"{name}.{l}"].astype(float)
if sl is not None: a = a[sl]
a *= W[f"ln1_w.{l}"][None, :]
return a - a.mean(axis=1, keepdims=True)
A, B, Vr = [], [], []
for l in range(4):
for j in range(6):
sl = slice(64*j,64*(j+1))
q,k,v = [scaled_read(n,l,sl) for n in ["W_q","W_k","W_v"]]
o = W[f"W_o.{l}"][:,sl].astype(float)
o -= o.mean(axis=0,keepdims=True)
A.append(q.T@k/8); B.append(o@v); Vr.append(v)
_, s, vt = svd(Vr[5], full_matrices=False, check_finite=False)
print("L0H5 effective V singular values:", s[:40])
T = vt[:27].T
print("\nL0 QK energy in L0H5 V-read top27; query / key")
for j in range(6):
a = A[j]
print(j, np.linalg.norm(T.T@a)**2/np.linalg.norm(a)**2,
np.linalg.norm(a@T)**2/np.linalg.norm(a)**2)
raw = W["W_q.0"][64*j:64*(j+1)].astype(float).T @ W["W_k.0"][64*j:64*(j+1)].astype(float)/8
ev = eigvals(raw, check_finite=False)
print(" raw QK largest eigenvalues:", ev[np.argsort(-np.abs(ev))[:12]])
print("\nProjected composition trace / Frobenius norm on the L0H5 value-read subspace")
print("For each later head: best earlier OV head, trace/Frobenius, "
"positive-eigenvalue mass of symmetric part / total absolute mass")
for l in range(1,4):
for j in range(6):
a = A[6*l+j]
rows = []
for m in range(6*l):
c = T.T @ a @ B[m] @ T
ratio = np.trace(c)/(np.linalg.norm(c)+1e-30)
ev = np.linalg.eigvalsh((c+c.T)/2)
pos = ev[ev>0].sum()/(np.abs(ev).sum()+1e-30)
rows.append((ratio, pos, m, np.linalg.norm(c)))
rows.sort(reverse=True)
print("L%dH%d:"%(l,j),
["L%dH%d ratio %.3f pos %.3f norm %.3f" %
(m//6,m%6,r,p,n) for r,p,m,n in rows[:3]])
# Where each OV map reads and writes, measured as subspace energy.
Uout = svd(W["W_proj.3"].astype(float), full_matrices=False, check_finite=False)[0][:,:28]
print("\nOV energy fractions: read L0H5-value top27 / write final-MLP top28")
for i,b in enumerate(B):
print("L%dH%d %.3f %.3f"%(i//6,i%6,
np.linalg.norm(b@T)**2/np.linalg.norm(b)**2,
np.linalg.norm(Uout.T@b)**2/np.linalg.norm(b)**2))
# Learned MLP-row counts versus small, nearly initialization-scale rows.
for l in range(4):
f = W[f"W_fc.{l}"].astype(float)
rn = np.linalg.norm(f,axis=1)
print("MLP",l,"row norm quantiles",np.quantile(rn,[0,.1,.25,.5,.75,.9,1]),
"rows norm>1:",np.sum(rn>1),"norm>2:",np.sum(rn>2))
```
{
"run": 4,
"turn": "t004",
"code": "import numpy as np\nfrom scipy.linalg import svd, eigvals\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(limits=2)\nnp.set_printoptions(precision=3, suppress=True, linewidth=160)\n\ndef scaled_read(name, l, sl=None):\n a = W[f\"{name}.{l}\"].astype(float)\n if sl is not None: a = a[sl]\n a *= W[f\"ln1_w.{l}\"][None, :]\n return a - a.mean(axis=1, keepdims=True)\n\nA, B, Vr = [], [], []\nfor l in range(4):\n for j in range(6):\n sl = slice(64*j,64*(j+1))\n q,k,v = [scaled_read(n,l,sl) for n in [\"W_q\",\"W_k\",\"W_v\"]]\n o = W[f\"W_o.{l}\"][:,sl].astype(float)\n o -= o.mean(axis=0,keepdims=True)\n A.append(q.T@k/8); B.append(o@v); Vr.append(v)\n\n_, s, vt = svd(Vr[5], full_matrices=False, check_finite=False)\nprint(\"L0H5 effective V singular values:\", s[:40])\nT = vt[:27].T\nprint(\"\\nL0 QK energy in L0H5 V-read top27; query / key\")\nfor j in range(6):\n a = A[j]\n print(j, np.linalg.norm(T.T@a)**2/np.linalg.norm(a)**2,\n np.linalg.norm(a@T)**2/np.linalg.norm(a)**2)\n raw = W[\"W_q.0\"][64*j:64*(j+1)].astype(float).T @ W[\"W_k.0\"][64*j:64*(j+1)].astype(float)/8\n ev = eigvals(raw, check_finite=False)\n print(\" raw QK largest eigenvalues:\", ev[np.argsort(-np.abs(ev))[:12]])\n\nprint(\"\\nProjected composition trace / Frobenius norm on the L0H5 value-read subspace\")\nprint(\"For each later head: best earlier OV head, trace/Frobenius, \"\n \"positive-eigenvalue mass of symmetric part / total absolute mass\")\nfor l in range(1,4):\n for j in range(6):\n a = A[6*l+j]\n rows = []\n for m in range(6*l):\n c = T.T @ a @ B[m] @ T\n ratio = np.trace(c)/(np.linalg.norm(c)+1e-30)\n ev = np.linalg.eigvalsh((c+c.T)/2)\n pos = ev[ev>0].sum()/(np.abs(ev).sum()+1e-30)\n rows.append((ratio, pos, m, np.linalg.norm(c)))\n rows.sort(reverse=True)\n print(\"L%dH%d:\"%(l,j),\n [\"L%dH%d ratio %.3f pos %.3f norm %.3f\" %\n (m//6,m%6,r,p,n) for r,p,m,n in rows[:3]])\n\n# Where each OV map reads and writes, measured as subspace energy.\nUout = svd(W[\"W_proj.3\"].astype(float), full_matrices=False, check_finite=False)[0][:,:28]\nprint(\"\\nOV energy fractions: read L0H5-value top27 / write final-MLP top28\")\nfor i,b in enumerate(B):\n print(\"L%dH%d %.3f %.3f\"%(i//6,i%6,\n np.linalg.norm(b@T)**2/np.linalg.norm(b)**2,\n np.linalg.norm(Uout.T@b)**2/np.linalg.norm(b)**2))\n\n# Learned MLP-row counts versus small, nearly initialization-scale rows.\nfor l in range(4):\n f = W[f\"W_fc.{l}\"].astype(float)\n rn = np.linalg.norm(f,axis=1)\n print(\"MLP\",l,\"row norm quantiles\",np.quantile(rn,[0,.1,.25,.5,.75,.9,1]),\n \"rows norm>1:\",np.sum(rn>1),\"norm>2:\",np.sum(rn>2))",
"out": "L0H5 effective V singular values: [0.493 0.473 0.446 0.437 0.426 0.408 0.404 0.393 0.386 0.376 0.363 0.361 0.356 0.345 0.333 0.329 0.322 0.316 0.31 0.304 0.299 0.289 0.285 0.277 0.274 0.267\n 0.252 0.245 0.241 0.235 0.233 0.221 0.216 0.212 0.208 0.203 0.199 0.194 0.188 0.179]\n\nL0 QK energy in L0H5 V-read top27; query / key\n0 0.06915506568490729 0.4005828987732276\n raw QK largest eigenvalues: [-0.307+0.j -0.065+0.j -0.031+0.j -0.018+0.005j -0.018-0.005j -0.01 +0.007j -0.01 -0.007j -0.012+0.j -0.011+0.003j -0.011-0.003j -0.007-0.007j\n -0.007+0.007j]\n1 0.14100247076637126 0.17875867300936604\n raw QK largest eigenvalues: [-2.036+0.j -0.868+1.059j -0.868-1.059j -0.64 +0.764j -0.64 -0.764j 0.382+0.574j 0.382-0.574j -0.221+0.286j -0.221-0.286j 0.178-0.217j 0.178+0.217j\n 0.002+0.28j ]\n2 0.1380088946278737 0.3036819626565164\n raw QK largest eigenvalues: [-0.267+0.j -0.188+0.035j -0.188-0.035j -0.173+0.j 0.118+0.066j 0.118-0.066j -0.097+0.053j -0.097-0.053j 0.103+0.j -0.093+0.j 0.031+0.045j\n 0.031-0.045j]\n3 0.12867697668337996 0.26320291058948364\n raw QK largest eigenvalues: [-0.32 +0.j -0.184+0.047j -0.184-0.047j -0.058+0.104j -0.058-0.104j -0.109+0.j 0.084+0.j -0.051+0.044j -0.051-0.044j -0.062+0.j -0.052+0.j\n 0.004+0.045j]\n4 0.09665577096102634 0.11399456953990159\n raw QK largest eigenvalues: [-0.012+0.007j -0.012-0.007j -0.011+0.j -0.004+0.j -0.004-0.j -0.003+0.003j -0.003-0.003j -0.003+0.002j -0.003-0.002j -0.004+0.j 0.003-0.002j\n 0.003+0.002j]\n5 0.16163991771158023 0.18783951477136343\n raw QK largest eigenvalues: [ 0.962+1.178j 0.962-1.178j 1.104+0.737j 1.104-0.737j 0.238+1.225j 0.238-1.225j 1.246+0.j 0.708+0.985j 0.708-0.985j -0.081-1.088j -0.081+1.088j\n -0.378-0.994j]\n\nProjected composition trace / Frobenius norm on the L0H5 value-read subspace\nFor each later head: best earlier OV head, trace/Frobenius, positive-eigenvalue mass of symmetric part / total absolute mass\nL1H0: ['L0H1 ratio 0.185 pos 0.532 norm 0.032', 'L0H3 ratio -0.012 pos 0.497 norm 0.013', 'L0H5 ratio -0.173 pos 0.470 norm 0.045']\nL1H1: ['L0H5 ratio 0.783 pos 0.640 norm 0.115', 'L0H1 ratio 0.512 pos 0.586 norm 0.053', 'L0H0 ratio 0.312 pos 0.623 norm 0.302']\nL1H2: ['L0H3 ratio 0.069 pos 0.519 norm 0.025', 'L0H1 ratio 0.044 pos 0.508 norm 0.041', 'L0H5 ratio -0.162 pos 0.473 norm 0.072']\nL1H3: ['L0H5 ratio 0.706 pos 0.622 norm 0.085', 'L0H3 ratio 0.214 pos 0.564 norm 0.031', 'L0H2 ratio 0.122 pos 0.532 norm 0.042']\nL1H4: ['L0H5 ratio 0.488 pos 0.588 norm 0.084', 'L0H1 ratio 0.414 pos 0.571 norm 0.039', 'L0H4 ratio 0.104 pos 0.541 norm 0.686']\nL1H5: ['L0H5 ratio 0.549 pos 0.604 norm 0.115', 'L0H4 ratio 0.299 pos 0.624 norm 1.103', 'L0H0 ratio 0.286 pos 0.616 norm 0.248']\nL2H0: ['L0H3 ratio 0.185 pos 0.535 norm 0.027', 'L1H2 ratio 0.144 pos 0.526 norm 0.162', 'L1H5 ratio 0.055 pos 0.510 norm 0.128']\nL2H1: ['L0H5 ratio 2.747 pos 0.922 norm 0.199', 'L1H4 ratio 1.740 pos 0.795 norm 0.169', 'L1H2 ratio 1.725 pos 0.789 norm 0.209']\nL2H2: ['L0H4 ratio 0.071 pos 0.524 norm 0.642', 'L0H3 ratio 0.017 pos 0.503 norm 0.038', 'L1H5 ratio -0.032 pos 0.495 norm 0.192']\nL2H3: ['L1H1 ratio 0.270 pos 0.548 norm 0.111', 'L0H1 ratio 0.215 pos 0.534 norm 0.073', 'L1H5 ratio 0.124 pos 0.520 norm 0.116']\nL2H4: ['L0H5 ratio 0.125 pos 0.521 norm 0.135', 'L0H3 ratio 0.121 pos 0.525 norm 0.034', 'L0H2 ratio 0.101 pos 0.524 norm 0.074']\nL2H5: ['L0H5 ratio 0.479 pos 0.578 norm 0.086', 'L1H2 ratio 0.468 pos 0.584 norm 0.119', 'L1H4 ratio 0.347 pos 0.562 norm 0.089']\nL3H0: ['L0H5 ratio 1.201 pos 0.700 norm 0.124', 'L2H2 ratio 0.883 pos 0.645 norm 0.160', 'L1H2 ratio 0.803 pos 0.640 norm 0.176']\nL3H1: ['L0H5 ratio 2.509 pos 0.886 norm 0.149', 'L2H2 ratio 2.169 pos 0.839 norm 0.181', 'L1H2 ratio 1.736 pos 0.788 norm 0.169']\nL3H2: ['L2H0 ratio 0.270 pos 0.546 norm 0.142', 'L0H2 ratio 0.229 pos 0.546 norm 0.046', 'L1H3 ratio 0.224 pos 0.539 norm 0.121']\nL3H3: ['L2H4 ratio 0.794 pos 0.639 norm 0.092', 'L0H2 ratio 0.591 pos 0.636 norm 0.025', 'L1H2 ratio 0.528 pos 0.589 norm 0.068']\nL3H4: ['L0H5 ratio 0.827 pos 0.637 norm 0.085', 'L2H2 ratio 0.513 pos 0.588 norm 0.123', 'L2H4 ratio 0.497 pos 0.584 norm 0.161']\nL3H5: ['L2H2 ratio 2.756 pos 0.919 norm 0.268', 'L0H5 ratio 2.637 pos 0.900 norm 0.209', 'L1H4 ratio 1.821 pos 0.806 norm 0.169']\n\nOV energy fractions: read L0H5-value top27 / write final-MLP top28\nL0H0 0.090 0.189\nL0H1 0.694 0.065\nL0H2 0.543 0.098\nL0H3 0.252 0.164\nL0H4 0.085 0.221\nL0H5 0.948 0.059\nL1H0 0.169 0.080\nL1H1 0.179 0.054\nL1H2 0.209 0.066\nL1H3 0.168 0.059\nL1H4 0.188 0.058\nL1H5 0.198 0.058\nL2H0 0.088 0.060\nL2H1 0.088 0.093\nL2H2 0.094 0.051\nL2H3 0.081 0.058\nL2H4 0.136 0.058\nL2H5 0.083 0.081\nL3H0 0.079 0.133\nL3H1 0.115 0.333\nL3H2 0.078 0.126\nL3H3 0.072 0.132\nL3H4 0.077 0.153\nL3H5 0.131 0.330\nMLP 0 row norm quantiles [0.33 0.371 0.394 0.437 1.766 2.393 4.287] rows norm>1: 518 norm>2: 286\nMLP 1 row norm quantiles [0.441 1.869 1.942 2.023 2.11 2.206 8.68 ] rows norm>1: 1532 norm>2: 887\nMLP 2 row norm quantiles [1.678 1.933 1.979 2.024 2.073 2.121 2.731] rows norm>1: 1536 norm>2: 963\nMLP 3 row norm quantiles [1.746 1.902 1.954 1.995 2.036 2.072 2.947] rows norm>1: 1536 norm>2: 715\n",
"seconds": 0.964,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "ea01d1a685a726022ead934e9f0e5176ec29a266e3df8df06e43a7532e79978c",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}L0H5 effective V singular values: [0.493 0.473 0.446 0.437 0.426 0.408 0.404 0.393 0.386 0.376 0.363 0.361 0.356 0.345 0.333 0.329 0.322 0.316 0.31 0.304 0.299 0.289 0.285 0.277 0.274 0.267 0.252 0.245 0.241 0.235 0.233 0.221 0.216 0.212 0.208 0.203 0.199 0.194 0.188 0.179] L0 QK energy in L0H5 V-read top27; query / key 0 0.06915506568490729 0.4005828987732276 raw QK largest eigenvalues: [-0.307+0.j -0.065+0.j -0.031+0.j -0.018+0.005j -0.018-0.005j -0.01 +0.007j -0.01 -0.007j -0.012+0.j -0.011+0.003j -0.011-0.003j -0.007-0.007j -0.007+0.007j] 1 0.14100247076637126 0.17875867300936604 raw QK largest eigenvalues: [-2.036+0.j -0.868+1.059j -0.868-1.059j -0.64 +0.764j -0.64 -0.764j 0.382+0.574j 0.382-0.574j -0.221+0.286j -0.221-0.286j 0.178-0.217j 0.178+0.217j 0.002+0.28j ] 2 0.1380088946278737 0.3036819626565164 raw QK largest eigenvalues: [-0.267+0.j -0.188+0.035j -0.188-0.035j -0.173+0.j 0.118+0.066j 0.118-0.066j -0.097+0.053j -0.097-0.053j 0.103+0.j -0.093+0.j 0.031+0.045j 0.031-0.045j] 3 0.12867697668337996 0.26320291058948364 raw QK largest eigenvalues: [-0.32 +0.j -0.184+0.047j -0.184-0.047j -0.058+0.104j -0.058-0.104j -0.109+0.j 0.084+0.j -0.051+0.044j -0.051-0.044j -0.062+0.j -0.052+0.j 0.004+0.045j] 4 0.09665577096102634 0.11399456953990159 raw QK largest eigenvalues: [-0.012+0.007j -0.012-0.007j -0.011+0.j -0.004+0.j -0.004-0.j -0.003+0.003j -0.003-0.003j -0.003+0.002j -0.003-0.002j -0.004+0.j 0.003-0.002j 0.003+0.002j] 5 0.16163991771158023 0.18783951477136343 raw QK largest eigenvalues: [ 0.962+1.178j 0.962-1.178j 1.104+0.737j 1.104-0.737j 0.238+1.225j 0.238-1.225j 1.246+0.j 0.708+0.985j 0.708-0.985j -0.081-1.088j -0.081+1.088j -0.378-0.994j] Projected composition trace / Frobenius norm on the L0H5 value-read subspace For each later head: best earlier OV head, trace/Frobenius, positive-eigenvalue mass of symmetric part / total absolute mass L1H0: ['L0H1 ratio 0.185 pos 0.532 norm 0.032', 'L0H3 ratio -0.012 pos 0.497 norm 0.013', 'L0H5 ratio -0.173 pos 0.470 norm 0.045'] L1H1: ['L0H5 ratio 0.783 pos 0.640 norm 0.115', 'L0H1 ratio 0.512 pos 0.586 norm 0.053', 'L0H0 ratio 0.312 pos 0.623 norm 0.302'] L1H2: ['L0H3 ratio 0.069 pos 0.519 norm 0.025', 'L0H1 ratio 0.044 pos 0.508 norm 0.041', 'L0H5 ratio -0.162 pos 0.473 norm 0.072'] L1H3: ['L0H5 ratio 0.706 pos 0.622 norm 0.085', 'L0H3 ratio 0.214 pos 0.564 norm 0.031', 'L0H2 ratio 0.122 pos 0.532 norm 0.042'] L1H4: ['L0H5 ratio 0.488 pos 0.588 norm 0.084', 'L0H1 ratio 0.414 pos 0.571 norm 0.039', 'L0H4 ratio 0.104 pos 0.541 norm 0.686'] L1H5: ['L0H5 ratio 0.549 pos 0.604 norm 0.115', 'L0H4 ratio 0.299 pos 0.624 norm 1.103', 'L0H0 ratio 0.286 pos 0.616 norm 0.248'] L2H0: ['L0H3 ratio 0.185 pos 0.535 norm 0.027', 'L1H2 ratio 0.144 pos 0.526 norm 0.162', 'L1H5 ratio 0.055 pos 0.510 norm 0.128'] L2H1: ['L0H5 ratio 2.747 pos 0.922 norm 0.199', 'L1H4 ratio 1.740 pos 0.795 norm 0.169', 'L1H2 ratio 1.725 pos 0.789 norm 0.209'] L2H2: ['L0H4 ratio 0.071 pos 0.524 norm 0.642', 'L0H3 ratio 0.017 pos 0.503 norm 0.038', 'L1H5 ratio -0.032 pos 0.495 norm 0.192'] L2H3: ['L1H1 ratio 0.270 pos 0.548 norm 0.111', 'L0H1 ratio 0.215 pos 0.534 norm 0.073', 'L1H5 ratio 0.124 pos 0.520 norm 0.116'] L2H4: ['L0H5 ratio 0.125 pos 0.521 norm 0.135', 'L0H3 ratio 0.121 pos 0.525 norm 0.034', 'L0H2 ratio 0.101 pos 0.524 norm 0.074'] L2H5: ['L0H5 ratio 0.479 pos 0.578 norm 0.086', 'L1H2 ratio 0.468 pos 0.584 norm 0.119', 'L1H4 ratio 0.347 pos 0.562 norm 0.089'] L3H0: ['L0H5 ratio 1.201 pos 0.700 norm 0.124', 'L2H2 ratio 0.883 pos 0.645 norm 0.160', 'L1H2 ratio 0.803 pos 0.640 norm 0.176'] L3H1: ['L0H5 ratio 2.509 pos 0.886 norm 0.149', 'L2H2 ratio 2.169 pos 0.839 norm 0.181', 'L1H2 ratio 1.736 pos 0.788 norm 0.169'] L3H2: ['L2H0 ratio 0.270 pos 0.546 norm 0.142', 'L0H2 ratio 0.229 pos 0.546 norm 0.046', 'L1H3 ratio 0.224 pos 0.539 norm 0.121'] L3H3: ['L2H4 ratio 0.794 pos 0.639 norm 0.092', 'L0H2 ratio 0.591 pos 0.636 norm 0.025', 'L1H2 ratio 0.528 pos 0.589 norm 0.068'] L3H4: ['L0H5 ratio 0.827 pos 0.637 norm 0.085', 'L2H2 ratio 0.513 pos 0.588 norm 0.123', 'L2H4 ratio 0.497 pos 0.584 norm 0.161'] L3H5: ['L2H2 ratio 2.756 pos 0.919 norm 0.268', 'L0H5 ratio 2.637 pos 0.900 norm 0.209', 'L1H4 ratio 1.821 pos 0.806 norm 0.169'] OV energy fractions: read L0H5-value top27 / write final-MLP top28 L0H0 0.090 0.189 L0H1 0.694 0.065 L0H2 0.543 0.098 L0H3 0.252 0.164 L0H4 0.085 0.221 L0H5 0.948 0.059 L1H0 0.169 0.080 L1H1 0.179 0.054 L1H2 0.209 0.066 L1H3 0.168 0.059 L1H4 0.188 0.058 L1H5 0.198 0.058 L2H0 0.088 0.060 L2H1 0.088 0.093 L2H2 0.094 0.051 L2H3 0.081 0.058 L2H4 0.136 0.058 L2H5 0.083 0.081 L3H0 0.079 0.133 L3H1 0.115 0.333 L3H2 0.078 0.126 L3H3 0.072 0.132 L3H4 0.077 0.153 L3H5 0.131 0.330 MLP 0 row norm quantiles [0.33 0.371 0.394 0.437 1.766 2.393 4.287] rows norm>1: 518 norm>2: 286 MLP 1 row norm quantiles [0.441 1.869 1.942 2.023 2.11 2.206 8.68 ] rows norm>1: 1532 norm>2: 887 MLP 2 row norm quantiles [1.678 1.933 1.979 2.024 2.073 2.121 2.731] rows norm>1: 1536 norm>2: 963 MLP 3 row norm quantiles [1.746 1.902 1.954 1.995 2.036 2.072 2.947] rows norm>1: 1536 norm>2: 715
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 4 of at most 100: ``` L0H5 effective V singular values: [0.493 0.473 0.446 0.437 0.426 0.408 0.404 0.393 0.386 0.376 0.363 0.361 0.356 0.345 0.333 0.329 0.322 0.316 0.31 0.304 0.299 0.289 0.285 0.277 0.274 0.267 0.252 0.245 0.241 0.235 0.233 0.221 0.216 0.212 0.208 0.203 0.199 0.194 0.188 0.179] L0 QK energy in L0H5 V-read top27; query / key 0 0.06915506568490729 0.4005828987732276 raw QK largest eigenvalues: [-0.307+0.j -0.065+0.j -0.031+0.j -0.018+0.005j -0.018-0.005j -0.01 +0.007j -0.01 -0.007j -0.012+0.j -0.011+0.003j -0.011-0.003j -0.007-0.007j -0.007+0.007j] 1 0.14100247076637126 0.17875867300936604 raw QK largest eigenvalues: [-2.036+0.j -0.868+1.059j -0.868-1.059j -0.64 +0.764j -0.64 -0.764j 0.382+0.574j 0.382-0.574j -0.221+0.286j -0.221-0.286j 0.178-0.217j 0.178+0.217j 0.002+0.28j ] 2 0.1380088946278737 0.3036819626565164 raw QK largest eigenvalues: [-0.267+0.j -0.188+0.035j -0.188-0.035j -0.173+0.j 0.118+0.066j 0.118-0.066j -0.097+0.053j -0.097-0.053j 0.103+0.j -0.093+0.j 0.031+0.045j 0.031-0.045j] 3 0.12867697668337996 0.26320291058948364 raw QK largest eigenvalues: [-0.32 +0.j -0.184+0.047j -0.184-0.047j -0.058+0.104j -0.058-0.104j -0.109+0.j 0.084+0.j -0.051+0.044j -0.051-0.044j -0.062+0.j -0.052+0.j 0.004+0.045j] 4 0.09665577096102634 0.11399456953990159 raw QK largest eigenvalues: [-0.012+0.007j -0.012-0.007j -0.011+0.j -0.004+0.j -0.004-0.j -0.003+0.003j -0.003-0.003j -0.003+0.002j -0.003-0.002j -0.004+0.j 0.003-0.002j 0.003+0.002j] 5 0.16163991771158023 0.18783951477136343 raw QK largest eigenvalues: [ 0.962+1.178j 0.962-1.178j 1.104+0.737j 1.104-0.737j 0.238+1.225j 0.238-1.225j 1.246+0.j 0.708+0.985j 0.708-0.985j -0.081-1.088j -0.081+1.088j -0.378-0.994j] Projected composition trace / Frobenius norm on the L0H5 value-read subspace For each later head: best earlier OV head, trace/Frobenius, positive-eigenvalue mass of symmetric part / total absolute mass L1H0: ['L0H1 ratio 0.185 pos 0.532 norm 0.032', 'L0H3 ratio -0.012 pos 0.497 norm 0.013', 'L0H5 ratio -0.173 pos 0.470 norm 0.045'] L1H1: ['L0H5 ratio 0.783 pos 0.640 norm 0.115', 'L0H1 ratio 0.512 pos 0.586 norm 0.053', 'L0H0 ratio 0.312 pos 0.623 norm 0.302'] L1H2: ['L0H3 ratio 0.069 pos 0.519 norm 0.025', 'L0H1 ratio 0.044 pos 0.508 norm 0.041', 'L0H5 ratio -0.162 pos 0.473 norm 0.072'] L1H3: ['L0H5 ratio 0.706 pos 0.622 norm 0.085', 'L0H3 ratio 0.214 pos 0.564 norm 0.031', 'L0H2 ratio 0.122 pos 0.532 norm 0.042'] L1H4: ['L0H5 ratio 0.488 pos 0.588 norm 0.084', 'L0H1 ratio 0.414 pos 0.571 norm 0.039', 'L0H4 ratio 0.104 pos 0.541 norm 0.686'] L1H5: ['L0H5 ratio 0.549 pos 0.604 norm 0.115', 'L0H4 ratio 0.299 pos 0.624 norm 1.103', 'L0H0 ratio 0.286 pos 0.616 norm 0.248'] L2H0: ['L0H3 ratio 0.185 pos 0.535 norm 0.027', 'L1H2 ratio 0.144 pos 0.526 norm 0.162', 'L1H5 ratio 0.055 pos 0.510 norm 0.128'] L2H1: ['L0H5 ratio 2.747 pos 0.922 norm 0.199', 'L1H4 ratio 1.740 pos 0.795 norm 0.169', 'L1H2 ratio 1.725 pos 0.789 norm 0.209'] L2H2: ['L0H4 ratio 0.071 pos 0.524 norm 0.642', 'L0H3 ratio 0.017 pos 0.503 norm 0.038', 'L1H5 ratio -0.032 pos 0.495 norm 0.192'] L2H3: ['L1H1 ratio 0.270 pos 0.548 norm 0.111', 'L0H1 ratio 0.215 pos 0.534 norm 0.073', 'L1H5 ratio 0.124 pos 0.520 norm 0.116'] L2H4: ['L0H5 ratio 0.125 pos 0.521 norm 0.135', 'L0H3 ratio 0.121 pos 0.525 norm 0.034', 'L0H2 ratio 0.101 pos 0.524 norm 0.074'] L2H5: ['L0H5 ratio 0.479 pos 0.578 norm 0.086', 'L1H2 ratio 0.468 pos 0.584 norm 0.119', 'L1H4 ratio 0.347 pos 0.562 norm 0.089'] L3H0: ['L0H5 ratio 1.201 pos 0.700 norm 0.124', 'L2H2 ratio 0.883 pos 0.645 norm 0.160', 'L1H2 ratio 0.803 pos 0.640 norm 0.176'] L3H1: ['L0H5 ratio 2.509 pos 0.886 norm 0.149', 'L2H2 ratio 2.169 pos 0.839 norm 0.181', 'L1H2 ratio 1.736 pos 0.788 norm 0.169'] L3H2: ['L2H0 ratio 0.270 pos 0.546 norm 0.142', 'L0H2 ratio 0.229 pos 0.546 norm 0.046', 'L1H3 ratio 0.224 pos 0.539 norm 0.121'] L3H3: ['L2H4 ratio 0.794 pos 0.639 norm 0.092', 'L0H2 ratio 0.591 pos 0.636 norm 0.025', 'L1H2 ratio 0.528 pos 0.589 norm 0.068'] L3H4: ['L0H5 ratio 0.827 pos 0.637 norm 0.085', 'L2H2 ratio 0.513 pos 0.588 norm 0.123', 'L2H4 ratio 0.497 pos 0.584 norm 0.161'] L3H5: ['L2H2 ratio 2.756 pos 0.919 norm 0.268', 'L0H5 ratio 2.637 pos 0.900 norm 0.209', 'L1H4 ratio 1.821 pos 0.806 norm 0.169'] OV energy fractions: read L0H5-value top27 / write final-MLP top28 L0H0 0.090 0.189 L0H1 0.694 0.065 L0H2 0.543 0.098 L0H3 0.252 0.164 L0H4 0.085 0.221 L0H5 0.948 0.059 L1H0 0.169 0.080 L1H1 0.179 0.054 L1H2 0.209 0.066 L1H3 0.168 0.059 L1H4 0.188 0.058 L1H5 0.198 0.058 L2H0 0.088 0.060 L2H1 0.088 0.093 L2H2 0.094 0.051 L2H3 0.081 0.058 L2H4 0.136 0.058 L2H5 0.083 0.081 L3H0 0.079 0.133 L3H1 0.115 0.333 L3H2 0.078 0.126 L3H3 0.072 0.132 L3H4 0.077 0.153 L3H5 0.131 0.330 MLP 0 row norm quantiles [0.33 0.371 0.394 0.437 1.766 2.393 4.287] rows norm>1: 518 norm>2: 286 MLP 1 row norm quantiles [0.441 1.869 1.942 2.023 2.11 2.206 8.68 ] rows norm>1: 1532 norm>2: 887 MLP 2 row norm quantiles [1.678 1.933 1.979 2.024 2.073 2.121 2.731] rows norm>1: 1536 norm>2: 963 MLP 3 row norm quantiles [1.746 1.902 1.954 1.995 2.036 2.072 2.947] rows norm>1: 1536 norm>2: 715 ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svd
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from threadpoolctl import threadpool_limits
threadpool_limits(limits=2)
np.set_printoptions(precision=3, suppress=True, linewidth=160)
g = W["ln1_w.0"].astype(float)
sl = slice(320,384)
v = W["W_v.0"][sl].astype(float)*g[None,:]
v -= v.mean(axis=1,keepdims=True)
o = W["W_o.0"][:,sl].astype(float)
o -= o.mean(axis=0,keepdims=True)
u,s,vt = svd(o@v,full_matrices=False,check_finite=False)
T = vt[:27].T
print("L0H5 effective OV first 40 singular values:",s[:40])
f = W["W_fc.0"].astype(float)*W["ln2_w.0"][None,:]
f -= f.mean(axis=1,keepdims=True)
ft = f@T
active = np.linalg.norm(W["W_fc.0"],axis=1)>1
print("MLP0 active row count:",active.sum())
print("MLP0 active rows restricted to value-read subspace: singular values:",
svd(ft[active],compute_uv=False,check_finite=False))
print("Fraction of each active MLP0 row's energy in that subspace, quantiles:",
np.quantile(np.sum(ft[active]**2,axis=1)/np.sum(f[active]**2,axis=1),[0,.1,.25,.5,.75,.9,1]))
# Cluster learned weight directions only.
datasets = {
"MLP0 active token-read row directions": ft[active],
"MLP3 write-column directions": W["W_proj.3"].astype(float).T,
}
for name,z in datasets.items():
z=z/np.linalg.norm(z,axis=1,keepdims=True)
print("\n",name)
for k in [8,16,20,24,28,32]:
km=KMeans(n_clusters=k,n_init=5,max_iter=150,random_state=0).fit(z)
labels=km.labels_
counts=np.bincount(labels,minlength=k)
centers=km.cluster_centers_
centers/=np.linalg.norm(centers,axis=1,keepdims=True)
cohesion=np.array([np.mean(z[labels==j]@centers[j]) for j in range(k)])
sil=silhouette_score(z,labels,metric="cosine",sample_size=min(700,len(z)),random_state=0)
print("k",k,"silhouette",round(sil,3),"mean cohesion",round(float(cohesion.mean()),3),
"sizes",np.sort(counts)[::-1])
if k==28:
idx=np.argsort(-counts)
print("cohesion by descending cluster size:",np.round(cohesion[idx],3))
cc=centers@centers.T
np.fill_diagonal(cc,-1)
print("largest inter-center cosines:",np.sort(cc.max(axis=1))[::-1])
```
{
"run": 5,
"turn": "t005",
"code": "import numpy as np\nfrom scipy.linalg import svd\nfrom sklearn.cluster import KMeans\nfrom sklearn.metrics import silhouette_score\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(limits=2)\nnp.set_printoptions(precision=3, suppress=True, linewidth=160)\n\ng = W[\"ln1_w.0\"].astype(float)\nsl = slice(320,384)\nv = W[\"W_v.0\"][sl].astype(float)*g[None,:]\nv -= v.mean(axis=1,keepdims=True)\no = W[\"W_o.0\"][:,sl].astype(float)\no -= o.mean(axis=0,keepdims=True)\nu,s,vt = svd(o@v,full_matrices=False,check_finite=False)\nT = vt[:27].T\nprint(\"L0H5 effective OV first 40 singular values:\",s[:40])\n\nf = W[\"W_fc.0\"].astype(float)*W[\"ln2_w.0\"][None,:]\nf -= f.mean(axis=1,keepdims=True)\nft = f@T\nactive = np.linalg.norm(W[\"W_fc.0\"],axis=1)>1\nprint(\"MLP0 active row count:\",active.sum())\nprint(\"MLP0 active rows restricted to value-read subspace: singular values:\",\n svd(ft[active],compute_uv=False,check_finite=False))\nprint(\"Fraction of each active MLP0 row's energy in that subspace, quantiles:\",\n np.quantile(np.sum(ft[active]**2,axis=1)/np.sum(f[active]**2,axis=1),[0,.1,.25,.5,.75,.9,1]))\n\n# Cluster learned weight directions only.\ndatasets = {\n \"MLP0 active token-read row directions\": ft[active],\n \"MLP3 write-column directions\": W[\"W_proj.3\"].astype(float).T,\n}\nfor name,z in datasets.items():\n z=z/np.linalg.norm(z,axis=1,keepdims=True)\n print(\"\\n\",name)\n for k in [8,16,20,24,28,32]:\n km=KMeans(n_clusters=k,n_init=5,max_iter=150,random_state=0).fit(z)\n labels=km.labels_\n counts=np.bincount(labels,minlength=k)\n centers=km.cluster_centers_\n centers/=np.linalg.norm(centers,axis=1,keepdims=True)\n cohesion=np.array([np.mean(z[labels==j]@centers[j]) for j in range(k)])\n sil=silhouette_score(z,labels,metric=\"cosine\",sample_size=min(700,len(z)),random_state=0)\n print(\"k\",k,\"silhouette\",round(sil,3),\"mean cohesion\",round(float(cohesion.mean()),3),\n \"sizes\",np.sort(counts)[::-1])\n if k==28:\n idx=np.argsort(-counts)\n print(\"cohesion by descending cluster size:\",np.round(cohesion[idx],3))\n cc=centers@centers.T\n np.fill_diagonal(cc,-1)\n print(\"largest inter-center cosines:\",np.sort(cc.max(axis=1))[::-1])",
"out": "L0H5 effective OV first 40 singular values: [1.059 1.044 0.96 0.948 0.909 0.899 0.855 0.836 0.817 0.77 0.753 0.731 0.709 0.676 0.656 0.646 0.604 0.552 0.533 0.454 0.42 0.346 0.332 0.307 0.299 0.292\n 0.274 0.259 0.237 0.227 0.223 0.216 0.211 0.2 0.189 0.184 0.167 0.153 0.135 0.125]\nMLP0 active row count: 518\nMLP0 active rows restricted to value-read subspace: singular values: [7.422 6.091 5.646 5.175 4.919 4.62 4.555 4.487 4.348 4.284 4.107 3.976 3.947 3.822 3.782 3.724 3.596 3.441 3.248 3.185 2.986 2.693 2.22 1.809 1.556 1.257\n 1.095]\nFraction of each active MLP0 row's energy in that subspace, quantiles: [0.02 0.055 0.096 0.191 0.289 0.433 0.66 ]\n\n MLP0 active token-read row directions\nk 8 silhouette 0.157 mean cohesion 0.571 sizes [119 88 87 84 44 36 32 28]\nk 16 silhouette 0.236 mean cohesion 0.661 sizes [80 58 51 39 33 29 28 27 27 26 25 22 21 19 19 14]\nk 20 silhouette 0.269 mean cohesion 0.688 sizes [73 37 33 31 30 27 27 25 25 24 22 21 20 20 19 19 18 18 16 13]\nk 24 silhouette 0.273 mean cohesion 0.716 sizes [65 33 33 32 24 23 22 21 21 21 20 19 19 18 18 17 17 17 16 14 14 13 13 8]\nk 28 silhouette 0.283 mean cohesion 0.736 sizes [47 27 26 26 25 24 24 23 21 19 18 17 17 17 16 16 15 15 15 14 13 13 13 13 12 12 10 10]\ncohesion by descending cluster size: [0.79 0.566 0.873 0.693 0.696 0.724 0.689 0.741 0.647 0.646 0.724 0.724 0.762 0.803 0.623 0.708 0.826 0.768 0.702 0.752 0.804 0.727 0.813 0.765 0.717 0.838\n 0.687 0.785]\nlargest inter-center cosines: [0.81 0.81 0.522 0.501 0.451 0.451 0.439 0.421 0.396 0.396 0.392 0.38 0.321 0.301 0.301 0.285 0.282 0.269 0.265 0.26 0.251 0.249 0.215 0.203 0.203 0.188\n 0.181 0.174]\nk 32 silhouette 0.289 mean cohesion 0.747 sizes [47 26 24 24 24 22 22 21 21 18 17 17 17 16 15 15 14 13 13 13 13 13 11 11 11 10 10 9 9 9 7 6]\n\n MLP3 write-column directions\nk 8 silhouette 0.073 mean cohesion 0.418 sizes [253 227 226 204 197 178 147 104]\nk 16 silhouette 0.071 mean cohesion 0.473 sizes [155 123 117 113 101 100 97 95 95 89 87 83 74 73 70 64]\nk 20 silhouette 0.068 mean cohesion 0.491 sizes [134 106 95 85 85 82 80 78 77 75 74 70 67 65 65 63 60 59 59 57]\nk 24 silhouette 0.061 mean cohesion 0.504 sizes [107 82 76 72 72 69 69 65 65 64 63 62 62 61 60 60 60 59 58 55 54 50 48 43]\nk 28 silhouette 0.063 mean cohesion 0.519 sizes [84 82 79 67 66 65 61 60 60 60 58 58 58 54 52 50 48 47 47 46 46 43 43 43 41 40 40 38]\ncohesion by descending cluster size: [0.69 0.523 0.473 0.519 0.507 0.495 0.49 0.508 0.669 0.481 0.528 0.526 0.523 0.506 0.482 0.465 0.536 0.611 0.537 0.479 0.5 0.544 0.516 0.477 0.474 0.519\n 0.475 0.483]\nlargest inter-center cosines: [0.746 0.746 0.657 0.57 0.57 0.536 0.493 0.493 0.459 0.435 0.413 0.412 0.385 0.385 0.376 0.368 0.368 0.353 0.347 0.341 0.341 0.339 0.336 0.316 0.302 0.297\n 0.293 0.257]\nk 32 silhouette 0.055 mean cohesion 0.53 sizes [71 61 58 57 56 54 53 53 52 52 52 51 50 50 50 48 48 48 48 46 45 45 44 43 42 41 41 40 40 36 34 27]\n",
"seconds": 1.908,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "14a750312ce5acc694aff29190bd362bd4fb9c5173bf4f90f3ef2a963539536a",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}L0H5 effective OV first 40 singular values: [1.059 1.044 0.96 0.948 0.909 0.899 0.855 0.836 0.817 0.77 0.753 0.731 0.709 0.676 0.656 0.646 0.604 0.552 0.533 0.454 0.42 0.346 0.332 0.307 0.299 0.292 0.274 0.259 0.237 0.227 0.223 0.216 0.211 0.2 0.189 0.184 0.167 0.153 0.135 0.125] MLP0 active row count: 518 MLP0 active rows restricted to value-read subspace: singular values: [7.422 6.091 5.646 5.175 4.919 4.62 4.555 4.487 4.348 4.284 4.107 3.976 3.947 3.822 3.782 3.724 3.596 3.441 3.248 3.185 2.986 2.693 2.22 1.809 1.556 1.257 1.095] Fraction of each active MLP0 row's energy in that subspace, quantiles: [0.02 0.055 0.096 0.191 0.289 0.433 0.66 ] MLP0 active token-read row directions k 8 silhouette 0.157 mean cohesion 0.571 sizes [119 88 87 84 44 36 32 28] k 16 silhouette 0.236 mean cohesion 0.661 sizes [80 58 51 39 33 29 28 27 27 26 25 22 21 19 19 14] k 20 silhouette 0.269 mean cohesion 0.688 sizes [73 37 33 31 30 27 27 25 25 24 22 21 20 20 19 19 18 18 16 13] k 24 silhouette 0.273 mean cohesion 0.716 sizes [65 33 33 32 24 23 22 21 21 21 20 19 19 18 18 17 17 17 16 14 14 13 13 8] k 28 silhouette 0.283 mean cohesion 0.736 sizes [47 27 26 26 25 24 24 23 21 19 18 17 17 17 16 16 15 15 15 14 13 13 13 13 12 12 10 10] cohesion by descending cluster size: [0.79 0.566 0.873 0.693 0.696 0.724 0.689 0.741 0.647 0.646 0.724 0.724 0.762 0.803 0.623 0.708 0.826 0.768 0.702 0.752 0.804 0.727 0.813 0.765 0.717 0.838 0.687 0.785] largest inter-center cosines: [0.81 0.81 0.522 0.501 0.451 0.451 0.439 0.421 0.396 0.396 0.392 0.38 0.321 0.301 0.301 0.285 0.282 0.269 0.265 0.26 0.251 0.249 0.215 0.203 0.203 0.188 0.181 0.174] k 32 silhouette 0.289 mean cohesion 0.747 sizes [47 26 24 24 24 22 22 21 21 18 17 17 17 16 15 15 14 13 13 13 13 13 11 11 11 10 10 9 9 9 7 6] MLP3 write-column directions k 8 silhouette 0.073 mean cohesion 0.418 sizes [253 227 226 204 197 178 147 104] k 16 silhouette 0.071 mean cohesion 0.473 sizes [155 123 117 113 101 100 97 95 95 89 87 83 74 73 70 64] k 20 silhouette 0.068 mean cohesion 0.491 sizes [134 106 95 85 85 82 80 78 77 75 74 70 67 65 65 63 60 59 59 57] k 24 silhouette 0.061 mean cohesion 0.504 sizes [107 82 76 72 72 69 69 65 65 64 63 62 62 61 60 60 60 59 58 55 54 50 48 43] k 28 silhouette 0.063 mean cohesion 0.519 sizes [84 82 79 67 66 65 61 60 60 60 58 58 58 54 52 50 48 47 47 46 46 43 43 43 41 40 40 38] cohesion by descending cluster size: [0.69 0.523 0.473 0.519 0.507 0.495 0.49 0.508 0.669 0.481 0.528 0.526 0.523 0.506 0.482 0.465 0.536 0.611 0.537 0.479 0.5 0.544 0.516 0.477 0.474 0.519 0.475 0.483] largest inter-center cosines: [0.746 0.746 0.657 0.57 0.57 0.536 0.493 0.493 0.459 0.435 0.413 0.412 0.385 0.385 0.376 0.368 0.368 0.353 0.347 0.341 0.341 0.339 0.336 0.316 0.302 0.297 0.293 0.257] k 32 silhouette 0.055 mean cohesion 0.53 sizes [71 61 58 57 56 54 53 53 52 52 52 51 50 50 50 48 48 48 48 46 45 45 44 43 42 41 41 40 40 36 34 27]
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 5 of at most 100: ``` L0H5 effective OV first 40 singular values: [1.059 1.044 0.96 0.948 0.909 0.899 0.855 0.836 0.817 0.77 0.753 0.731 0.709 0.676 0.656 0.646 0.604 0.552 0.533 0.454 0.42 0.346 0.332 0.307 0.299 0.292 0.274 0.259 0.237 0.227 0.223 0.216 0.211 0.2 0.189 0.184 0.167 0.153 0.135 0.125] MLP0 active row count: 518 MLP0 active rows restricted to value-read subspace: singular values: [7.422 6.091 5.646 5.175 4.919 4.62 4.555 4.487 4.348 4.284 4.107 3.976 3.947 3.822 3.782 3.724 3.596 3.441 3.248 3.185 2.986 2.693 2.22 1.809 1.556 1.257 1.095] Fraction of each active MLP0 row's energy in that subspace, quantiles: [0.02 0.055 0.096 0.191 0.289 0.433 0.66 ] MLP0 active token-read row directions k 8 silhouette 0.157 mean cohesion 0.571 sizes [119 88 87 84 44 36 32 28] k 16 silhouette 0.236 mean cohesion 0.661 sizes [80 58 51 39 33 29 28 27 27 26 25 22 21 19 19 14] k 20 silhouette 0.269 mean cohesion 0.688 sizes [73 37 33 31 30 27 27 25 25 24 22 21 20 20 19 19 18 18 16 13] k 24 silhouette 0.273 mean cohesion 0.716 sizes [65 33 33 32 24 23 22 21 21 21 20 19 19 18 18 17 17 17 16 14 14 13 13 8] k 28 silhouette 0.283 mean cohesion 0.736 sizes [47 27 26 26 25 24 24 23 21 19 18 17 17 17 16 16 15 15 15 14 13 13 13 13 12 12 10 10] cohesion by descending cluster size: [0.79 0.566 0.873 0.693 0.696 0.724 0.689 0.741 0.647 0.646 0.724 0.724 0.762 0.803 0.623 0.708 0.826 0.768 0.702 0.752 0.804 0.727 0.813 0.765 0.717 0.838 0.687 0.785] largest inter-center cosines: [0.81 0.81 0.522 0.501 0.451 0.451 0.439 0.421 0.396 0.396 0.392 0.38 0.321 0.301 0.301 0.285 0.282 0.269 0.265 0.26 0.251 0.249 0.215 0.203 0.203 0.188 0.181 0.174] k 32 silhouette 0.289 mean cohesion 0.747 sizes [47 26 24 24 24 22 22 21 21 18 17 17 17 16 15 15 14 13 13 13 13 13 11 11 11 10 10 9 9 9 7 6] MLP3 write-column directions k 8 silhouette 0.073 mean cohesion 0.418 sizes [253 227 226 204 197 178 147 104] k 16 silhouette 0.071 mean cohesion 0.473 sizes [155 123 117 113 101 100 97 95 95 89 87 83 74 73 70 64] k 20 silhouette 0.068 mean cohesion 0.491 sizes [134 106 95 85 85 82 80 78 77 75 74 70 67 65 65 63 60 59 59 57] k 24 silhouette 0.061 mean cohesion 0.504 sizes [107 82 76 72 72 69 69 65 65 64 63 62 62 61 60 60 60 59 58 55 54 50 48 43] k 28 silhouette 0.063 mean cohesion 0.519 sizes [84 82 79 67 66 65 61 60 60 60 58 58 58 54 52 50 48 47 47 46 46 43 43 43 41 40 40 38] cohesion by descending cluster size: [0.69 0.523 0.473 0.519 0.507 0.495 0.49 0.508 0.669 0.481 0.528 0.526 0.523 0.506 0.482 0.465 0.536 0.611 0.537 0.479 0.5 0.544 0.516 0.477 0.474 0.519 0.475 0.483] largest inter-center cosines: [0.746 0.746 0.657 0.57 0.57 0.536 0.493 0.493 0.459 0.435 0.413 0.412 0.385 0.385 0.376 0.368 0.368 0.353 0.347 0.341 0.341 0.339 0.336 0.316 0.302 0.297 0.293 0.257] k 32 silhouette 0.055 mean cohesion 0.53 sizes [71 61 58 57 56 54 53 53 52 52 52 51 50 50 50 48 48 48 48 46 45 45 44 43 42 41 41 40 40 36 34 27] ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svd
from sklearn.cluster import KMeans
from threadpoolctl import threadpool_limits
threadpool_limits(limits=2)
np.set_printoptions(precision=3, suppress=True, linewidth=160)
def ov0(j):
sl=slice(64*j,64*(j+1))
v=W["W_v.0"][sl].astype(float)*W["ln1_w.0"][None,:]
v-=v.mean(axis=1,keepdims=True)
o=W["W_o.0"][:,sl].astype(float)
o-=o.mean(axis=0,keepdims=True)
return o@v
b=ov0(5)
T=svd(b,full_matrices=False,check_finite=False)[2][:27].T
f=W["W_fc.0"].astype(float)*W["ln2_w.0"][None,:]
f-=f.mean(axis=1,keepdims=True)
active=np.linalg.norm(W["W_fc.0"],axis=1)>1
f=f[active]
direct=f@T
transported=f@b@T
dn=np.linalg.norm(direct,axis=1)
pn=np.linalg.norm(transported,axis=1)
du=direct/dn[:,None]
pu=transported/pn[:,None]
# Cluster the two sets of MLP weight covectors jointly, without evaluating
# any neuron or feeding a representation into the model.
z=np.concatenate([du,pu])
km=KMeans(n_clusters=28,n_init=20,max_iter=300,random_state=42).fit(z)
centers=km.cluster_centers_
centers/=np.linalg.norm(centers,axis=1,keepdims=True)
dc=du@centers.T
pc=pu@centers.T
di=dc.argmax(axis=1); pi=pc.argmax(axis=1)
ds=dc.max(axis=1); ps=pc.max(axis=1)
print("Direct best-center cosine quantiles:",np.quantile(ds,[0,.1,.25,.5,.75,.9,1]))
print("Transported best-center cosine quantiles:",np.quantile(ps,[0,.1,.25,.5,.75,.9,1]))
print("Weight norm quantiles direct / transported:",np.quantile(dn,[.1,.5,.9]),np.quantile(pn,[.1,.5,.9]))
print("Cluster: direct count, transported count, direct/transported mean best cosine")
for c in range(28):
print(c,np.sum(di==c),np.sum(pi==c),
round(float(ds[di==c].mean()),3) if np.any(di==c) else None,
round(float(ps[pi==c].mean()),3) if np.any(pi==c) else None)
for threshold in [.60,.75]:
good=(ds>threshold)&(ps>threshold)
counts=np.zeros((28,28),dtype=int)
np.add.at(counts,(pi[good],di[good]),1)
print("\nThreshold",threshold,"paired neurons",good.sum(),"nonzero directed pairs",np.count_nonzero(counts),
"diagonal count",np.trace(counts))
for c in range(28):
targets=np.argsort(-counts[c])[:5]
print(c,[(int(t),int(counts[c,t])) for t in targets if counts[c,t]])
print("\nDirect versus transported directions of the SAME neuron, cosine quantiles:",
np.quantile(np.sum(du*pu,axis=1),[0,.1,.25,.5,.75,.9,1]))
print("Cluster-center cosine matrix, rounded:")
print(np.round(centers@centers.T,1))
```
{
"run": 6,
"turn": "t006",
"code": "import numpy as np\nfrom scipy.linalg import svd\nfrom sklearn.cluster import KMeans\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(limits=2)\nnp.set_printoptions(precision=3, suppress=True, linewidth=160)\n\ndef ov0(j):\n sl=slice(64*j,64*(j+1))\n v=W[\"W_v.0\"][sl].astype(float)*W[\"ln1_w.0\"][None,:]\n v-=v.mean(axis=1,keepdims=True)\n o=W[\"W_o.0\"][:,sl].astype(float)\n o-=o.mean(axis=0,keepdims=True)\n return o@v\n\nb=ov0(5)\nT=svd(b,full_matrices=False,check_finite=False)[2][:27].T\nf=W[\"W_fc.0\"].astype(float)*W[\"ln2_w.0\"][None,:]\nf-=f.mean(axis=1,keepdims=True)\nactive=np.linalg.norm(W[\"W_fc.0\"],axis=1)>1\nf=f[active]\ndirect=f@T\ntransported=f@b@T\ndn=np.linalg.norm(direct,axis=1)\npn=np.linalg.norm(transported,axis=1)\ndu=direct/dn[:,None]\npu=transported/pn[:,None]\n\n# Cluster the two sets of MLP weight covectors jointly, without evaluating\n# any neuron or feeding a representation into the model.\nz=np.concatenate([du,pu])\nkm=KMeans(n_clusters=28,n_init=20,max_iter=300,random_state=42).fit(z)\ncenters=km.cluster_centers_\ncenters/=np.linalg.norm(centers,axis=1,keepdims=True)\ndc=du@centers.T\npc=pu@centers.T\ndi=dc.argmax(axis=1); pi=pc.argmax(axis=1)\nds=dc.max(axis=1); ps=pc.max(axis=1)\nprint(\"Direct best-center cosine quantiles:\",np.quantile(ds,[0,.1,.25,.5,.75,.9,1]))\nprint(\"Transported best-center cosine quantiles:\",np.quantile(ps,[0,.1,.25,.5,.75,.9,1]))\nprint(\"Weight norm quantiles direct / transported:\",np.quantile(dn,[.1,.5,.9]),np.quantile(pn,[.1,.5,.9]))\nprint(\"Cluster: direct count, transported count, direct/transported mean best cosine\")\nfor c in range(28):\n print(c,np.sum(di==c),np.sum(pi==c),\n round(float(ds[di==c].mean()),3) if np.any(di==c) else None,\n round(float(ps[pi==c].mean()),3) if np.any(pi==c) else None)\n\nfor threshold in [.60,.75]:\n good=(ds>threshold)&(ps>threshold)\n counts=np.zeros((28,28),dtype=int)\n np.add.at(counts,(pi[good],di[good]),1)\n print(\"\\nThreshold\",threshold,\"paired neurons\",good.sum(),\"nonzero directed pairs\",np.count_nonzero(counts),\n \"diagonal count\",np.trace(counts))\n for c in range(28):\n targets=np.argsort(-counts[c])[:5]\n print(c,[(int(t),int(counts[c,t])) for t in targets if counts[c,t]])\n\nprint(\"\\nDirect versus transported directions of the SAME neuron, cosine quantiles:\",\n np.quantile(np.sum(du*pu,axis=1),[0,.1,.25,.5,.75,.9,1]))\nprint(\"Cluster-center cosine matrix, rounded:\")\nprint(np.round(centers@centers.T,1))",
"out": "Direct best-center cosine quantiles: [0.294 0.458 0.562 0.666 0.809 0.922 0.987]\nTransported best-center cosine quantiles: [0.259 0.431 0.51 0.607 0.715 0.804 0.915]\nWeight norm quantiles direct / transported: [0.469 0.847 1.277] [0.205 0.493 0.758]\nCluster: direct count, transported count, direct/transported mean best cosine\n0 10 9 0.63 0.542\n1 56 17 0.767 0.74\n2 24 24 0.529 0.525\n3 16 26 0.615 0.531\n4 10 25 0.703 0.647\n5 25 18 0.593 0.596\n6 28 17 0.664 0.654\n7 21 22 0.523 0.653\n8 16 16 0.621 0.469\n9 10 47 0.609 0.662\n10 24 16 0.658 0.596\n11 25 12 0.715 0.567\n12 10 15 0.776 0.689\n13 27 11 0.846 0.707\n14 16 19 0.574 0.539\n15 25 16 0.69 0.634\n16 16 7 0.753 0.661\n17 22 8 0.743 0.672\n18 16 11 0.737 0.566\n19 16 14 0.688 0.574\n20 16 19 0.772 0.635\n21 1 41 0.344 0.61\n22 1 26 0.527 0.65\n23 15 3 0.765 0.515\n24 10 27 0.579 0.564\n25 29 11 0.544 0.498\n26 20 25 0.749 0.682\n27 13 16 0.662 0.56\n\nThreshold 0.6 paired neurons 174 nonzero directed pairs 122 diagonal count 30\n0 [(1, 1), (3, 1), (23, 1)]\n1 [(1, 8), (13, 1), (24, 1), (17, 1), (7, 1)]\n2 [(3, 2), (1, 1)]\n3 [(1, 1), (3, 1), (16, 1), (26, 1), (15, 1)]\n4 [(0, 4), (24, 1), (20, 1), (15, 1), (10, 1)]\n5 [(1, 2), (10, 2), (13, 1)]\n6 [(6, 4), (26, 2), (27, 1), (1, 1), (19, 1)]\n7 [(6, 3), (17, 1), (1, 1), (15, 1), (5, 1)]\n8 [(27, 1), (1, 1), (20, 1)]\n9 [(17, 3), (1, 2), (10, 2), (13, 1), (25, 1)]\n10 [(15, 2), (10, 1), (11, 1), (14, 1)]\n11 [(1, 2), (11, 2), (15, 1)]\n12 [(20, 1), (15, 1), (12, 1)]\n13 [(13, 2), (1, 2), (24, 1)]\n14 [(16, 1)]\n15 [(27, 2), (1, 2), (17, 2), (5, 1), (18, 1)]\n16 []\n17 [(6, 2), (17, 1), (14, 1)]\n18 [(18, 2), (13, 1)]\n19 [(19, 2), (1, 1), (20, 1), (11, 1), (12, 1)]\n20 [(1, 1), (2, 1), (6, 1), (7, 1), (9, 1)]\n21 [(1, 3), (25, 3), (5, 3), (10, 1), (18, 1)]\n22 [(8, 2), (10, 2), (6, 1)]\n23 []\n24 [(1, 4), (13, 2), (3, 1), (18, 1)]\n25 [(27, 1), (17, 1)]\n26 [(26, 3), (15, 2), (16, 1), (24, 1), (4, 1)]\n27 [(1, 3), (13, 2)]\n\nThreshold 0.75 paired neurons 24 nonzero directed pairs 17 diagonal count 7\n0 []\n1 [(1, 3), (24, 1)]\n2 []\n3 []\n4 [(0, 3)]\n5 [(10, 1)]\n6 [(6, 1)]\n7 [(6, 3)]\n8 []\n9 [(1, 1), (26, 1)]\n10 []\n11 []\n12 []\n13 [(13, 1), (1, 1)]\n14 []\n15 [(17, 1)]\n16 []\n17 [(17, 1)]\n18 []\n19 []\n20 [(6, 1), (9, 1)]\n21 []\n22 []\n23 []\n24 [(1, 1)]\n25 []\n26 [(15, 2), (26, 1)]\n27 []\n\nDirect versus transported directions of the SAME neuron, cosine quantiles: [-0.953 -0.661 -0.41 -0.071 0.257 0.475 0.884]\nCluster-center cosine matrix, rounded:\n[[ 1. 0.2 0. 0.3 0.1 0.2 -0.4 0. 0. 0.1 0.2 -0.1 -0.3 0.1 0.2 0.4 0. 0.1 -0.4 0.2 0.1 0.1 0. 0. -0. 0.2 0. 0.1]\n [ 0.2 1. -0.2 -0.1 0. -0.2 -0.1 -0.2 -0.1 -0. -0.1 0. 0.1 0.3 0.5 -0. 0.1 -0.2 0. 0. -0. 0.4 0. 0.1 -0.8 -0.1 0.1 0.6]\n [ 0. -0.2 1. -0.5 0.1 0.2 0.2 0.1 -0.1 -0.1 0. -0.2 -0.2 -0. -0.1 0. 0.3 0.1 -0. 0.2 0.1 -0.2 0. -0. 0.1 0.2 0.3 -0. ]\n [ 0.3 -0.1 -0.5 1. -0.1 0.1 -0. 0.3 0.1 -0.1 -0.1 0.2 0.1 -0. -0. 0.1 0. -0. -0.1 -0.2 -0. -0.1 -0.2 0.3 0.1 0.1 -0. 0.1]\n [ 0.1 0. 0.1 -0.1 1. -0.1 -0.1 -0.2 0. 0.2 -0.4 -0.2 0.2 0. -0.3 -0.3 -0.1 -0.1 0. -0.2 0.1 0.3 0.1 -0.1 -0.1 -0.1 0.2 -0.4]\n [ 0.2 -0.2 0.2 0.1 -0.1 1. -0.2 0.5 0.3 -0.4 -0. -0.1 -0. 0. -0.2 0.1 0.1 0. -0.1 0.2 0.3 -0.8 -0.4 0.2 0.3 0.2 0.2 0.1]\n [-0.4 -0.1 0.2 -0. -0.1 -0.2 1. -0.2 -0. 0.2 0. -0.1 0.1 -0. 0.4 -0.2 0.1 -0. -0.1 0. -0.1 -0. 0.1 -0.1 -0. 0.1 -0.1 -0.2]\n [ 0. -0.2 0.1 0.3 -0.2 0.5 -0.2 1. 0. -0.8 -0.2 0.2 0.1 -0. -0.5 0.3 0.1 0.3 0.3 0.2 -0.3 -0.3 -0.2 0.1 0.2 0.2 0.2 0.4]\n [ 0. -0.1 -0.1 0.1 0. 0.3 -0. 0. 1. -0.1 0. -0.1 -0. 0.1 0. -0.1 -0.1 -0.1 -0.1 0.1 0.2 -0.3 0.2 -0. -0. -0.1 0. -0.1]\n [ 0.1 -0. -0.1 -0.1 0.2 -0.4 0.2 -0.8 -0.1 1. 0.3 -0.3 -0.1 0. 0.3 -0.3 -0.1 -0.1 -0.3 -0.2 0.4 0.3 0.1 -0.2 0. -0.3 -0.2 -0.6]\n [ 0.2 -0.1 0. -0.1 -0.4 -0. 0. -0.2 0. 0.3 1. -0.1 -0.2 0.1 0.4 -0.1 -0. 0. -0.3 0. -0.1 -0.2 -0.3 -0. 0.1 -0.1 0. -0.1]\n [-0.1 0. -0.2 0.2 -0.2 -0.1 -0.1 0.2 -0.1 -0.3 -0.1 1. -0.2 0.1 0.1 -0. 0.1 -0. -0.2 -0. -0.4 -0. 0. 0.1 -0. 0.1 -0. 0.3]\n [-0.3 0.1 -0.2 0.1 0.2 -0. 0.1 0.1 -0. -0.1 -0.2 -0.2 1. -0.1 -0.2 -0.2 -0. -0.2 -0. -0.5 0.1 0.1 -0.1 -0.1 -0.1 -0.3 -0. 0.1]\n [ 0.1 0.3 -0. -0. 0. 0. -0. -0. 0.1 0. 0.1 0.1 -0.1 1. 0.3 0. 0.1 -0. 0. -0.1 0.1 0.2 -0.3 0.2 -0.4 -0.1 0. 0.2]\n [ 0.2 0.5 -0.1 -0. -0.3 -0.2 0.4 -0.5 0. 0.3 0.4 0.1 -0.2 0.3 1. 0.2 0.1 -0.4 -0.4 0.1 -0. 0.1 -0. 0.1 -0.5 0. 0.1 0.3]\n [ 0.4 -0. 0. 0.1 -0.3 0.1 -0.2 0.3 -0.1 -0.3 -0.1 -0. -0.2 0. 0.2 1. -0. 0.1 -0.1 0.1 -0.2 0.2 0.1 -0.1 0.1 0.1 0. 0.2]\n [ 0. 0.1 0.3 0. -0.1 0.1 0.1 0.1 -0.1 -0.1 -0. 0.1 -0. 0.1 0.1 -0. 1. 0.1 -0. 0.1 -0.1 -0.1 0.1 -0. 0. -0.1 0. 0.1]\n [ 0.1 -0.2 0.1 -0. -0.1 0. -0. 0.3 -0.1 -0.1 0. -0. -0.2 -0. -0.4 0.1 0.1 1. -0. 0.1 -0.1 -0. 0.1 0. 0.1 0. -0. -0.1]\n [-0.4 0. -0. -0.1 0. -0.1 -0.1 0.3 -0.1 -0.3 -0.3 -0.2 -0. 0. -0.4 -0.1 -0. -0. 1. 0.2 -0. 0. -0.1 -0.1 -0. -0.2 0. -0. ]\n [ 0.2 0. 0.2 -0.2 -0.2 0.2 0. 0.2 0.1 -0.2 0. -0. -0.5 -0.1 0.1 0.1 0.1 0.1 0.2 1. -0. -0.3 0.2 0.1 0.2 -0. -0.1 0.1]\n [ 0.1 -0. 0.1 -0. 0.1 0.3 -0.1 -0.3 0.2 0.4 -0.1 -0.4 0.1 0.1 -0. -0.2 -0.1 -0.1 -0. -0. 1. -0.2 -0. 0.1 -0.1 -0. 0. -0.2]\n [ 0.1 0.4 -0.2 -0.1 0.3 -0.8 -0. -0.3 -0.3 0.3 -0.2 -0. 0.1 0.2 0.1 0.2 -0.1 -0. 0. -0.3 -0.2 1. 0.4 -0.4 -0.4 -0.1 0. 0. ]\n [ 0. 0. 0. -0.2 0.1 -0.4 0.1 -0.2 0.2 0.1 -0.3 0. -0.1 -0.3 -0. 0.1 0.1 0.1 -0.1 0.2 -0. 0.4 1. -0.5 -0.1 -0. 0. -0.1]\n [ 0. 0.1 -0. 0.3 -0.1 0.2 -0.1 0.1 -0. -0.2 -0. 0.1 -0.1 0.2 0.1 -0.1 -0. 0. -0.1 0.1 0.1 -0.4 -0.5 1. -0.1 0.2 0. 0.2]\n [-0. -0.8 0.1 0.1 -0.1 0.3 -0. 0.2 -0. 0. 0.1 -0. -0.1 -0.4 -0.5 0.1 0. 0.1 -0. 0.2 -0.1 -0.4 -0.1 -0.1 1. -0. -0.4 -0.5]\n [ 0.2 -0.1 0.2 0.1 -0.1 0.2 0.1 0.2 -0.1 -0.3 -0.1 0.1 -0.3 -0.1 0. 0.1 -0.1 0. -0.2 -0. -0. -0.1 -0. 0.2 -0. 1. 0.1 0.2]\n [ 0. 0.1 0.3 -0. 0.2 0.2 -0.1 0.2 0. -0.2 0. -0. -0. 0. 0.1 0. 0. -0. 0. -0.1 0. 0. 0. 0. -0.4 0.1 1. 0.1]\n [ 0.1 0.6 -0. 0.1 -0.4 0.1 -0.2 0.4 -0.1 -0.6 -0.1 0.3 0.1 0.2 0.3 0.2 0.1 -0.1 -0. 0.1 -0.2 0. -0.1 0.2 -0.5 0.2 0.1 1. ]]\n",
"seconds": 1.343,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "019abcf2c1c572d885aa93c38866744fd6e6c90b96800d0521c370bbb533701e",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}Direct best-center cosine quantiles: [0.294 0.458 0.562 0.666 0.809 0.922 0.987] Transported best-center cosine quantiles: [0.259 0.431 0.51 0.607 0.715 0.804 0.915] Weight norm quantiles direct / transported: [0.469 0.847 1.277] [0.205 0.493 0.758] Cluster: direct count, transported count, direct/transported mean best cosine 0 10 9 0.63 0.542 1 56 17 0.767 0.74 2 24 24 0.529 0.525 3 16 26 0.615 0.531 4 10 25 0.703 0.647 5 25 18 0.593 0.596 6 28 17 0.664 0.654 7 21 22 0.523 0.653 8 16 16 0.621 0.469 9 10 47 0.609 0.662 10 24 16 0.658 0.596 11 25 12 0.715 0.567 12 10 15 0.776 0.689 13 27 11 0.846 0.707 14 16 19 0.574 0.539 15 25 16 0.69 0.634 16 16 7 0.753 0.661 17 22 8 0.743 0.672 18 16 11 0.737 0.566 19 16 14 0.688 0.574 20 16 19 0.772 0.635 21 1 41 0.344 0.61 22 1 26 0.527 0.65 23 15 3 0.765 0.515 24 10 27 0.579 0.564 25 29 11 0.544 0.498 26 20 25 0.749 0.682 27 13 16 0.662 0.56 Threshold 0.6 paired neurons 174 nonzero directed pairs 122 diagonal count 30 0 [(1, 1), (3, 1), (23, 1)] 1 [(1, 8), (13, 1), (24, 1), (17, 1), (7, 1)] 2 [(3, 2), (1, 1)] 3 [(1, 1), (3, 1), (16, 1), (26, 1), (15, 1)] 4 [(0, 4), (24, 1), (20, 1), (15, 1), (10, 1)] 5 [(1, 2), (10, 2), (13, 1)] 6 [(6, 4), (26, 2), (27, 1), (1, 1), (19, 1)] 7 [(6, 3), (17, 1), (1, 1), (15, 1), (5, 1)] 8 [(27, 1), (1, 1), (20, 1)] 9 [(17, 3), (1, 2), (10, 2), (13, 1), (25, 1)] 10 [(15, 2), (10, 1), (11, 1), (14, 1)] 11 [(1, 2), (11, 2), (15, 1)] 12 [(20, 1), (15, 1), (12, 1)] 13 [(13, 2), (1, 2), (24, 1)] 14 [(16, 1)] 15 [(27, 2), (1, 2), (17, 2), (5, 1), (18, 1)] 16 [] 17 [(6, 2), (17, 1), (14, 1)] 18 [(18, 2), (13, 1)] 19 [(19, 2), (1, 1), (20, 1), (11, 1), (12, 1)] 20 [(1, 1), (2, 1), (6, 1), (7, 1), (9, 1)] 21 [(1, 3), (25, 3), (5, 3), (10, 1), (18, 1)] 22 [(8, 2), (10, 2), (6, 1)] 23 [] 24 [(1, 4), (13, 2), (3, 1), (18, 1)] 25 [(27, 1), (17, 1)] 26 [(26, 3), (15, 2), (16, 1), (24, 1), (4, 1)] 27 [(1, 3), (13, 2)] Threshold 0.75 paired neurons 24 nonzero directed pairs 17 diagonal count 7 0 [] 1 [(1, 3), (24, 1)] 2 [] 3 [] 4 [(0, 3)] 5 [(10, 1)] 6 [(6, 1)] 7 [(6, 3)] 8 [] 9 [(1, 1), (26, 1)] 10 [] 11 [] 12 [] 13 [(13, 1), (1, 1)] 14 [] 15 [(17, 1)] 16 [] 17 [(17, 1)] 18 [] 19 [] 20 [(6, 1), (9, 1)] 21 [] 22 [] 23 [] 24 [(1, 1)] 25 [] 26 [(15, 2), (26, 1)] 27 [] Direct versus transported directions of the SAME neuron, cosine quantiles: [-0.953 -0.661 -0.41 -0.071 0.257 0.475 0.884] Cluster-center cosine matrix, rounded: [[ 1. 0.2 0. 0.3 0.1 0.2 -0.4 0. 0. 0.1 0.2 -0.1 -0.3 0.1 0.2 0.4 0. 0.1 -0.4 0.2 0.1 0.1 0. 0. -0. 0.2 0. 0.1] [ 0.2 1. -0.2 -0.1 0. -0.2 -0.1 -0.2 -0.1 -0. -0.1 0. 0.1 0.3 0.5 -0. 0.1 -0.2 0. 0. -0. 0.4 0. 0.1 -0.8 -0.1 0.1 0.6] [ 0. -0.2 1. -0.5 0.1 0.2 0.2 0.1 -0.1 -0.1 0. -0.2 -0.2 -0. -0.1 0. 0.3 0.1 -0. 0.2 0.1 -0.2 0. -0. 0.1 0.2 0.3 -0. ] [ 0.3 -0.1 -0.5 1. -0.1 0.1 -0. 0.3 0.1 -0.1 -0.1 0.2 0.1 -0. -0. 0.1 0. -0. -0.1 -0.2 -0. -0.1 -0.2 0.3 0.1 0.1 -0. 0.1] [ 0.1 0. 0.1 -0.1 1. -0.1 -0.1 -0.2 0. 0.2 -0.4 -0.2 0.2 0. -0.3 -0.3 -0.1 -0.1 0. -0.2 0.1 0.3 0.1 -0.1 -0.1 -0.1 0.2 -0.4] [ 0.2 -0.2 0.2 0.1 -0.1 1. -0.2 0.5 0.3 -0.4 -0. -0.1 -0. 0. -0.2 0.1 0.1 0. -0.1 0.2 0.3 -0.8 -0.4 0.2 0.3 0.2 0.2 0.1] [-0.4 -0.1 0.2 -0. -0.1 -0.2 1. -0.2 -0. 0.2 0. -0.1 0.1 -0. 0.4 -0.2 0.1 -0. -0.1 0. -0.1 -0. 0.1 -0.1 -0. 0.1 -0.1 -0.2] [ 0. -0.2 0.1 0.3 -0.2 0.5 -0.2 1. 0. -0.8 -0.2 0.2 0.1 -0. -0.5 0.3 0.1 0.3 0.3 0.2 -0.3 -0.3 -0.2 0.1 0.2 0.2 0.2 0.4] [ 0. -0.1 -0.1 0.1 0. 0.3 -0. 0. 1. -0.1 0. -0.1 -0. 0.1 0. -0.1 -0.1 -0.1 -0.1 0.1 0.2 -0.3 0.2 -0. -0. -0.1 0. -0.1] [ 0.1 -0. -0.1 -0.1 0.2 -0.4 0.2 -0.8 -0.1 1. 0.3 -0.3 -0.1 0. 0.3 -0.3 -0.1 -0.1 -0.3 -0.2 0.4 0.3 0.1 -0.2 0. -0.3 -0.2 -0.6] [ 0.2 -0.1 0. -0.1 -0.4 -0. 0. -0.2 0. 0.3 1. -0.1 -0.2 0.1 0.4 -0.1 -0. 0. -0.3 0. -0.1 -0.2 -0.3 -0. 0.1 -0.1 0. -0.1] [-0.1 0. -0.2 0.2 -0.2 -0.1 -0.1 0.2 -0.1 -0.3 -0.1 1. -0.2 0.1 0.1 -0. 0.1 -0. -0.2 -0. -0.4 -0. 0. 0.1 -0. 0.1 -0. 0.3] [-0.3 0.1 -0.2 0.1 0.2 -0. 0.1 0.1 -0. -0.1 -0.2 -0.2 1. -0.1 -0.2 -0.2 -0. -0.2 -0. -0.5 0.1 0.1 -0.1 -0.1 -0.1 -0.3 -0. 0.1] [ 0.1 0.3 -0. -0. 0. 0. -0. -0. 0.1 0. 0.1 0.1 -0.1 1. 0.3 0. 0.1 -0. 0. -0.1 0.1 0.2 -0.3 0.2 -0.4 -0.1 0. 0.2] [ 0.2 0.5 -0.1 -0. -0.3 -0.2 0.4 -0.5 0. 0.3 0.4 0.1 -0.2 0.3 1. 0.2 0.1 -0.4 -0.4 0.1 -0. 0.1 -0. 0.1 -0.5 0. 0.1 0.3] [ 0.4 -0. 0. 0.1 -0.3 0.1 -0.2 0.3 -0.1 -0.3 -0.1 -0. -0.2 0. 0.2 1. -0. 0.1 -0.1 0.1 -0.2 0.2 0.1 -0.1 0.1 0.1 0. 0.2] [ 0. 0.1 0.3 0. -0.1 0.1 0.1 0.1 -0.1 -0.1 -0. 0.1 -0. 0.1 0.1 -0. 1. 0.1 -0. 0.1 -0.1 -0.1 0.1 -0. 0. -0.1 0. 0.1] [ 0.1 -0.2 0.1 -0. -0.1 0. -0. 0.3 -0.1 -0.1 0. -0. -0.2 -0. -0.4 0.1 0.1 1. -0. 0.1 -0.1 -0. 0.1 0. 0.1 0. -0. -0.1] [-0.4 0. -0. -0.1 0. -0.1 -0.1 0.3 -0.1 -0.3 -0.3 -0.2 -0. 0. -0.4 -0.1 -0. -0. 1. 0.2 -0. 0. -0.1 -0.1 -0. -0.2 0. -0. ] [ 0.2 0. 0.2 -0.2 -0.2 0.2 0. 0.2 0.1 -0.2 0. -0. -0.5 -0.1 0.1 0.1 0.1 0.1 0.2 1. -0. -0.3 0.2 0.1 0.2 -0. -0.1 0.1] [ 0.1 -0. 0.1 -0. 0.1 0.3 -0.1 -0.3 0.2 0.4 -0.1 -0.4 0.1 0.1 -0. -0.2 -0.1 -0.1 -0. -0. 1. -0.2 -0. 0.1 -0.1 -0. 0. -0.2] [ 0.1 0.4 -0.2 -0.1 0.3 -0.8 -0. -0.3 -0.3 0.3 -0.2 -0. 0.1 0.2 0.1 0.2 -0.1 -0. 0. -0.3 -0.2 1. 0.4 -0.4 -0.4 -0.1 0. 0. ] [ 0. 0. 0. -0.2 0.1 -0.4 0.1 -0.2 0.2 0.1 -0.3 0. -0.1 -0.3 -0. 0.1 0.1 0.1 -0.1 0.2 -0. 0.4 1. -0.5 -0.1 -0. 0. -0.1] [ 0. 0.1 -0. 0.3 -0.1 0.2 -0.1 0.1 -0. -0.2 -0. 0.1 -0.1 0.2 0.1 -0.1 -0. 0. -0.1 0.1 0.1 -0.4 -0.5 1. -0.1 0.2 0. 0.2] [-0. -0.8 0.1 0.1 -0.1 0.3 -0. 0.2 -0. 0. 0.1 -0. -0.1 -0.4 -0.5 0.1 0. 0.1 -0. 0.2 -0.1 -0.4 -0.1 -0.1 1. -0. -0.4 -0.5] [ 0.2 -0.1 0.2 0.1 -0.1 0.2 0.1 0.2 -0.1 -0.3 -0.1 0.1 -0.3 -0.1 0. 0.1 -0.1 0. -0.2 -0. -0. -0.1 -0. 0.2 -0. 1. 0.1 0.2] [ 0. 0.1 0.3 -0. 0.2 0.2 -0.1 0.2 0. -0.2 0. -0. -0. 0. 0.1 0. 0. -0. 0. -0.1 0. 0. 0. 0. -0.4 0.1 1. 0.1] [ 0.1 0.6 -0. 0.1 -0.4 0.1 -0.2 0.4 -0.1 -0.6 -0.1 0.3 0.1 0.2 0.3 0.2 0.1 -0.1 -0. 0.1 -0.2 0. -0.1 0.2 -0.5 0.2 0.1 1. ]]
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 6 of at most 100: ``` Direct best-center cosine quantiles: [0.294 0.458 0.562 0.666 0.809 0.922 0.987] Transported best-center cosine quantiles: [0.259 0.431 0.51 0.607 0.715 0.804 0.915] Weight norm quantiles direct / transported: [0.469 0.847 1.277] [0.205 0.493 0.758] Cluster: direct count, transported count, direct/transported mean best cosine 0 10 9 0.63 0.542 1 56 17 0.767 0.74 2 24 24 0.529 0.525 3 16 26 0.615 0.531 4 10 25 0.703 0.647 5 25 18 0.593 0.596 6 28 17 0.664 0.654 7 21 22 0.523 0.653 8 16 16 0.621 0.469 9 10 47 0.609 0.662 10 24 16 0.658 0.596 11 25 12 0.715 0.567 12 10 15 0.776 0.689 13 27 11 0.846 0.707 14 16 19 0.574 0.539 15 25 16 0.69 0.634 16 16 7 0.753 0.661 17 22 8 0.743 0.672 18 16 11 0.737 0.566 19 16 14 0.688 0.574 20 16 19 0.772 0.635 21 1 41 0.344 0.61 22 1 26 0.527 0.65 23 15 3 0.765 0.515 24 10 27 0.579 0.564 25 29 11 0.544 0.498 26 20 25 0.749 0.682 27 13 16 0.662 0.56 Threshold 0.6 paired neurons 174 nonzero directed pairs 122 diagonal count 30 0 [(1, 1), (3, 1), (23, 1)] 1 [(1, 8), (13, 1), (24, 1), (17, 1), (7, 1)] 2 [(3, 2), (1, 1)] 3 [(1, 1), (3, 1), (16, 1), (26, 1), (15, 1)] 4 [(0, 4), (24, 1), (20, 1), (15, 1), (10, 1)] 5 [(1, 2), (10, 2), (13, 1)] 6 [(6, 4), (26, 2), (27, 1), (1, 1), (19, 1)] 7 [(6, 3), (17, 1), (1, 1), (15, 1), (5, 1)] 8 [(27, 1), (1, 1), (20, 1)] 9 [(17, 3), (1, 2), (10, 2), (13, 1), (25, 1)] 10 [(15, 2), (10, 1), (11, 1), (14, 1)] 11 [(1, 2), (11, 2), (15, 1)] 12 [(20, 1), (15, 1), (12, 1)] 13 [(13, 2), (1, 2), (24, 1)] 14 [(16, 1)] 15 [(27, 2), (1, 2), (17, 2), (5, 1), (18, 1)] 16 [] 17 [(6, 2), (17, 1), (14, 1)] 18 [(18, 2), (13, 1)] 19 [(19, 2), (1, 1), (20, 1), (11, 1), (12, 1)] 20 [(1, 1), (2, 1), (6, 1), (7, 1), (9, 1)] 21 [(1, 3), (25, 3), (5, 3), (10, 1), (18, 1)] 22 [(8, 2), (10, 2), (6, 1)] 23 [] 24 [(1, 4), (13, 2), (3, 1), (18, 1)] 25 [(27, 1), (17, 1)] 26 [(26, 3), (15, 2), (16, 1), (24, 1), (4, 1)] 27 [(1, 3), (13, 2)] Threshold 0.75 paired neurons 24 nonzero directed pairs 17 diagonal count 7 0 [] 1 [(1, 3), (24, 1)] 2 [] 3 [] 4 [(0, 3)] 5 [(10, 1)] 6 [(6, 1)] 7 [(6, 3)] 8 [] 9 [(1, 1), (26, 1)] 10 [] 11 [] 12 [] 13 [(13, 1), (1, 1)] 14 [] 15 [(17, 1)] 16 [] 17 [(17, 1)] 18 [] 19 [] 20 [(6, 1), (9, 1)] 21 [] 22 [] 23 [] 24 [(1, 1)] 25 [] 26 [(15, 2), (26, 1)] 27 [] Direct versus transported directions of the SAME neuron, cosine quantiles: [-0.953 -0.661 -0.41 -0.071 0.257 0.475 0.884] Cluster-center cosine matrix, rounded: [[ 1. 0.2 0. 0.3 0.1 0.2 -0.4 0. 0. 0.1 0.2 -0.1 -0.3 0.1 0.2 0.4 0. 0.1 -0.4 0.2 0.1 0.1 0. 0. -0. 0.2 0. 0.1] [ 0.2 1. -0.2 -0.1 0. -0.2 -0.1 -0.2 -0.1 -0. -0.1 0. 0.1 0.3 0.5 -0. 0.1 -0.2 0. 0. -0. 0.4 0. 0.1 -0.8 -0.1 0.1 0.6] [ 0. -0.2 1. -0.5 0.1 0.2 0.2 0.1 -0.1 -0.1 0. -0.2 -0.2 -0. -0.1 0. 0.3 0.1 -0. 0.2 0.1 -0.2 0. -0. 0.1 0.2 0.3 -0. ] [ 0.3 -0.1 -0.5 1. -0.1 0.1 -0. 0.3 0.1 -0.1 -0.1 0.2 0.1 -0. -0. 0.1 0. -0. -0.1 -0.2 -0. -0.1 -0.2 0.3 0.1 0.1 -0. 0.1] [ 0.1 0. 0.1 -0.1 1. -0.1 -0.1 -0.2 0. 0.2 -0.4 -0.2 0.2 0. -0.3 -0.3 -0.1 -0.1 0. -0.2 0.1 0.3 0.1 -0.1 -0.1 -0.1 0.2 -0.4] [ 0.2 -0.2 0.2 0.1 -0.1 1. -0.2 0.5 0.3 -0.4 -0. -0.1 -0. 0. -0.2 0.1 0.1 0. -0.1 0.2 0.3 -0.8 -0.4 0.2 0.3 0.2 0.2 0.1] [-0.4 -0.1 0.2 -0. -0.1 -0.2 1. -0.2 -0. 0.2 0. -0.1 0.1 -0. 0.4 -0.2 0.1 -0. -0.1 0. -0.1 -0. 0.1 -0.1 -0. 0.1 -0.1 -0.2] [ 0. -0.2 0.1 0.3 -0.2 0.5 -0.2 1. 0. -0.8 -0.2 0.2 0.1 -0. -0.5 0.3 0.1 0.3 0.3 0.2 -0.3 -0.3 -0.2 0.1 0.2 0.2 0.2 0.4] [ 0. -0.1 -0.1 0.1 0. 0.3 -0. 0. 1. -0.1 0. -0.1 -0. 0.1 0. -0.1 -0.1 -0.1 -0.1 0.1 0.2 -0.3 0.2 -0. -0. -0.1 0. -0.1] [ 0.1 -0. -0.1 -0.1 0.2 -0.4 0.2 -0.8 -0.1 1. 0.3 -0.3 -0.1 0. 0.3 -0.3 -0.1 -0.1 -0.3 -0.2 0.4 0.3 0.1 -0.2 0. -0.3 -0.2 -0.6] [ 0.2 -0.1 0. -0.1 -0.4 -0. 0. -0.2 0. 0.3 1. -0.1 -0.2 0.1 0.4 -0.1 -0. 0. -0.3 0. -0.1 -0.2 -0.3 -0. 0.1 -0.1 0. -0.1] [-0.1 0. -0.2 0.2 -0.2 -0.1 -0.1 0.2 -0.1 -0.3 -0.1 1. -0.2 0.1 0.1 -0. 0.1 -0. -0.2 -0. -0.4 -0. 0. 0.1 -0. 0.1 -0. 0.3] [-0.3 0.1 -0.2 0.1 0.2 -0. 0.1 0.1 -0. -0.1 -0.2 -0.2 1. -0.1 -0.2 -0.2 -0. -0.2 -0. -0.5 0.1 0.1 -0.1 -0.1 -0.1 -0.3 -0. 0.1] [ 0.1 0.3 -0. -0. 0. 0. -0. -0. 0.1 0. 0.1 0.1 -0.1 1. 0.3 0. 0.1 -0. 0. -0.1 0.1 0.2 -0.3 0.2 -0.4 -0.1 0. 0.2] [ 0.2 0.5 -0.1 -0. -0.3 -0.2 0.4 -0.5 0. 0.3 0.4 0.1 -0.2 0.3 1. 0.2 0.1 -0.4 -0.4 0.1 -0. 0.1 -0. 0.1 -0.5 0. 0.1 0.3] [ 0.4 -0. 0. 0.1 -0.3 0.1 -0.2 0.3 -0.1 -0.3 -0.1 -0. -0.2 0. 0.2 1. -0. 0.1 -0.1 0.1 -0.2 0.2 0.1 -0.1 0.1 0.1 0. 0.2] [ 0. 0.1 0.3 0. -0.1 0.1 0.1 0.1 -0.1 -0.1 -0. 0.1 -0. 0.1 0.1 -0. 1. 0.1 -0. 0.1 -0.1 -0.1 0.1 -0. 0. -0.1 0. 0.1] [ 0.1 -0.2 0.1 -0. -0.1 0. -0. 0.3 -0.1 -0.1 0. -0. -0.2 -0. -0.4 0.1 0.1 1. -0. 0.1 -0.1 -0. 0.1 0. 0.1 0. -0. -0.1] [-0.4 0. -0. -0.1 0. -0.1 -0.1 0.3 -0.1 -0.3 -0.3 -0.2 -0. 0. -0.4 -0.1 -0. -0. 1. 0.2 -0. 0. -0.1 -0.1 -0. -0.2 0. -0. ] [ 0.2 0. 0.2 -0.2 -0.2 0.2 0. 0.2 0.1 -0.2 0. -0. -0.5 -0.1 0.1 0.1 0.1 0.1 0.2 1. -0. -0.3 0.2 0.1 0.2 -0. -0.1 0.1] [ 0.1 -0. 0.1 -0. 0.1 0.3 -0.1 -0.3 0.2 0.4 -0.1 -0.4 0.1 0.1 -0. -0.2 -0.1 -0.1 -0. -0. 1. -0.2 -0. 0.1 -0.1 -0. 0. -0.2] [ 0.1 0.4 -0.2 -0.1 0.3 -0.8 -0. -0.3 -0.3 0.3 -0.2 -0. 0.1 0.2 0.1 0.2 -0.1 -0. 0. -0.3 -0.2 1. 0.4 -0.4 -0.4 -0.1 0. 0. ] [ 0. 0. 0. -0.2 0.1 -0.4 0.1 -0.2 0.2 0.1 -0.3 0. -0.1 -0.3 -0. 0.1 0.1 0.1 -0.1 0.2 -0. 0.4 1. -0.5 -0.1 -0. 0. -0.1] [ 0. 0.1 -0. 0.3 -0.1 0.2 -0.1 0.1 -0. -0.2 -0. 0.1 -0.1 0.2 0.1 -0.1 -0. 0. -0.1 0.1 0.1 -0.4 -0.5 1. -0.1 0.2 0. 0.2] [-0. -0.8 0.1 0.1 -0.1 0.3 -0. 0.2 -0. 0. 0.1 -0. -0.1 -0.4 -0.5 0.1 0. 0.1 -0. 0.2 -0.1 -0.4 -0.1 -0.1 1. -0. -0.4 -0.5] [ 0.2 -0.1 0.2 0.1 -0.1 0.2 0.1 0.2 -0.1 -0.3 -0.1 0.1 -0.3 -0.1 0. 0.1 -0.1 0. -0.2 -0. -0. -0.1 -0. 0.2 -0. 1. 0.1 0.2] [ 0. 0.1 0.3 -0. 0.2 0.2 -0.1 0.2 0. -0.2 0. -0. -0. 0. 0.1 0. 0. -0. 0. -0.1 0. 0. 0. 0. -0.4 0.1 1. 0.1] [ 0.1 0.6 -0. 0.1 -0.4 0.1 -0.2 0.4 -0.1 -0.6 -0.1 0.3 0.1 0.2 0.3 0.2 0.1 -0.1 -0. 0.1 -0.2 0. -0.1 0.2 -0.5 0.2 0.1 1. ]] ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svd
from threadpoolctl import threadpool_limits
threadpool_limits(limits=2)
np.set_printoptions(precision=3,suppress=True,linewidth=160)
A=[]; B=[]; kbias=[]
for l in range(4):
for j in range(6):
sl=slice(64*j,64*(j+1))
qr=W[f"W_q.{l}"][sl].astype(float)
kr=W[f"W_k.{l}"][sl].astype(float)
vr=W[f"W_v.{l}"][sl].astype(float)
g=W[f"ln1_w.{l}"].astype(float)
q=qr*g; k=kr*g; v=vr*g
q-=q.mean(axis=1,keepdims=True)
k-=k.mean(axis=1,keepdims=True)
v-=v.mean(axis=1,keepdims=True)
o=W[f"W_o.{l}"][:,sl].astype(float)
o-=o.mean(axis=0,keepdims=True)
A.append(q.T@k/8)
B.append(o@v)
kbias.append(k.T@(qr@W[f"ln1_b.{l}"])/8)
u4,s4,v4=svd(B[4],full_matrices=False,check_finite=False)
u5,s5,v5=svd(B[5],full_matrices=False,check_finite=False)
uP,sP,vP=svd(np.concatenate([A[1],A[1].T,A[5],A[5].T],axis=1),
full_matrices=False,check_finite=False)
bases={
"uniform-head-write3":u4[:,:3],
"uniform-head-read3":v4[:3].T,
"H5-value-read27":v5[:27].T,
"H5-value-write27":u5[:,:27],
"L0-H1H5-QK-shared20":uP[:,:20],
}
print("L0H4 effective OV singular values:",s4[:12])
print("Basis pair overlaps, normalized by smaller rank:")
names=list(bases)
print(names)
print(np.array([[np.linalg.norm(bases[a].T@bases[b])**2/min(bases[a].shape[1],bases[b].shape[1])
for b in names]for a in names]))
for name,U in bases.items():
print("\n",name,": QK query-energy/key-energy; bias-vector energy")
for i in range(24):
a=A[i]; z=kbias[i]
print("L%dH%d %.3f %.3f %.3f"%(i//6,i%6,
np.linalg.norm(U.T@a)**2/np.linalg.norm(a)**2,
np.linalg.norm(a@U)**2/np.linalg.norm(a)**2,
np.linalg.norm(U.T@z)**2/(np.linalg.norm(z)**2+1e-30)))
print("\nPairwise cosine of bias-derived key coefficients across all heads")
Z=np.stack(kbias);Z/=np.linalg.norm(Z,axis=1,keepdims=True)
for i in range(24):
row=Z@Z[i];row[i]=-2
idx=np.argsort(-row)[:4]
print("L%dH%d"%(i//6,i%6),
[("L%dH%d"%(k//6,k%6),round(float(row[k]),3))for k in idx])
```
{
"run": 7,
"turn": "t007",
"code": "import numpy as np\nfrom scipy.linalg import svd\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(limits=2)\nnp.set_printoptions(precision=3,suppress=True,linewidth=160)\n\nA=[]; B=[]; kbias=[]\nfor l in range(4):\n for j in range(6):\n sl=slice(64*j,64*(j+1))\n qr=W[f\"W_q.{l}\"][sl].astype(float)\n kr=W[f\"W_k.{l}\"][sl].astype(float)\n vr=W[f\"W_v.{l}\"][sl].astype(float)\n g=W[f\"ln1_w.{l}\"].astype(float)\n q=qr*g; k=kr*g; v=vr*g\n q-=q.mean(axis=1,keepdims=True)\n k-=k.mean(axis=1,keepdims=True)\n v-=v.mean(axis=1,keepdims=True)\n o=W[f\"W_o.{l}\"][:,sl].astype(float)\n o-=o.mean(axis=0,keepdims=True)\n A.append(q.T@k/8)\n B.append(o@v)\n kbias.append(k.T@(qr@W[f\"ln1_b.{l}\"])/8)\n\nu4,s4,v4=svd(B[4],full_matrices=False,check_finite=False)\nu5,s5,v5=svd(B[5],full_matrices=False,check_finite=False)\nuP,sP,vP=svd(np.concatenate([A[1],A[1].T,A[5],A[5].T],axis=1),\n full_matrices=False,check_finite=False)\nbases={\n \"uniform-head-write3\":u4[:,:3],\n \"uniform-head-read3\":v4[:3].T,\n \"H5-value-read27\":v5[:27].T,\n \"H5-value-write27\":u5[:,:27],\n \"L0-H1H5-QK-shared20\":uP[:,:20],\n}\nprint(\"L0H4 effective OV singular values:\",s4[:12])\nprint(\"Basis pair overlaps, normalized by smaller rank:\")\nnames=list(bases)\nprint(names)\nprint(np.array([[np.linalg.norm(bases[a].T@bases[b])**2/min(bases[a].shape[1],bases[b].shape[1])\n for b in names]for a in names]))\n\nfor name,U in bases.items():\n print(\"\\n\",name,\": QK query-energy/key-energy; bias-vector energy\")\n for i in range(24):\n a=A[i]; z=kbias[i]\n print(\"L%dH%d %.3f %.3f %.3f\"%(i//6,i%6,\n np.linalg.norm(U.T@a)**2/np.linalg.norm(a)**2,\n np.linalg.norm(a@U)**2/np.linalg.norm(a)**2,\n np.linalg.norm(U.T@z)**2/(np.linalg.norm(z)**2+1e-30)))\n\nprint(\"\\nPairwise cosine of bias-derived key coefficients across all heads\")\nZ=np.stack(kbias);Z/=np.linalg.norm(Z,axis=1,keepdims=True)\nfor i in range(24):\n row=Z@Z[i];row[i]=-2\n idx=np.argsort(-row)[:4]\n print(\"L%dH%d\"%(i//6,i%6),\n [(\"L%dH%d\"%(k//6,k%6),round(float(row[k]),3))for k in idx])",
"out": "L0H4 effective OV singular values: [13.877 9.644 2.507 0.968 0.594 0.554 0.483 0.433 0.345 0.295 0.28 0.228]\nBasis pair overlaps, normalized by smaller rank:\n['uniform-head-write3', 'uniform-head-read3', 'H5-value-read27', 'H5-value-write27', 'L0-H1H5-QK-shared20']\n[[1. 0.098 0.137 0.031 0.26 ]\n [0.098 1. 0.075 0.06 0.848]\n [0.137 0.075 1. 0.066 0.179]\n [0.031 0.06 0.066 1. 0.205]\n [0.26 0.848 0.179 0.205 1. ]]\n\n uniform-head-write3 : QK query-energy/key-energy; bias-vector energy\nL0H0 0.280 0.102 0.094\nL0H1 0.079 0.054 0.126\nL0H2 0.144 0.025 0.005\nL0H3 0.319 0.049 0.061\nL0H4 0.138 0.054 0.067\nL0H5 0.031 0.039 0.119\nL1H0 0.061 0.342 0.765\nL1H1 0.107 0.627 0.905\nL1H2 0.108 0.631 0.908\nL1H3 0.137 0.678 0.931\nL1H4 0.119 0.645 0.919\nL1H5 0.084 0.525 0.825\nL2H0 0.010 0.051 0.465\nL2H1 0.005 0.005 0.040\nL2H2 0.014 0.070 0.858\nL2H3 0.022 0.095 0.804\nL2H4 0.023 0.095 0.672\nL2H5 0.006 0.041 0.270\nL3H0 0.002 0.017 0.086\nL3H1 0.003 0.009 0.078\nL3H2 0.003 0.025 0.234\nL3H3 0.005 0.010 0.052\nL3H4 0.004 0.025 0.160\nL3H5 0.003 0.005 0.074\n\n uniform-head-read3 : QK query-energy/key-energy; bias-vector energy\nL0H0 0.755 0.223 0.298\nL0H1 0.567 0.608 0.643\nL0H2 0.680 0.511 0.633\nL0H3 0.623 0.414 0.302\nL0H4 0.030 0.086 0.216\nL0H5 0.249 0.281 0.148\nL1H0 0.009 0.039 0.101\nL1H1 0.015 0.090 0.137\nL1H2 0.018 0.072 0.111\nL1H3 0.012 0.082 0.128\nL1H4 0.022 0.084 0.135\nL1H5 0.008 0.072 0.105\nL2H0 0.008 0.014 0.074\nL2H1 0.010 0.009 0.010\nL2H2 0.009 0.018 0.127\nL2H3 0.010 0.018 0.126\nL2H4 0.008 0.022 0.114\nL2H5 0.009 0.012 0.028\nL3H0 0.008 0.008 0.013\nL3H1 0.008 0.010 0.017\nL3H2 0.007 0.010 0.061\nL3H3 0.006 0.009 0.010\nL3H4 0.007 0.011 0.017\nL3H5 0.009 0.008 0.002\n\n H5-value-read27 : QK query-energy/key-energy; bias-vector energy\nL0H0 0.059 0.391 0.423\nL0H1 0.105 0.143 0.256\nL0H2 0.131 0.293 0.199\nL0H3 0.113 0.264 0.194\nL0H4 0.085 0.101 0.101\nL0H5 0.150 0.168 0.219\nL1H0 0.074 0.175 0.264\nL1H1 0.121 0.233 0.290\nL1H2 0.084 0.199 0.273\nL1H3 0.100 0.184 0.230\nL1H4 0.070 0.200 0.253\nL1H5 0.097 0.227 0.288\nL2H0 0.076 0.084 0.141\nL2H1 0.087 0.078 0.073\nL2H2 0.102 0.114 0.194\nL2H3 0.071 0.097 0.224\nL2H4 0.073 0.091 0.176\nL2H5 0.066 0.081 0.146\nL3H0 0.085 0.075 0.132\nL3H1 0.085 0.071 0.083\nL3H2 0.075 0.078 0.100\nL3H3 0.069 0.074 0.061\nL3H4 0.070 0.076 0.109\nL3H5 0.077 0.061 0.097\n\n H5-value-write27 : QK query-energy/key-energy; bias-vector energy\nL0H0 0.054 0.056 0.039\nL0H1 0.108 0.094 0.069\nL0H2 0.052 0.050 0.063\nL0H3 0.051 0.072 0.101\nL0H4 0.059 0.069 0.071\nL0H5 0.172 0.163 0.081\nL1H0 0.162 0.109 0.033\nL1H1 0.122 0.060 0.022\nL1H2 0.134 0.052 0.019\nL1H3 0.138 0.056 0.014\nL1H4 0.149 0.059 0.014\nL1H5 0.146 0.079 0.038\nL2H0 0.091 0.094 0.052\nL2H1 0.120 0.135 0.113\nL2H2 0.159 0.118 0.012\nL2H3 0.077 0.089 0.031\nL2H4 0.091 0.098 0.124\nL2H5 0.072 0.078 0.095\nL3H0 0.079 0.083 0.067\nL3H1 0.084 0.118 0.068\nL3H2 0.071 0.077 0.071\nL3H3 0.060 0.067 0.050\nL3H4 0.062 0.064 0.047\nL3H5 0.143 0.138 0.084\n\n L0-H1H5-QK-shared20 : QK query-energy/key-energy; bias-vector energy\nL0H0 0.752 0.277 0.311\nL0H1 0.946 0.953 0.766\nL0H2 0.690 0.627 0.703\nL0H3 0.625 0.510 0.301\nL0H4 0.094 0.190 0.261\nL0H5 0.900 0.916 0.790\nL1H0 0.059 0.129 0.249\nL1H1 0.056 0.222 0.330\nL1H2 0.056 0.200 0.308\nL1H3 0.057 0.219 0.331\nL1H4 0.070 0.217 0.348\nL1H5 0.055 0.191 0.295\nL2H0 0.050 0.063 0.214\nL2H1 0.047 0.050 0.039\nL2H2 0.048 0.065 0.310\nL2H3 0.052 0.073 0.313\nL2H4 0.045 0.075 0.263\nL2H5 0.051 0.058 0.089\nL3H0 0.048 0.054 0.065\nL3H1 0.047 0.053 0.089\nL3H2 0.048 0.056 0.162\nL3H3 0.049 0.055 0.064\nL3H4 0.050 0.055 0.061\nL3H5 0.050 0.051 0.063\n\nPairwise cosine of bias-derived key coefficients across all heads\nL0H0 [('L0H4', 0.213), ('L0H5', 0.199), ('L0H2', 0.196), ('L0H3', 0.066)]\nL0H1 [('L1H1', 0.387), ('L2H2', 0.354), ('L1H0', 0.353), ('L1H4', 0.347)]\nL0H2 [('L0H3', 0.353), ('L0H0', 0.196), ('L0H5', 0.178), ('L3H1', 0.104)]\nL0H3 [('L0H2', 0.353), ('L0H4', 0.088), ('L0H0', 0.066), ('L3H4', 0.056)]\nL0H4 [('L0H0', 0.213), ('L0H3', 0.088), ('L3H5', 0.05), ('L2H1', 0.021)]\nL0H5 [('L0H0', 0.199), ('L0H2', 0.178), ('L3H1', -0.013), ('L0H3', -0.017)]\nL1H0 [('L2H3', 0.826), ('L1H4', 0.81), ('L1H2', 0.808), ('L1H3', 0.801)]\nL1H1 [('L1H3', 0.924), ('L1H2', 0.923), ('L1H4', 0.921), ('L1H5', 0.897)]\nL1H2 [('L1H1', 0.923), ('L1H4', 0.9), ('L1H3', 0.899), ('L1H5', 0.868)]\nL1H3 [('L1H4', 0.926), ('L1H1', 0.924), ('L1H5', 0.899), ('L1H2', 0.899)]\nL1H4 [('L1H3', 0.926), ('L1H1', 0.921), ('L1H2', 0.9), ('L1H5', 0.883)]\nL1H5 [('L1H3', 0.899), ('L1H1', 0.897), ('L1H4', 0.883), ('L1H2', 0.868)]\nL2H0 [('L1H3', 0.67), ('L1H4', 0.641), ('L1H1', 0.623), ('L1H2', 0.619)]\nL2H1 [('L2H0', 0.298), ('L3H4', 0.276), ('L2H5', 0.272), ('L3H5', 0.206)]\nL2H2 [('L1H3', 0.885), ('L1H4', 0.88), ('L1H1', 0.867), ('L1H2', 0.864)]\nL2H3 [('L1H2', 0.864), ('L1H4', 0.833), ('L2H2', 0.831), ('L1H0', 0.826)]\nL2H4 [('L1H1', 0.816), ('L1H3', 0.812), ('L1H5', 0.8), ('L1H4', 0.781)]\nL2H5 [('L1H0', 0.527), ('L1H2', 0.519), ('L1H1', 0.495), ('L2H3', 0.495)]\nL3H0 [('L3H4', 0.364), ('L1H0', 0.303), ('L3H2', 0.297), ('L2H3', 0.274)]\nL3H1 [('L3H5', 0.296), ('L3H0', 0.211), ('L2H1', 0.193), ('L3H3', 0.166)]\nL3H2 [('L1H4', 0.461), ('L1H5', 0.456), ('L1H1', 0.455), ('L1H3', 0.454)]\nL3H3 [('L3H2', 0.253), ('L2H3', 0.244), ('L1H1', 0.229), ('L3H4', 0.227)]\nL3H4 [('L1H2', 0.435), ('L2H3', 0.423), ('L3H2', 0.399), ('L1H1', 0.396)]\nL3H5 [('L3H1', 0.296), ('L2H1', 0.206), ('L3H4', 0.075), ('L3H0', 0.06)]\n",
"seconds": 0.603,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "67eb2aaf58d9104f01404463bd00a57a1d524d596e36d6f3f82e02b52576a917",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}L0H4 effective OV singular values: [13.877 9.644 2.507 0.968 0.594 0.554 0.483 0.433 0.345 0.295 0.28 0.228]
Basis pair overlaps, normalized by smaller rank:
['uniform-head-write3', 'uniform-head-read3', 'H5-value-read27', 'H5-value-write27', 'L0-H1H5-QK-shared20']
[[1. 0.098 0.137 0.031 0.26 ]
[0.098 1. 0.075 0.06 0.848]
[0.137 0.075 1. 0.066 0.179]
[0.031 0.06 0.066 1. 0.205]
[0.26 0.848 0.179 0.205 1. ]]
uniform-head-write3 : QK query-energy/key-energy; bias-vector energy
L0H0 0.280 0.102 0.094
L0H1 0.079 0.054 0.126
L0H2 0.144 0.025 0.005
L0H3 0.319 0.049 0.061
L0H4 0.138 0.054 0.067
L0H5 0.031 0.039 0.119
L1H0 0.061 0.342 0.765
L1H1 0.107 0.627 0.905
L1H2 0.108 0.631 0.908
L1H3 0.137 0.678 0.931
L1H4 0.119 0.645 0.919
L1H5 0.084 0.525 0.825
L2H0 0.010 0.051 0.465
L2H1 0.005 0.005 0.040
L2H2 0.014 0.070 0.858
L2H3 0.022 0.095 0.804
L2H4 0.023 0.095 0.672
L2H5 0.006 0.041 0.270
L3H0 0.002 0.017 0.086
L3H1 0.003 0.009 0.078
L3H2 0.003 0.025 0.234
L3H3 0.005 0.010 0.052
L3H4 0.004 0.025 0.160
L3H5 0.003 0.005 0.074
uniform-head-read3 : QK query-energy/key-energy; bias-vector energy
L0H0 0.755 0.223 0.298
L0H1 0.567 0.608 0.643
L0H2 0.680 0.511 0.633
L0H3 0.623 0.414 0.302
L0H4 0.030 0.086 0.216
L0H5 0.249 0.281 0.148
L1H0 0.009 0.039 0.101
L1H1 0.015 0.090 0.137
L1H2 0.018 0.072 0.111
L1H3 0.012 0.082 0.128
L1H4 0.022 0.084 0.135
L1H5 0.008 0.072 0.105
L2H0 0.008 0.014 0.074
L2H1 0.010 0.009 0.010
L2H2 0.009 0.018 0.127
L2H3 0.010 0.018 0.126
L2H4 0.008 0.022 0.114
L2H5 0.009 0.012 0.028
L3H0 0.008 0.008 0.013
L3H1 0.008 0.010 0.017
L3H2 0.007 0.010 0.061
L3H3 0.006 0.009 0.010
L3H4 0.007 0.011 0.017
L3H5 0.009 0.008 0.002
H5-value-read27 : QK query-energy/key-energy; bias-vector energy
L0H0 0.059 0.391 0.423
L0H1 0.105 0.143 0.256
L0H2 0.131 0.293 0.199
L0H3 0.113 0.264 0.194
L0H4 0.085 0.101 0.101
L0H5 0.150 0.168 0.219
L1H0 0.074 0.175 0.264
L1H1 0.121 0.233 0.290
L1H2 0.084 0.199 0.273
L1H3 0.100 0.184 0.230
L1H4 0.070 0.200 0.253
L1H5 0.097 0.227 0.288
L2H0 0.076 0.084 0.141
L2H1 0.087 0.078 0.073
L2H2 0.102 0.114 0.194
L2H3 0.071 0.097 0.224
L2H4 0.073 0.091 0.176
L2H5 0.066 0.081 0.146
L3H0 0.085 0.075 0.132
L3H1 0.085 0.071 0.083
L3H2 0.075 0.078 0.100
L3H3 0.069 0.074 0.061
L3H4 0.070 0.076 0.109
L3H5 0.077 0.061 0.097
H5-value-write27 : QK query-energy/key-energy; bias-vector energy
L0H0 0.054 0.056 0.039
L0H1 0.108 0.094 0.069
L0H2 0.052 0.050 0.063
L0H3 0.051 0.072 0.101
L0H4 0.059 0.069 0.071
L0H5 0.172 0.163 0.081
L1H0 0.162 0.109 0.033
L1H1 0.122 0.060 0.022
L1H2 0.134 0.052 0.019
L1H3 0.138 0.056 0.014
L1H4 0.149 0.059 0.014
L1H5 0.146 0.079 0.038
L2H0 0.091 0.094 0.052
L2H1 0.120 0.135 0.113
L2H2 0.159 0.118 0.012
L2H3 0.077 0.089 0.031
L2H4 0.091 0.098 0.124
L2H5 0.072 0.078 0.095
L3H0 0.079 0.083 0.067
L3H1 0.084 0.118 0.068
L3H2 0.071 0.077 0.071
L3H3 0.060 0.067 0.050
L3H4 0.062 0.064 0.047
L3H5 0.143 0.138 0.084
L0-H1H5-QK-shared20 : QK query-energy/key-energy; bias-vector energy
L0H0 0.752 0.277 0.311
L0H1 0.946 0.953 0.766
L0H2 0.690 0.627 0.703
L0H3 0.625 0.510 0.301
L0H4 0.094 0.190 0.261
L0H5 0.900 0.916 0.790
L1H0 0.059 0.129 0.249
L1H1 0.056 0.222 0.330
L1H2 0.056 0.200 0.308
L1H3 0.057 0.219 0.331
L1H4 0.070 0.217 0.348
L1H5 0.055 0.191 0.295
L2H0 0.050 0.063 0.214
L2H1 0.047 0.050 0.039
L2H2 0.048 0.065 0.310
L2H3 0.052 0.073 0.313
L2H4 0.045 0.075 0.263
L2H5 0.051 0.058 0.089
L3H0 0.048 0.054 0.065
L3H1 0.047 0.053 0.089
L3H2 0.048 0.056 0.162
L3H3 0.049 0.055 0.064
L3H4 0.050 0.055 0.061
L3H5 0.050 0.051 0.063
Pairwise cosine of bias-derived key coefficients across all heads
L0H0 [('L0H4', 0.213), ('L0H5', 0.199), ('L0H2', 0.196), ('L0H3', 0.066)]
L0H1 [('L1H1', 0.387), ('L2H2', 0.354), ('L1H0', 0.353), ('L1H4', 0.347)]
L0H2 [('L0H3', 0.353), ('L0H0', 0.196), ('L0H5', 0.178), ('L3H1', 0.104)]
L0H3 [('L0H2', 0.353), ('L0H4', 0.088), ('L0H0', 0.066), ('L3H4', 0.056)]
L0H4 [('L0H0', 0.213), ('L0H3', 0.088), ('L3H5', 0.05), ('L2H1', 0.021)]
L0H5 [('L0H0', 0.199), ('L0H2', 0.178), ('L3H1', -0.013), ('L0H3', -0.017)]
L1H0 [('L2H3', 0.826), ('L1H4', 0.81), ('L1H2', 0.808), ('L1H3', 0.801)]
L1H1 [('L1H3', 0.924), ('L1H2', 0.923), ('L1H4', 0.921), ('L1H5', 0.897)]
L1H2 [('L1H1', 0.923), ('L1H4', 0.9), ('L1H3', 0.899), ('L1H5', 0.868)]
L1H3 [('L1H4', 0.926), ('L1H1', 0.924), ('L1H5', 0.899), ('L1H2', 0.899)]
L1H4 [('L1H3', 0.926), ('L1H1', 0.921), ('L1H2', 0.9), ('L1H5', 0.883)]
L1H5 [('L1H3', 0.899), ('L1H1', 0.897), ('L1H4', 0.883), ('L1H2', 0.868)]
L2H0 [('L1H3', 0.67), ('L1H4', 0.641), ('L1H1', 0.623), ('L1H2', 0.619)]
L2H1 [('L2H0', 0.298), ('L3H4', 0.276), ('L2H5', 0.272), ('L3H5', 0.206)]
L2H2 [('L1H3', 0.885), ('L1H4', 0.88), ('L1H1', 0.867), ('L1H2', 0.864)]
L2H3 [('L1H2', 0.864), ('L1H4', 0.833), ('L2H2', 0.831), ('L1H0', 0.826)]
L2H4 [('L1H1', 0.816), ('L1H3', 0.812), ('L1H5', 0.8), ('L1H4', 0.781)]
L2H5 [('L1H0', 0.527), ('L1H2', 0.519), ('L1H1', 0.495), ('L2H3', 0.495)]
L3H0 [('L3H4', 0.364), ('L1H0', 0.303), ('L3H2', 0.297), ('L2H3', 0.274)]
L3H1 [('L3H5', 0.296), ('L3H0', 0.211), ('L2H1', 0.193), ('L3H3', 0.166)]
L3H2 [('L1H4', 0.461), ('L1H5', 0.456), ('L1H1', 0.455), ('L1H3', 0.454)]
L3H3 [('L3H2', 0.253), ('L2H3', 0.244), ('L1H1', 0.229), ('L3H4', 0.227)]
L3H4 [('L1H2', 0.435), ('L2H3', 0.423), ('L3H2', 0.399), ('L1H1', 0.396)]
L3H5 [('L3H1', 0.296), ('L2H1', 0.206), ('L3H4', 0.075), ('L3H0', 0.06)]
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 7 of at most 100:
```
L0H4 effective OV singular values: [13.877 9.644 2.507 0.968 0.594 0.554 0.483 0.433 0.345 0.295 0.28 0.228]
Basis pair overlaps, normalized by smaller rank:
['uniform-head-write3', 'uniform-head-read3', 'H5-value-read27', 'H5-value-write27', 'L0-H1H5-QK-shared20']
[[1. 0.098 0.137 0.031 0.26 ]
[0.098 1. 0.075 0.06 0.848]
[0.137 0.075 1. 0.066 0.179]
[0.031 0.06 0.066 1. 0.205]
[0.26 0.848 0.179 0.205 1. ]]
uniform-head-write3 : QK query-energy/key-energy; bias-vector energy
L0H0 0.280 0.102 0.094
L0H1 0.079 0.054 0.126
L0H2 0.144 0.025 0.005
L0H3 0.319 0.049 0.061
L0H4 0.138 0.054 0.067
L0H5 0.031 0.039 0.119
L1H0 0.061 0.342 0.765
L1H1 0.107 0.627 0.905
L1H2 0.108 0.631 0.908
L1H3 0.137 0.678 0.931
L1H4 0.119 0.645 0.919
L1H5 0.084 0.525 0.825
L2H0 0.010 0.051 0.465
L2H1 0.005 0.005 0.040
L2H2 0.014 0.070 0.858
L2H3 0.022 0.095 0.804
L2H4 0.023 0.095 0.672
L2H5 0.006 0.041 0.270
L3H0 0.002 0.017 0.086
L3H1 0.003 0.009 0.078
L3H2 0.003 0.025 0.234
L3H3 0.005 0.010 0.052
L3H4 0.004 0.025 0.160
L3H5 0.003 0.005 0.074
uniform-head-read3 : QK query-energy/key-energy; bias-vector energy
L0H0 0.755 0.223 0.298
L0H1 0.567 0.608 0.643
L0H2 0.680 0.511 0.633
L0H3 0.623 0.414 0.302
L0H4 0.030 0.086 0.216
L0H5 0.249 0.281 0.148
L1H0 0.009 0.039 0.101
L1H1 0.015 0.090 0.137
L1H2 0.018 0.072 0.111
L1H3 0.012 0.082 0.128
L1H4 0.022 0.084 0.135
L1H5 0.008 0.072 0.105
L2H0 0.008 0.014 0.074
L2H1 0.010 0.009 0.010
L2H2 0.009 0.018 0.127
L2H3 0.010 0.018 0.126
L2H4 0.008 0.022 0.114
L2H5 0.009 0.012 0.028
L3H0 0.008 0.008 0.013
L3H1 0.008 0.010 0.017
L3H2 0.007 0.010 0.061
L3H3 0.006 0.009 0.010
L3H4 0.007 0.011 0.017
L3H5 0.009 0.008 0.002
H5-value-read27 : QK query-energy/key-energy; bias-vector energy
L0H0 0.059 0.391 0.423
L0H1 0.105 0.143 0.256
L0H2 0.131 0.293 0.199
L0H3 0.113 0.264 0.194
L0H4 0.085 0.101 0.101
L0H5 0.150 0.168 0.219
L1H0 0.074 0.175 0.264
L1H1 0.121 0.233 0.290
L1H2 0.084 0.199 0.273
L1H3 0.100 0.184 0.230
L1H4 0.070 0.200 0.253
L1H5 0.097 0.227 0.288
L2H0 0.076 0.084 0.141
L2H1 0.087 0.078 0.073
L2H2 0.102 0.114 0.194
L2H3 0.071 0.097 0.224
L2H4 0.073 0.091 0.176
L2H5 0.066 0.081 0.146
L3H0 0.085 0.075 0.132
L3H1 0.085 0.071 0.083
L3H2 0.075 0.078 0.100
L3H3 0.069 0.074 0.061
L3H4 0.070 0.076 0.109
L3H5 0.077 0.061 0.097
H5-value-write27 : QK query-energy/key-energy; bias-vector energy
L0H0 0.054 0.056 0.039
L0H1 0.108 0.094 0.069
L0H2 0.052 0.050 0.063
L0H3 0.051 0.072 0.101
L0H4 0.059 0.069 0.071
L0H5 0.172 0.163 0.081
L1H0 0.162 0.109 0.033
L1H1 0.122 0.060 0.022
L1H2 0.134 0.052 0.019
L1H3 0.138 0.056 0.014
L1H4 0.149 0.059 0.014
L1H5 0.146 0.079 0.038
L2H0 0.091 0.094 0.052
L2H1 0.120 0.135 0.113
L2H2 0.159 0.118 0.012
L2H3 0.077 0.089 0.031
L2H4 0.091 0.098 0.124
L2H5 0.072 0.078 0.095
L3H0 0.079 0.083 0.067
L3H1 0.084 0.118 0.068
L3H2 0.071 0.077 0.071
L3H3 0.060 0.067 0.050
L3H4 0.062 0.064 0.047
L3H5 0.143 0.138 0.084
L0-H1H5-QK-shared20 : QK query-energy/key-energy; bias-vector energy
L0H0 0.752 0.277 0.311
L0H1 0.946 0.953 0.766
L0H2 0.690 0.627 0.703
L0H3 0.625 0.510 0.301
L0H4 0.094 0.190 0.261
L0H5 0.900 0.916 0.790
L1H0 0.059 0.129 0.249
L1H1 0.056 0.222 0.330
L1H2 0.056 0.200 0.308
L1H3 0.057 0.219 0.331
L1H4 0.070 0.217 0.348
L1H5 0.055 0.191 0.295
L2H0 0.050 0.063 0.214
L2H1 0.047 0.050 0.039
L2H2 0.048 0.065 0.310
L2H3 0.052 0.073 0.313
L2H4 0.045 0.075 0.263
L2H5 0.051 0.058 0.089
L3H0 0.048 0.054 0.065
L3H1 0.047 0.053 0.089
L3H2 0.048 0.056 0.162
L3H3 0.049 0.055 0.064
L3H4 0.050 0.055 0.061
L3H5 0.050 0.051 0.063
Pairwise cosine of bias-derived key coefficients across all heads
L0H0 [('L0H4', 0.213), ('L0H5', 0.199), ('L0H2', 0.196), ('L0H3', 0.066)]
L0H1 [('L1H1', 0.387), ('L2H2', 0.354), ('L1H0', 0.353), ('L1H4', 0.347)]
L0H2 [('L0H3', 0.353), ('L0H0', 0.196), ('L0H5', 0.178), ('L3H1', 0.104)]
L0H3 [('L0H2', 0.353), ('L0H4', 0.088), ('L0H0', 0.066), ('L3H4', 0.056)]
L0H4 [('L0H0', 0.213), ('L0H3', 0.088), ('L3H5', 0.05), ('L2H1', 0.021)]
L0H5 [('L0H0', 0.199), ('L0H2', 0.178), ('L3H1', -0.013), ('L0H3', -0.017)]
L1H0 [('L2H3', 0.826), ('L1H4', 0.81), ('L1H2', 0.808), ('L1H3', 0.801)]
L1H1 [('L1H3', 0.924), ('L1H2', 0.923), ('L1H4', 0.921), ('L1H5', 0.897)]
L1H2 [('L1H1', 0.923), ('L1H4', 0.9), ('L1H3', 0.899), ('L1H5', 0.868)]
L1H3 [('L1H4', 0.926), ('L1H1', 0.924), ('L1H5', 0.899), ('L1H2', 0.899)]
L1H4 [('L1H3', 0.926), ('L1H1', 0.921), ('L1H2', 0.9), ('L1H5', 0.883)]
L1H5 [('L1H3', 0.899), ('L1H1', 0.897), ('L1H4', 0.883), ('L1H2', 0.868)]
L2H0 [('L1H3', 0.67), ('L1H4', 0.641), ('L1H1', 0.623), ('L1H2', 0.619)]
L2H1 [('L2H0', 0.298), ('L3H4', 0.276), ('L2H5', 0.272), ('L3H5', 0.206)]
L2H2 [('L1H3', 0.885), ('L1H4', 0.88), ('L1H1', 0.867), ('L1H2', 0.864)]
L2H3 [('L1H2', 0.864), ('L1H4', 0.833), ('L2H2', 0.831), ('L1H0', 0.826)]
L2H4 [('L1H1', 0.816), ('L1H3', 0.812), ('L1H5', 0.8), ('L1H4', 0.781)]
L2H5 [('L1H0', 0.527), ('L1H2', 0.519), ('L1H1', 0.495), ('L2H3', 0.495)]
L3H0 [('L3H4', 0.364), ('L1H0', 0.303), ('L3H2', 0.297), ('L2H3', 0.274)]
L3H1 [('L3H5', 0.296), ('L3H0', 0.211), ('L2H1', 0.193), ('L3H3', 0.166)]
L3H2 [('L1H4', 0.461), ('L1H5', 0.456), ('L1H1', 0.455), ('L1H3', 0.454)]
L3H3 [('L3H2', 0.253), ('L2H3', 0.244), ('L1H1', 0.229), ('L3H4', 0.227)]
L3H4 [('L1H2', 0.435), ('L2H3', 0.423), ('L3H2', 0.399), ('L1H1', 0.396)]
L3H5 [('L3H1', 0.296), ('L2H1', 0.206), ('L3H4', 0.075), ('L3H0', 0.06)]
```Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svd, eig
from threadpoolctl import threadpool_limits
threadpool_limits(limits=2)
np.set_printoptions(precision=3,suppress=True,linewidth=160)
# A congruence-invariant matrix-pencil check for shared positional rotations.
# No position vectors or attention patterns are constructed.
raw=[]
for j in [1,5]:
sl=slice(64*j,64*(j+1))
q=W["W_q.0"][sl].astype(float)
k=W["W_k.0"][sl].astype(float)
raw.append(q.T@k/8)
a1,a5=raw
u=svd(np.concatenate([a1,a1.T,a5,a5.T],axis=1),
full_matrices=False,check_finite=False)[0]
for r in [12,16,18,20,24,28,32]:
p=u[:,:r]
b1=p.T@a1@p
b5=p.T@a5@p
vals,vecs=eig(b5,b5.T,check_finite=False)
rows=[]
for n,lam in enumerate(vals):
if np.imag(lam)<=1e-5 or not np.isfinite(lam): continue
z=vecs[:,n]
c5=np.vdot(z,b5@z)
c1=np.vdot(z,b1@z)
rows.append((float(abs(lam)),float(np.angle(c5)),
float(np.angle(c1)),float(abs(c5)),float(abs(c1))))
rows=np.array(rows)
print("\nrank",r,"[pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1]")
print(rows)
if len(rows):
good=np.abs(np.log(rows[:,0]))<0.25
t=rows[good]
if len(t):
weights=np.sqrt(t[:,3]*t[:,4])
scores=[np.average(np.cos(t[:,2]-k*t[:,1]),weights=weights) for k in range(-4,9)]
print("Phase agreement for integer multiples -4..8:",np.round(scores,3))
```
{
"run": 8,
"turn": "t008",
"code": "import numpy as np\nfrom scipy.linalg import svd, eig\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(limits=2)\nnp.set_printoptions(precision=3,suppress=True,linewidth=160)\n\n# A congruence-invariant matrix-pencil check for shared positional rotations.\n# No position vectors or attention patterns are constructed.\nraw=[]\nfor j in [1,5]:\n sl=slice(64*j,64*(j+1))\n q=W[\"W_q.0\"][sl].astype(float)\n k=W[\"W_k.0\"][sl].astype(float)\n raw.append(q.T@k/8)\na1,a5=raw\nu=svd(np.concatenate([a1,a1.T,a5,a5.T],axis=1),\n full_matrices=False,check_finite=False)[0]\n\nfor r in [12,16,18,20,24,28,32]:\n p=u[:,:r]\n b1=p.T@a1@p\n b5=p.T@a5@p\n vals,vecs=eig(b5,b5.T,check_finite=False)\n rows=[]\n for n,lam in enumerate(vals):\n if np.imag(lam)<=1e-5 or not np.isfinite(lam): continue\n z=vecs[:,n]\n c5=np.vdot(z,b5@z)\n c1=np.vdot(z,b1@z)\n rows.append((float(abs(lam)),float(np.angle(c5)),\n float(np.angle(c1)),float(abs(c5)),float(abs(c1))))\n rows=np.array(rows)\n print(\"\\nrank\",r,\"[pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1]\")\n print(rows)\n if len(rows):\n good=np.abs(np.log(rows[:,0]))<0.25\n t=rows[good]\n if len(t):\n weights=np.sqrt(t[:,3]*t[:,4])\n scores=[np.average(np.cos(t[:,2]-k*t[:,1]),weights=weights) for k in range(-4,9)]\n print(\"Phase agreement for integer multiples -4..8:\",np.round(scores,3))",
"out": "\nrank 12 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1]\n[[ 1. 0.683 1.724 1.402 0.231]\n [ 1. 0.907 2.104 1.433 0.293]\n [ 1. -1.794 2.319 0.698 0.735]\n [ 1.039 -1.996 2.237 0. 0.72 ]\n [ 0.962 1.095 2.213 0. 0.712]]\nPhase agreement for integer multiples -4..8: [ 0.264 -0.569 -0.423 -0.23 -0.467 0.062 0.941 0.609 -0.197 -0.182 -0.218 -0.694 -0.312]\n\nrank 16 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1]\n[[ 1. 0.541 0.949 1.052 0.264]\n [ 1. 0.734 1.488 0.832 0.162]\n [ 1. 0.893 1.997 1.525 0.295]\n [ 1. -2.018 2.204 0.817 1.024]\n [ 1. -1.939 2.296 1.003 1.291]\n [ 1. 1.393 3.041 1.241 0.222]]\nPhase agreement for integer multiples -4..8: [ 0.272 -0.536 -0.219 0.239 -0.425 0.014 0.989 0.188 -0.418 0.152 -0.238 -0.606 0.217]\n\nrank 18 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1]\n[[ 1. 0.554 1.464 1.339 0.261]\n [ 1. 0.86 1.246 1.002 0.495]\n [ 1. 0.914 1.378 1.069 0.41 ]\n [ 1. 0.953 1.282 0.897 0.501]\n [ 1. -1.607 2.876 0.794 0.197]\n [ 1. 1.397 3.056 1.147 0.225]\n [ 1. -1.942 2.29 1.038 1.328]\n [ 1. -2.024 2.197 0.828 1.037]]\nPhase agreement for integer multiples -4..8: [ 0.157 -0.621 -0.35 0.096 -0.284 0.2 0.94 0.084 -0.589 -0.061 -0.089 -0.214 0.388]\n\nrank 20 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1]\n[[ 1. 0.512 1.565 0.94 0.208]\n [ 1. 0.771 2.084 0.724 0.252]\n [ 1. 0.909 1.081 0.644 0.703]\n [ 1. 0.969 1.892 1.046 0.284]\n [ 1. -2.026 2.202 0.793 1.033]\n [ 1. -1.882 2.395 0.186 0.609]\n [ 1. 1.308 2.51 0.15 0.367]]\nPhase agreement for integer multiples -4..8: [ 0.358 -0.514 -0.538 -0.118 -0.29 0.236 0.915 0.195 -0.349 -0.012 -0.238 -0.273 0.395]\n\nrank 24 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1]\n[[ 1. 0.481 1.274 0.594 0.171]\n [ 1. 0.64 1.575 0.523 0.182]\n [ 1.381 -0.313 1.784 0. 0.36 ]\n [ 1. 0.876 1.683 1.069 0.282]\n [ 0.724 -2.024 1.914 0. 0.241]\n [ 1. -1.604 2.926 0.583 0.213]\n [ 1. 1.117 2.184 0.205 0.14 ]\n [ 1. -1.971 2.31 0.617 0.791]\n [ 1. -1.941 2.076 0.343 0.442]\n [ 1.039 -1.82 2.408 0. 0.282]\n [ 0.963 1.275 2.411 0. 0.308]]\nPhase agreement for integer multiples -4..8: [ 0.159 -0.619 -0.31 0.094 -0.384 0.094 0.98 0.274 -0.37 0.13 -0.084 -0.54 0.067]\n\nrank 28 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1]\n[[ 2.458 -2.355 2.039 0. 0.113]\n [ 1.356 0.415 2.071 0. 0.317]\n [ 1. 0.631 1.403 0.243 0.196]\n [ 1. 0.79 1.637 0.473 0.12 ]\n [ 0.407 0.413 1.374 0. 0.034]\n [ 0.737 0.477 1.8 0. 0.198]\n [ 1. -1.958 2.309 0.583 0.796]\n [ 1. -1.725 2.658 0.117 0.159]\n [ 1. 1.248 2.651 0.191 0.161]\n [ 1. 1.35 2.744 0.172 0.199]\n [ 1. -1.846 2.325 0.349 0.538]]\nPhase agreement for integer multiples -4..8: [ 0.207 -0.579 -0.036 0.287 -0.571 -0.089 0.989 0.134 -0.601 0.193 0.098 -0.562 0.053]\n\nrank 32 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1]\n[[ 4.56 -0.212 2.765 0. 0.041]\n [ 1.681 -2.728 1.503 0. 0.09 ]\n [ 1. 0.522 0.686 0.117 0.059]\n [ 1. 0.785 1.528 0.111 0.124]\n [ 0.595 0.627 1.853 0. 0.056]\n [ 1.073 -2.248 2.153 0. 0.268]\n [ 0.932 0.942 1.767 0. 0.151]\n [ 1. -1.951 2.288 0.435 0.61 ]\n [ 1.031 -1.881 2.377 0. 0.278]\n [ 0.97 1.31 2.461 0. 0.247]\n [ 1.083 1.544 -3.133 0. 0.126]\n [ 0.923 -1.853 3.026 0. 0.127]\n [ 0.219 0.229 2.938 0. 0.021]]\nPhase agreement for integer multiples -4..8: [ 0.402 -0.851 -0.213 0.61 -0.377 -0.094 0.989 -0.018 -0.554 0.462 -0.148 -0.926 0.311]\n",
"seconds": 0.671,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "552c45e571b6026e7663b7955757d7c961e45cc4b84e902f1d80946a54371a2d",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}rank 12 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.683 1.724 1.402 0.231] [ 1. 0.907 2.104 1.433 0.293] [ 1. -1.794 2.319 0.698 0.735] [ 1.039 -1.996 2.237 0. 0.72 ] [ 0.962 1.095 2.213 0. 0.712]] Phase agreement for integer multiples -4..8: [ 0.264 -0.569 -0.423 -0.23 -0.467 0.062 0.941 0.609 -0.197 -0.182 -0.218 -0.694 -0.312] rank 16 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.541 0.949 1.052 0.264] [ 1. 0.734 1.488 0.832 0.162] [ 1. 0.893 1.997 1.525 0.295] [ 1. -2.018 2.204 0.817 1.024] [ 1. -1.939 2.296 1.003 1.291] [ 1. 1.393 3.041 1.241 0.222]] Phase agreement for integer multiples -4..8: [ 0.272 -0.536 -0.219 0.239 -0.425 0.014 0.989 0.188 -0.418 0.152 -0.238 -0.606 0.217] rank 18 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.554 1.464 1.339 0.261] [ 1. 0.86 1.246 1.002 0.495] [ 1. 0.914 1.378 1.069 0.41 ] [ 1. 0.953 1.282 0.897 0.501] [ 1. -1.607 2.876 0.794 0.197] [ 1. 1.397 3.056 1.147 0.225] [ 1. -1.942 2.29 1.038 1.328] [ 1. -2.024 2.197 0.828 1.037]] Phase agreement for integer multiples -4..8: [ 0.157 -0.621 -0.35 0.096 -0.284 0.2 0.94 0.084 -0.589 -0.061 -0.089 -0.214 0.388] rank 20 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.512 1.565 0.94 0.208] [ 1. 0.771 2.084 0.724 0.252] [ 1. 0.909 1.081 0.644 0.703] [ 1. 0.969 1.892 1.046 0.284] [ 1. -2.026 2.202 0.793 1.033] [ 1. -1.882 2.395 0.186 0.609] [ 1. 1.308 2.51 0.15 0.367]] Phase agreement for integer multiples -4..8: [ 0.358 -0.514 -0.538 -0.118 -0.29 0.236 0.915 0.195 -0.349 -0.012 -0.238 -0.273 0.395] rank 24 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.481 1.274 0.594 0.171] [ 1. 0.64 1.575 0.523 0.182] [ 1.381 -0.313 1.784 0. 0.36 ] [ 1. 0.876 1.683 1.069 0.282] [ 0.724 -2.024 1.914 0. 0.241] [ 1. -1.604 2.926 0.583 0.213] [ 1. 1.117 2.184 0.205 0.14 ] [ 1. -1.971 2.31 0.617 0.791] [ 1. -1.941 2.076 0.343 0.442] [ 1.039 -1.82 2.408 0. 0.282] [ 0.963 1.275 2.411 0. 0.308]] Phase agreement for integer multiples -4..8: [ 0.159 -0.619 -0.31 0.094 -0.384 0.094 0.98 0.274 -0.37 0.13 -0.084 -0.54 0.067] rank 28 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 2.458 -2.355 2.039 0. 0.113] [ 1.356 0.415 2.071 0. 0.317] [ 1. 0.631 1.403 0.243 0.196] [ 1. 0.79 1.637 0.473 0.12 ] [ 0.407 0.413 1.374 0. 0.034] [ 0.737 0.477 1.8 0. 0.198] [ 1. -1.958 2.309 0.583 0.796] [ 1. -1.725 2.658 0.117 0.159] [ 1. 1.248 2.651 0.191 0.161] [ 1. 1.35 2.744 0.172 0.199] [ 1. -1.846 2.325 0.349 0.538]] Phase agreement for integer multiples -4..8: [ 0.207 -0.579 -0.036 0.287 -0.571 -0.089 0.989 0.134 -0.601 0.193 0.098 -0.562 0.053] rank 32 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 4.56 -0.212 2.765 0. 0.041] [ 1.681 -2.728 1.503 0. 0.09 ] [ 1. 0.522 0.686 0.117 0.059] [ 1. 0.785 1.528 0.111 0.124] [ 0.595 0.627 1.853 0. 0.056] [ 1.073 -2.248 2.153 0. 0.268] [ 0.932 0.942 1.767 0. 0.151] [ 1. -1.951 2.288 0.435 0.61 ] [ 1.031 -1.881 2.377 0. 0.278] [ 0.97 1.31 2.461 0. 0.247] [ 1.083 1.544 -3.133 0. 0.126] [ 0.923 -1.853 3.026 0. 0.127] [ 0.219 0.229 2.938 0. 0.021]] Phase agreement for integer multiples -4..8: [ 0.402 -0.851 -0.213 0.61 -0.377 -0.094 0.989 -0.018 -0.554 0.462 -0.148 -0.926 0.311]
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 8 of at most 100: ``` rank 12 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.683 1.724 1.402 0.231] [ 1. 0.907 2.104 1.433 0.293] [ 1. -1.794 2.319 0.698 0.735] [ 1.039 -1.996 2.237 0. 0.72 ] [ 0.962 1.095 2.213 0. 0.712]] Phase agreement for integer multiples -4..8: [ 0.264 -0.569 -0.423 -0.23 -0.467 0.062 0.941 0.609 -0.197 -0.182 -0.218 -0.694 -0.312] rank 16 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.541 0.949 1.052 0.264] [ 1. 0.734 1.488 0.832 0.162] [ 1. 0.893 1.997 1.525 0.295] [ 1. -2.018 2.204 0.817 1.024] [ 1. -1.939 2.296 1.003 1.291] [ 1. 1.393 3.041 1.241 0.222]] Phase agreement for integer multiples -4..8: [ 0.272 -0.536 -0.219 0.239 -0.425 0.014 0.989 0.188 -0.418 0.152 -0.238 -0.606 0.217] rank 18 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.554 1.464 1.339 0.261] [ 1. 0.86 1.246 1.002 0.495] [ 1. 0.914 1.378 1.069 0.41 ] [ 1. 0.953 1.282 0.897 0.501] [ 1. -1.607 2.876 0.794 0.197] [ 1. 1.397 3.056 1.147 0.225] [ 1. -1.942 2.29 1.038 1.328] [ 1. -2.024 2.197 0.828 1.037]] Phase agreement for integer multiples -4..8: [ 0.157 -0.621 -0.35 0.096 -0.284 0.2 0.94 0.084 -0.589 -0.061 -0.089 -0.214 0.388] rank 20 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.512 1.565 0.94 0.208] [ 1. 0.771 2.084 0.724 0.252] [ 1. 0.909 1.081 0.644 0.703] [ 1. 0.969 1.892 1.046 0.284] [ 1. -2.026 2.202 0.793 1.033] [ 1. -1.882 2.395 0.186 0.609] [ 1. 1.308 2.51 0.15 0.367]] Phase agreement for integer multiples -4..8: [ 0.358 -0.514 -0.538 -0.118 -0.29 0.236 0.915 0.195 -0.349 -0.012 -0.238 -0.273 0.395] rank 24 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 1. 0.481 1.274 0.594 0.171] [ 1. 0.64 1.575 0.523 0.182] [ 1.381 -0.313 1.784 0. 0.36 ] [ 1. 0.876 1.683 1.069 0.282] [ 0.724 -2.024 1.914 0. 0.241] [ 1. -1.604 2.926 0.583 0.213] [ 1. 1.117 2.184 0.205 0.14 ] [ 1. -1.971 2.31 0.617 0.791] [ 1. -1.941 2.076 0.343 0.442] [ 1.039 -1.82 2.408 0. 0.282] [ 0.963 1.275 2.411 0. 0.308]] Phase agreement for integer multiples -4..8: [ 0.159 -0.619 -0.31 0.094 -0.384 0.094 0.98 0.274 -0.37 0.13 -0.084 -0.54 0.067] rank 28 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 2.458 -2.355 2.039 0. 0.113] [ 1.356 0.415 2.071 0. 0.317] [ 1. 0.631 1.403 0.243 0.196] [ 1. 0.79 1.637 0.473 0.12 ] [ 0.407 0.413 1.374 0. 0.034] [ 0.737 0.477 1.8 0. 0.198] [ 1. -1.958 2.309 0.583 0.796] [ 1. -1.725 2.658 0.117 0.159] [ 1. 1.248 2.651 0.191 0.161] [ 1. 1.35 2.744 0.172 0.199] [ 1. -1.846 2.325 0.349 0.538]] Phase agreement for integer multiples -4..8: [ 0.207 -0.579 -0.036 0.287 -0.571 -0.089 0.989 0.134 -0.601 0.193 0.098 -0.562 0.053] rank 32 [pencil modulus, phase H5, phase H1, magnitude H5, magnitude H1] [[ 4.56 -0.212 2.765 0. 0.041] [ 1.681 -2.728 1.503 0. 0.09 ] [ 1. 0.522 0.686 0.117 0.059] [ 1. 0.785 1.528 0.111 0.124] [ 0.595 0.627 1.853 0. 0.056] [ 1.073 -2.248 2.153 0. 0.268] [ 0.932 0.942 1.767 0. 0.151] [ 1. -1.951 2.288 0.435 0.61 ] [ 1.031 -1.881 2.377 0. 0.278] [ 0.97 1.31 2.461 0. 0.247] [ 1.083 1.544 -3.133 0. 0.126] [ 0.923 -1.853 3.026 0. 0.127] [ 0.219 0.229 2.938 0. 0.021]] Phase agreement for integer multiples -4..8: [ 0.402 -0.851 -0.213 0.61 -0.377 -0.094 0.989 -0.018 -0.554 0.462 -0.148 -0.926 0.311] ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svd, eig
from threadpoolctl import threadpool_limits
threadpool_limits(limits=2)
np.set_printoptions(precision=3,suppress=True,linewidth=160)
A=[]
for j in range(6):
sl=slice(64*j,64*(j+1))
A.append(W["W_q.0"][sl].astype(float).T @ W["W_k.0"][sl].astype(float)/8)
u=svd(np.concatenate([A[1],A[1].T,A[5],A[5].T],axis=1),
full_matrices=False,check_finite=False)[0]
for r in [16,24]:
p=u[:,:r]
mats=[p.T@a@p for a in A]
vals,vecs=eig(mats[5],mats[5].T,check_finite=False)
zs=[]
for n,lam in enumerate(vals):
if not np.isfinite(lam) or np.imag(lam)<=1e-5 or abs(np.log(abs(lam)))>.1:
continue
z=vecs[:,n]
c=np.vdot(z,mats[5]@z)
if abs(c)<.02: continue
if np.angle(c)<0: z=z.conj()
zs.append(z)
zs.sort(key=lambda z: np.angle(np.vdot(z,mats[5]@z)))
coeff=np.array([[np.vdot(z,m@z) for z in zs]for m in mats])
theta=np.angle(coeff[5])
print("\nrank",r,"reference H5 mode phases:",theta)
for j in range(6):
mag=np.abs(coeff[j])
phase=np.angle(coeff[j])
print("H%d magnitudes"%j,mag)
print("H%d phases"%j,phase)
ks=np.arange(0,13)
scores=np.array([np.average(np.cos(phase-k*theta),weights=mag+1e-20)for k in ks])
print("phase agreement for multipliers 0..12:",np.round(scores,3))
# Signed agreement of the strongest induction candidates, across several
# independently defined first-layer read subspaces and ranks.
def products(l,j):
sl=slice(j*64,(j+1)*64)
g=W[f"ln1_w.{l}"].astype(float)
q=W[f"W_q.{l}"][sl].astype(float)*g
k=W[f"W_k.{l}"][sl].astype(float)*g
v=W[f"W_v.{l}"][sl].astype(float)*g
q-=q.mean(axis=1,keepdims=True)
k-=k.mean(axis=1,keepdims=True)
v-=v.mean(axis=1,keepdims=True)
o=W[f"W_o.{l}"][:,sl].astype(float)
o-=o.mean(axis=0,keepdims=True)
return q.T@k/8,o@v
pairs=[products(l,j)for l in range(4)for j in range(6)]
for source in [1,5]:
_,_,vt=svd(pairs[source][1],full_matrices=False,check_finite=False)
for rank in [16,24,32]:
t=vt[:rank].T
print("\nRead basis from L0H%d OV, rank%d"%(source,rank))
for target,earlier in [(13,5),(19,5),(23,5),(19,14),(23,14),(12,5),(18,5)]:
c=t.T@pairs[target][0]@pairs[earlier][1]@t
print("L%dH%d <- L%dH%d: trace/norm %.3f"%(target//6,target%6,
earlier//6,earlier%6,np.trace(c)/(np.linalg.norm(c)+1e-30)))
```
{
"run": 9,
"turn": "t009",
"code": "import numpy as np\nfrom scipy.linalg import svd, eig\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(limits=2)\nnp.set_printoptions(precision=3,suppress=True,linewidth=160)\n\nA=[]\nfor j in range(6):\n sl=slice(64*j,64*(j+1))\n A.append(W[\"W_q.0\"][sl].astype(float).T @ W[\"W_k.0\"][sl].astype(float)/8)\nu=svd(np.concatenate([A[1],A[1].T,A[5],A[5].T],axis=1),\n full_matrices=False,check_finite=False)[0]\n\nfor r in [16,24]:\n p=u[:,:r]\n mats=[p.T@a@p for a in A]\n vals,vecs=eig(mats[5],mats[5].T,check_finite=False)\n zs=[]\n for n,lam in enumerate(vals):\n if not np.isfinite(lam) or np.imag(lam)<=1e-5 or abs(np.log(abs(lam)))>.1:\n continue\n z=vecs[:,n]\n c=np.vdot(z,mats[5]@z)\n if abs(c)<.02: continue\n if np.angle(c)<0: z=z.conj()\n zs.append(z)\n zs.sort(key=lambda z: np.angle(np.vdot(z,mats[5]@z)))\n coeff=np.array([[np.vdot(z,m@z) for z in zs]for m in mats])\n theta=np.angle(coeff[5])\n print(\"\\nrank\",r,\"reference H5 mode phases:\",theta)\n for j in range(6):\n mag=np.abs(coeff[j])\n phase=np.angle(coeff[j])\n print(\"H%d magnitudes\"%j,mag)\n print(\"H%d phases\"%j,phase)\n ks=np.arange(0,13)\n scores=np.array([np.average(np.cos(phase-k*theta),weights=mag+1e-20)for k in ks])\n print(\"phase agreement for multipliers 0..12:\",np.round(scores,3))\n\n# Signed agreement of the strongest induction candidates, across several\n# independently defined first-layer read subspaces and ranks.\ndef products(l,j):\n sl=slice(j*64,(j+1)*64)\n g=W[f\"ln1_w.{l}\"].astype(float)\n q=W[f\"W_q.{l}\"][sl].astype(float)*g\n k=W[f\"W_k.{l}\"][sl].astype(float)*g\n v=W[f\"W_v.{l}\"][sl].astype(float)*g\n q-=q.mean(axis=1,keepdims=True)\n k-=k.mean(axis=1,keepdims=True)\n v-=v.mean(axis=1,keepdims=True)\n o=W[f\"W_o.{l}\"][:,sl].astype(float)\n o-=o.mean(axis=0,keepdims=True)\n return q.T@k/8,o@v\npairs=[products(l,j)for l in range(4)for j in range(6)]\nfor source in [1,5]:\n _,_,vt=svd(pairs[source][1],full_matrices=False,check_finite=False)\n for rank in [16,24,32]:\n t=vt[:rank].T\n print(\"\\nRead basis from L0H%d OV, rank%d\"%(source,rank))\n for target,earlier in [(13,5),(19,5),(23,5),(19,14),(23,14),(12,5),(18,5)]:\n c=t.T@pairs[target][0]@pairs[earlier][1]@t\n print(\"L%dH%d <- L%dH%d: trace/norm %.3f\"%(target//6,target%6,\n earlier//6,earlier%6,np.trace(c)/(np.linalg.norm(c)+1e-30)))",
"out": "\nrank 16 reference H5 mode phases: [0.541 0.734 0.893 1.393 1.939 2.018]\nH0 magnitudes [0.004 0.003 0.003 0. 0.001 0.002]\nH0 phases [ 2.822 2.255 -2.344 -1.159 -2.389 1.843]\nphase agreement for multipliers 0..12: [-0.676 -0.299 -0.012 0.336 0.727 0.531 0.08 -0.117 -0.333 -0.405 -0.109 0.07 0.068]\nH1 magnitudes [0.264 0.162 0.295 0.222 1.291 1.024]\nH1 phases [ 0.949 1.488 1.997 3.041 -2.296 -2.204]\nphase agreement for multipliers 0..12: [-0.503 -0.184 0.992 -0.025 -0.55 0.473 -0.124 -0.746 0.402 0.246 -0.58 0.397 0.495]\nH2 magnitudes [0.022 0.011 0.002 0.006 0.001 0.009]\nH2 phases [-0.627 -0.642 -2.212 -1.884 1.901 1.804]\nphase agreement for multipliers 0..12: [ 0.412 0.249 -0.319 -0.455 -0.322 -0.628 -0.449 0.284 0.348 0.201 0.441 0.308 0.046]\nH3 magnitudes [0.043 0.025 0.009 0.004 0.001 0.005]\nH3 phases [-2.978 -2.808 -2.553 0.747 -1.477 -3.015]\nphase agreement for multipliers 0..12: [-0.859 -0.776 -0.432 -0.01 0.553 0.864 0.632 0.258 -0.038 -0.317 -0.402 -0.334 -0.302]\nH4 magnitudes [0. 0. 0. 0. 0. 0.001]\nH4 phases [ 2.147 -3.091 -2.876 2.944 -1.82 2.934]\nphase agreement for multipliers 0..12: [-0.878 -0.178 0.308 0.025 0.469 0.563 -0.223 -0.239 -0.127 -0.568 -0.091 0.549 0.074]\nH5 magnitudes [1.052 0.832 1.525 1.241 1.003 0.817]\nH5 phases [0.541 0.734 0.893 1.393 1.939 2.018]\nphase agreement for multipliers 0..12: [ 0.306 1. 0.306 -0.335 -0.131 -0.297 -0.416 0.054 0.062 -0.028 0.42 0.162 -0.516]\n\nrank 24 reference H5 mode phases: [0.481 0.64 0.876 1.117 1.604 1.941 1.971]\nH0 magnitudes [0.055 0.073 0.005 0.114 0.036 0.037 0.03 ]\nH0 phases [-3.102 3.072 2.453 -3.125 2.909 2.973 3.099]\nphase agreement for multipliers 0..12: [-0.992 -0.33 0.286 0.175 0.24 0.281 -0.061 0.05 0.411 0.126 -0.297 -0.407 -0.466]\nH1 magnitudes [0.171 0.182 0.282 0.14 0.213 0.442 0.791]\nH1 phases [ 1.274 1.575 1.683 2.184 -2.926 -2.076 -2.31 ]\nphase agreement for multipliers 0..12: [-0.458 -0.073 0.981 0.126 -0.499 0.317 -0.007 -0.629 0.212 0.296 -0.465 0.053 0.312]\nH2 magnitudes [0.023 0.026 0.008 0.036 0.015 0.006 0.012]\nH2 phases [-2.533 -2.074 -0.251 2.963 2.681 2.934 2.616]\nphase agreement for multipliers 0..12: [-0.707 -0.263 -0.035 -0.258 -0.136 0.065 0.085 0.399 0.654 0.321 -0.154 -0.484 -0.574]\nH3 magnitudes [0.07 0.108 0.023 0.125 0.036 0.034 0.033]\nH3 phases [ 2.882 2.829 3.056 -2.931 2.753 2.811 2.88 ]\nphase agreement for multipliers 0..12: [-0.963 -0.344 0.241 0.32 0.482 0.383 -0.086 -0.071 0.201 0.009 -0.251 -0.352 -0.431]\nH4 magnitudes [0.001 0.002 0.001 0.001 0.001 0.001 0. ]\nH4 phases [-2.942 -3.033 2.752 3.069 -2.804 3.055 3.025]\nphase agreement for multipliers 0..12: [-0.983 -0.415 0.162 0.109 0.27 0.372 0.036 0.103 0.356 0. -0.283 -0.203 -0.372]\nH5 magnitudes [0.594 0.523 1.069 0.205 0.583 0.343 0.617]\nH5 phases [0.481 0.64 0.876 1.117 1.604 1.941 1.971]\nphase agreement for multipliers 0..12: [ 0.34 1. 0.34 -0.278 -0.075 -0.28 -0.544 -0.037 0.224 -0.004 0.052 -0.022 -0.203]\n\nRead basis from L0H1 OV, rank16\nL2H1 <- L0H5: trace/norm 2.580\nL3H1 <- L0H5: trace/norm 2.362\nL3H5 <- L0H5: trace/norm 2.590\nL3H1 <- L2H2: trace/norm 2.391\nL3H5 <- L2H2: trace/norm 2.819\nL2H0 <- L0H5: trace/norm -0.266\nL3H0 <- L0H5: trace/norm 1.241\n\nRead basis from L0H1 OV, rank24\nL2H1 <- L0H5: trace/norm 2.768\nL3H1 <- L0H5: trace/norm 2.646\nL3H5 <- L0H5: trace/norm 2.807\nL3H1 <- L2H2: trace/norm 2.417\nL3H5 <- L2H2: trace/norm 2.847\nL2H0 <- L0H5: trace/norm -0.351\nL3H0 <- L0H5: trace/norm 1.245\n\nRead basis from L0H1 OV, rank32\nL2H1 <- L0H5: trace/norm 2.628\nL3H1 <- L0H5: trace/norm 2.506\nL3H5 <- L0H5: trace/norm 2.658\nL3H1 <- L2H2: trace/norm 2.247\nL3H5 <- L2H2: trace/norm 2.826\nL2H0 <- L0H5: trace/norm -0.277\nL3H0 <- L0H5: trace/norm 1.143\n\nRead basis from L0H5 OV, rank16\nL2H1 <- L0H5: trace/norm 2.898\nL3H1 <- L0H5: trace/norm 2.703\nL3H5 <- L0H5: trace/norm 2.837\nL3H1 <- L2H2: trace/norm 2.258\nL3H5 <- L2H2: trace/norm 2.738\nL2H0 <- L0H5: trace/norm -0.463\nL3H0 <- L0H5: trace/norm 1.376\n\nRead basis from L0H5 OV, rank24\nL2H1 <- L0H5: trace/norm 2.819\nL3H1 <- L0H5: trace/norm 2.663\nL3H5 <- L0H5: trace/norm 2.766\nL3H1 <- L2H2: trace/norm 2.234\nL3H5 <- L2H2: trace/norm 2.772\nL2H0 <- L0H5: trace/norm -0.328\nL3H0 <- L0H5: trace/norm 1.261\n\nRead basis from L0H5 OV, rank32\nL2H1 <- L0H5: trace/norm 2.671\nL3H1 <- L0H5: trace/norm 2.420\nL3H5 <- L0H5: trace/norm 2.539\nL3H1 <- L2H2: trace/norm 2.109\nL3H5 <- L2H2: trace/norm 2.632\nL2H0 <- L0H5: trace/norm -0.261\nL3H0 <- L0H5: trace/norm 1.107\n",
"seconds": 0.764,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "531abcc2b78ba877658667833787f9f812e8acb186a4c458b61b48ce0565a26f",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}rank 16 reference H5 mode phases: [0.541 0.734 0.893 1.393 1.939 2.018] H0 magnitudes [0.004 0.003 0.003 0. 0.001 0.002] H0 phases [ 2.822 2.255 -2.344 -1.159 -2.389 1.843] phase agreement for multipliers 0..12: [-0.676 -0.299 -0.012 0.336 0.727 0.531 0.08 -0.117 -0.333 -0.405 -0.109 0.07 0.068] H1 magnitudes [0.264 0.162 0.295 0.222 1.291 1.024] H1 phases [ 0.949 1.488 1.997 3.041 -2.296 -2.204] phase agreement for multipliers 0..12: [-0.503 -0.184 0.992 -0.025 -0.55 0.473 -0.124 -0.746 0.402 0.246 -0.58 0.397 0.495] H2 magnitudes [0.022 0.011 0.002 0.006 0.001 0.009] H2 phases [-0.627 -0.642 -2.212 -1.884 1.901 1.804] phase agreement for multipliers 0..12: [ 0.412 0.249 -0.319 -0.455 -0.322 -0.628 -0.449 0.284 0.348 0.201 0.441 0.308 0.046] H3 magnitudes [0.043 0.025 0.009 0.004 0.001 0.005] H3 phases [-2.978 -2.808 -2.553 0.747 -1.477 -3.015] phase agreement for multipliers 0..12: [-0.859 -0.776 -0.432 -0.01 0.553 0.864 0.632 0.258 -0.038 -0.317 -0.402 -0.334 -0.302] H4 magnitudes [0. 0. 0. 0. 0. 0.001] H4 phases [ 2.147 -3.091 -2.876 2.944 -1.82 2.934] phase agreement for multipliers 0..12: [-0.878 -0.178 0.308 0.025 0.469 0.563 -0.223 -0.239 -0.127 -0.568 -0.091 0.549 0.074] H5 magnitudes [1.052 0.832 1.525 1.241 1.003 0.817] H5 phases [0.541 0.734 0.893 1.393 1.939 2.018] phase agreement for multipliers 0..12: [ 0.306 1. 0.306 -0.335 -0.131 -0.297 -0.416 0.054 0.062 -0.028 0.42 0.162 -0.516] rank 24 reference H5 mode phases: [0.481 0.64 0.876 1.117 1.604 1.941 1.971] H0 magnitudes [0.055 0.073 0.005 0.114 0.036 0.037 0.03 ] H0 phases [-3.102 3.072 2.453 -3.125 2.909 2.973 3.099] phase agreement for multipliers 0..12: [-0.992 -0.33 0.286 0.175 0.24 0.281 -0.061 0.05 0.411 0.126 -0.297 -0.407 -0.466] H1 magnitudes [0.171 0.182 0.282 0.14 0.213 0.442 0.791] H1 phases [ 1.274 1.575 1.683 2.184 -2.926 -2.076 -2.31 ] phase agreement for multipliers 0..12: [-0.458 -0.073 0.981 0.126 -0.499 0.317 -0.007 -0.629 0.212 0.296 -0.465 0.053 0.312] H2 magnitudes [0.023 0.026 0.008 0.036 0.015 0.006 0.012] H2 phases [-2.533 -2.074 -0.251 2.963 2.681 2.934 2.616] phase agreement for multipliers 0..12: [-0.707 -0.263 -0.035 -0.258 -0.136 0.065 0.085 0.399 0.654 0.321 -0.154 -0.484 -0.574] H3 magnitudes [0.07 0.108 0.023 0.125 0.036 0.034 0.033] H3 phases [ 2.882 2.829 3.056 -2.931 2.753 2.811 2.88 ] phase agreement for multipliers 0..12: [-0.963 -0.344 0.241 0.32 0.482 0.383 -0.086 -0.071 0.201 0.009 -0.251 -0.352 -0.431] H4 magnitudes [0.001 0.002 0.001 0.001 0.001 0.001 0. ] H4 phases [-2.942 -3.033 2.752 3.069 -2.804 3.055 3.025] phase agreement for multipliers 0..12: [-0.983 -0.415 0.162 0.109 0.27 0.372 0.036 0.103 0.356 0. -0.283 -0.203 -0.372] H5 magnitudes [0.594 0.523 1.069 0.205 0.583 0.343 0.617] H5 phases [0.481 0.64 0.876 1.117 1.604 1.941 1.971] phase agreement for multipliers 0..12: [ 0.34 1. 0.34 -0.278 -0.075 -0.28 -0.544 -0.037 0.224 -0.004 0.052 -0.022 -0.203] Read basis from L0H1 OV, rank16 L2H1 <- L0H5: trace/norm 2.580 L3H1 <- L0H5: trace/norm 2.362 L3H5 <- L0H5: trace/norm 2.590 L3H1 <- L2H2: trace/norm 2.391 L3H5 <- L2H2: trace/norm 2.819 L2H0 <- L0H5: trace/norm -0.266 L3H0 <- L0H5: trace/norm 1.241 Read basis from L0H1 OV, rank24 L2H1 <- L0H5: trace/norm 2.768 L3H1 <- L0H5: trace/norm 2.646 L3H5 <- L0H5: trace/norm 2.807 L3H1 <- L2H2: trace/norm 2.417 L3H5 <- L2H2: trace/norm 2.847 L2H0 <- L0H5: trace/norm -0.351 L3H0 <- L0H5: trace/norm 1.245 Read basis from L0H1 OV, rank32 L2H1 <- L0H5: trace/norm 2.628 L3H1 <- L0H5: trace/norm 2.506 L3H5 <- L0H5: trace/norm 2.658 L3H1 <- L2H2: trace/norm 2.247 L3H5 <- L2H2: trace/norm 2.826 L2H0 <- L0H5: trace/norm -0.277 L3H0 <- L0H5: trace/norm 1.143 Read basis from L0H5 OV, rank16 L2H1 <- L0H5: trace/norm 2.898 L3H1 <- L0H5: trace/norm 2.703 L3H5 <- L0H5: trace/norm 2.837 L3H1 <- L2H2: trace/norm 2.258 L3H5 <- L2H2: trace/norm 2.738 L2H0 <- L0H5: trace/norm -0.463 L3H0 <- L0H5: trace/norm 1.376 Read basis from L0H5 OV, rank24 L2H1 <- L0H5: trace/norm 2.819 L3H1 <- L0H5: trace/norm 2.663 L3H5 <- L0H5: trace/norm 2.766 L3H1 <- L2H2: trace/norm 2.234 L3H5 <- L2H2: trace/norm 2.772 L2H0 <- L0H5: trace/norm -0.328 L3H0 <- L0H5: trace/norm 1.261 Read basis from L0H5 OV, rank32 L2H1 <- L0H5: trace/norm 2.671 L3H1 <- L0H5: trace/norm 2.420 L3H5 <- L0H5: trace/norm 2.539 L3H1 <- L2H2: trace/norm 2.109 L3H5 <- L2H2: trace/norm 2.632 L2H0 <- L0H5: trace/norm -0.261 L3H0 <- L0H5: trace/norm 1.107
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 9 of at most 100: ``` rank 16 reference H5 mode phases: [0.541 0.734 0.893 1.393 1.939 2.018] H0 magnitudes [0.004 0.003 0.003 0. 0.001 0.002] H0 phases [ 2.822 2.255 -2.344 -1.159 -2.389 1.843] phase agreement for multipliers 0..12: [-0.676 -0.299 -0.012 0.336 0.727 0.531 0.08 -0.117 -0.333 -0.405 -0.109 0.07 0.068] H1 magnitudes [0.264 0.162 0.295 0.222 1.291 1.024] H1 phases [ 0.949 1.488 1.997 3.041 -2.296 -2.204] phase agreement for multipliers 0..12: [-0.503 -0.184 0.992 -0.025 -0.55 0.473 -0.124 -0.746 0.402 0.246 -0.58 0.397 0.495] H2 magnitudes [0.022 0.011 0.002 0.006 0.001 0.009] H2 phases [-0.627 -0.642 -2.212 -1.884 1.901 1.804] phase agreement for multipliers 0..12: [ 0.412 0.249 -0.319 -0.455 -0.322 -0.628 -0.449 0.284 0.348 0.201 0.441 0.308 0.046] H3 magnitudes [0.043 0.025 0.009 0.004 0.001 0.005] H3 phases [-2.978 -2.808 -2.553 0.747 -1.477 -3.015] phase agreement for multipliers 0..12: [-0.859 -0.776 -0.432 -0.01 0.553 0.864 0.632 0.258 -0.038 -0.317 -0.402 -0.334 -0.302] H4 magnitudes [0. 0. 0. 0. 0. 0.001] H4 phases [ 2.147 -3.091 -2.876 2.944 -1.82 2.934] phase agreement for multipliers 0..12: [-0.878 -0.178 0.308 0.025 0.469 0.563 -0.223 -0.239 -0.127 -0.568 -0.091 0.549 0.074] H5 magnitudes [1.052 0.832 1.525 1.241 1.003 0.817] H5 phases [0.541 0.734 0.893 1.393 1.939 2.018] phase agreement for multipliers 0..12: [ 0.306 1. 0.306 -0.335 -0.131 -0.297 -0.416 0.054 0.062 -0.028 0.42 0.162 -0.516] rank 24 reference H5 mode phases: [0.481 0.64 0.876 1.117 1.604 1.941 1.971] H0 magnitudes [0.055 0.073 0.005 0.114 0.036 0.037 0.03 ] H0 phases [-3.102 3.072 2.453 -3.125 2.909 2.973 3.099] phase agreement for multipliers 0..12: [-0.992 -0.33 0.286 0.175 0.24 0.281 -0.061 0.05 0.411 0.126 -0.297 -0.407 -0.466] H1 magnitudes [0.171 0.182 0.282 0.14 0.213 0.442 0.791] H1 phases [ 1.274 1.575 1.683 2.184 -2.926 -2.076 -2.31 ] phase agreement for multipliers 0..12: [-0.458 -0.073 0.981 0.126 -0.499 0.317 -0.007 -0.629 0.212 0.296 -0.465 0.053 0.312] H2 magnitudes [0.023 0.026 0.008 0.036 0.015 0.006 0.012] H2 phases [-2.533 -2.074 -0.251 2.963 2.681 2.934 2.616] phase agreement for multipliers 0..12: [-0.707 -0.263 -0.035 -0.258 -0.136 0.065 0.085 0.399 0.654 0.321 -0.154 -0.484 -0.574] H3 magnitudes [0.07 0.108 0.023 0.125 0.036 0.034 0.033] H3 phases [ 2.882 2.829 3.056 -2.931 2.753 2.811 2.88 ] phase agreement for multipliers 0..12: [-0.963 -0.344 0.241 0.32 0.482 0.383 -0.086 -0.071 0.201 0.009 -0.251 -0.352 -0.431] H4 magnitudes [0.001 0.002 0.001 0.001 0.001 0.001 0. ] H4 phases [-2.942 -3.033 2.752 3.069 -2.804 3.055 3.025] phase agreement for multipliers 0..12: [-0.983 -0.415 0.162 0.109 0.27 0.372 0.036 0.103 0.356 0. -0.283 -0.203 -0.372] H5 magnitudes [0.594 0.523 1.069 0.205 0.583 0.343 0.617] H5 phases [0.481 0.64 0.876 1.117 1.604 1.941 1.971] phase agreement for multipliers 0..12: [ 0.34 1. 0.34 -0.278 -0.075 -0.28 -0.544 -0.037 0.224 -0.004 0.052 -0.022 -0.203] Read basis from L0H1 OV, rank16 L2H1 <- L0H5: trace/norm 2.580 L3H1 <- L0H5: trace/norm 2.362 L3H5 <- L0H5: trace/norm 2.590 L3H1 <- L2H2: trace/norm 2.391 L3H5 <- L2H2: trace/norm 2.819 L2H0 <- L0H5: trace/norm -0.266 L3H0 <- L0H5: trace/norm 1.241 Read basis from L0H1 OV, rank24 L2H1 <- L0H5: trace/norm 2.768 L3H1 <- L0H5: trace/norm 2.646 L3H5 <- L0H5: trace/norm 2.807 L3H1 <- L2H2: trace/norm 2.417 L3H5 <- L2H2: trace/norm 2.847 L2H0 <- L0H5: trace/norm -0.351 L3H0 <- L0H5: trace/norm 1.245 Read basis from L0H1 OV, rank32 L2H1 <- L0H5: trace/norm 2.628 L3H1 <- L0H5: trace/norm 2.506 L3H5 <- L0H5: trace/norm 2.658 L3H1 <- L2H2: trace/norm 2.247 L3H5 <- L2H2: trace/norm 2.826 L2H0 <- L0H5: trace/norm -0.277 L3H0 <- L0H5: trace/norm 1.143 Read basis from L0H5 OV, rank16 L2H1 <- L0H5: trace/norm 2.898 L3H1 <- L0H5: trace/norm 2.703 L3H5 <- L0H5: trace/norm 2.837 L3H1 <- L2H2: trace/norm 2.258 L3H5 <- L2H2: trace/norm 2.738 L2H0 <- L0H5: trace/norm -0.463 L3H0 <- L0H5: trace/norm 1.376 Read basis from L0H5 OV, rank24 L2H1 <- L0H5: trace/norm 2.819 L3H1 <- L0H5: trace/norm 2.663 L3H5 <- L0H5: trace/norm 2.766 L3H1 <- L2H2: trace/norm 2.234 L3H5 <- L2H2: trace/norm 2.772 L2H0 <- L0H5: trace/norm -0.328 L3H0 <- L0H5: trace/norm 1.261 Read basis from L0H5 OV, rank32 L2H1 <- L0H5: trace/norm 2.671 L3H1 <- L0H5: trace/norm 2.420 L3H5 <- L0H5: trace/norm 2.539 L3H1 <- L2H2: trace/norm 2.109 L3H5 <- L2H2: trace/norm 2.632 L2H0 <- L0H5: trace/norm -0.261 L3H0 <- L0H5: trace/norm 1.107 ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from scipy.linalg import svd
from sklearn.cluster import KMeans
from threadpoolctl import threadpool_limits
threadpool_limits(limits=2)
np.set_printoptions(precision=3,suppress=True,linewidth=160)
A=[]; B=[]
for j in range(6):
sl=slice(64*j,64*(j+1))
g=W["ln1_w.0"].astype(float)
q=W["W_q.0"][sl].astype(float)*g
k=W["W_k.0"][sl].astype(float)*g
v=W["W_v.0"][sl].astype(float)*g
for x in [q,k,v]: x-=x.mean(axis=1,keepdims=True)
o=W["W_o.0"][:,sl].astype(float)
o-=o.mean(axis=0,keepdims=True)
A.append(q.T@k/8); B.append(o@v)
P=svd(np.concatenate([A[1],A[1].T,A[5],A[5].T],axis=1),
full_matrices=False,check_finite=False)[0][:,:20]
clean=B[5]-(B[5]@P)@P.T
_,s,vt=svd(clean,full_matrices=False,check_finite=False)
T=vt[:27].T
print("Position-orthogonalized H5 OV singular values:",s[:40])
f=W["W_fc.0"].astype(float)*W["ln2_w.0"][None,:]
f-=f.mean(axis=1,keepdims=True)
active=np.linalg.norm(W["W_fc.0"],axis=1)>1
original_indices=np.where(active)[0]
f=f[active]
cov=[f@T,f@B[5]@T,f@B[1]@T]
norms=[np.linalg.norm(z,axis=1) for z in cov]
units=[z/n[:,None]for z,n in zip(cov,norms)]
# Direct-read clusters provide an unlabeled dictionary of weight directions.
km=KMeans(n_clusters=28,n_init=20,max_iter=300,random_state=42).fit(units[0])
cent=km.cluster_centers_
cent/=np.linalg.norm(cent,axis=1,keepdims=True)
scores=[z@cent.T for z in units]
idx=[z.argmax(axis=1)for z in scores]
best=[z.max(axis=1)for z in scores]
print("Best cluster cosine quantiles, direct / H5-transported / H1-transported")
for z in best: print(np.quantile(z,[.1,.25,.5,.75,.9]))
print("Cluster: direct size; repeated pair count; repeated triple count; "
"mean direct/H5/H1 covector norm for direct members")
for c in range(28):
mask=idx[0]==c
pair=mask&(idx[1]==c)&(best[0]>.55)&(best[1]>.55)
triple=pair&(idx[2]==c)&(best[2]>.55)
print(c,int(mask.sum()),int(pair.sum()),int(triple.sum()),
[round(float(n[mask].mean()),3)for n in norms])
samecos=np.stack([np.sum(units[0]*units[1],axis=1),
np.sum(units[0]*units[2],axis=1),
np.sum(units[1]*units[2],axis=1)],axis=1)
triple_score=np.min(samecos,axis=1)
top=np.argsort(-triple_score)[:20]
print("\nMost mutually aligned direct/H5/H1 MLP covectors:")
for i in top:
print("neuron",original_indices[i],"cluster",idx[0][i],
"pairwise cosines",samecos[i],
"norms",[round(float(n[i]),3)for n in norms],
"center cosines",[round(float(z[i]),3)for z in best])
# Connections of content-sensitive L0 key-read singular covectors to these
# learned MLP-row clusters. Both sides of every comparison are weights.
for j in [0,2,3]:
ak=A[j]@T
u,s,vh=svd(ak,full_matrices=False,check_finite=False)
print("\nL0H%d token-restricted key singular values"%j,s[:5])
for r in range(2):
align=cent@vh[r]
order=np.argsort(-np.abs(align))[:6]
print("key direction",r,[(int(c),round(float(align[c]),3))for c in order])
```
{
"run": 10,
"turn": "t010",
"code": "import numpy as np\nfrom scipy.linalg import svd\nfrom sklearn.cluster import KMeans\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(limits=2)\nnp.set_printoptions(precision=3,suppress=True,linewidth=160)\n\nA=[]; B=[]\nfor j in range(6):\n sl=slice(64*j,64*(j+1))\n g=W[\"ln1_w.0\"].astype(float)\n q=W[\"W_q.0\"][sl].astype(float)*g\n k=W[\"W_k.0\"][sl].astype(float)*g\n v=W[\"W_v.0\"][sl].astype(float)*g\n for x in [q,k,v]: x-=x.mean(axis=1,keepdims=True)\n o=W[\"W_o.0\"][:,sl].astype(float)\n o-=o.mean(axis=0,keepdims=True)\n A.append(q.T@k/8); B.append(o@v)\n\nP=svd(np.concatenate([A[1],A[1].T,A[5],A[5].T],axis=1),\n full_matrices=False,check_finite=False)[0][:,:20]\nclean=B[5]-(B[5]@P)@P.T\n_,s,vt=svd(clean,full_matrices=False,check_finite=False)\nT=vt[:27].T\nprint(\"Position-orthogonalized H5 OV singular values:\",s[:40])\n\nf=W[\"W_fc.0\"].astype(float)*W[\"ln2_w.0\"][None,:]\nf-=f.mean(axis=1,keepdims=True)\nactive=np.linalg.norm(W[\"W_fc.0\"],axis=1)>1\noriginal_indices=np.where(active)[0]\nf=f[active]\ncov=[f@T,f@B[5]@T,f@B[1]@T]\nnorms=[np.linalg.norm(z,axis=1) for z in cov]\nunits=[z/n[:,None]for z,n in zip(cov,norms)]\n\n# Direct-read clusters provide an unlabeled dictionary of weight directions.\nkm=KMeans(n_clusters=28,n_init=20,max_iter=300,random_state=42).fit(units[0])\ncent=km.cluster_centers_\ncent/=np.linalg.norm(cent,axis=1,keepdims=True)\nscores=[z@cent.T for z in units]\nidx=[z.argmax(axis=1)for z in scores]\nbest=[z.max(axis=1)for z in scores]\nprint(\"Best cluster cosine quantiles, direct / H5-transported / H1-transported\")\nfor z in best: print(np.quantile(z,[.1,.25,.5,.75,.9]))\n\nprint(\"Cluster: direct size; repeated pair count; repeated triple count; \"\n \"mean direct/H5/H1 covector norm for direct members\")\nfor c in range(28):\n mask=idx[0]==c\n pair=mask&(idx[1]==c)&(best[0]>.55)&(best[1]>.55)\n triple=pair&(idx[2]==c)&(best[2]>.55)\n print(c,int(mask.sum()),int(pair.sum()),int(triple.sum()),\n [round(float(n[mask].mean()),3)for n in norms])\n\nsamecos=np.stack([np.sum(units[0]*units[1],axis=1),\n np.sum(units[0]*units[2],axis=1),\n np.sum(units[1]*units[2],axis=1)],axis=1)\ntriple_score=np.min(samecos,axis=1)\ntop=np.argsort(-triple_score)[:20]\nprint(\"\\nMost mutually aligned direct/H5/H1 MLP covectors:\")\nfor i in top:\n print(\"neuron\",original_indices[i],\"cluster\",idx[0][i],\n \"pairwise cosines\",samecos[i],\n \"norms\",[round(float(n[i]),3)for n in norms],\n \"center cosines\",[round(float(z[i]),3)for z in best])\n\n# Connections of content-sensitive L0 key-read singular covectors to these\n# learned MLP-row clusters. Both sides of every comparison are weights.\nfor j in [0,2,3]:\n ak=A[j]@T\n u,s,vh=svd(ak,full_matrices=False,check_finite=False)\n print(\"\\nL0H%d token-restricted key singular values\"%j,s[:5])\n for r in range(2):\n align=cent@vh[r]\n order=np.argsort(-np.abs(align))[:6]\n print(\"key direction\",r,[(int(c),round(float(align[c]),3))for c in order])",
"out": "Position-orthogonalized H5 OV singular values: [1.049 1.031 0.948 0.936 0.899 0.879 0.841 0.829 0.808 0.751 0.743 0.716 0.698 0.657 0.645 0.613 0.591 0.537 0.504 0.44 0.385 0.3 0.272 0.263 0.238 0.209\n 0.186 0.178 0.163 0.157 0.142 0.132 0.128 0.118 0.111 0.103 0.101 0.098 0.085 0.083]\nBest cluster cosine quantiles, direct / H5-transported / H1-transported\n[0.526 0.622 0.736 0.855 0.934]\n[0.283 0.375 0.463 0.562 0.675]\n[0.302 0.37 0.464 0.564 0.672]\nCluster: direct size; repeated pair count; repeated triple count; mean direct/H5/H1 covector norm for direct members\n0 7 0 0 [0.933, 0.615, 0.368]\n1 65 10 1 [0.956, 0.545, 0.417]\n2 25 0 0 [0.855, 0.436, 0.33]\n3 21 0 0 [0.952, 0.51, 0.394]\n4 14 0 0 [0.56, 0.401, 0.342]\n5 18 0 0 [0.807, 0.41, 0.314]\n6 11 1 0 [0.663, 0.441, 0.302]\n7 20 0 0 [1.007, 0.573, 0.358]\n8 14 3 2 [0.817, 0.399, 0.303]\n9 9 0 0 [0.843, 0.364, 0.293]\n10 25 2 0 [1.037, 0.402, 0.318]\n11 10 0 0 [1.042, 0.433, 0.275]\n12 16 1 1 [0.857, 0.45, 0.317]\n13 12 1 1 [0.987, 0.491, 0.331]\n14 23 2 1 [0.963, 0.502, 0.416]\n15 26 1 0 [0.924, 0.48, 0.307]\n16 23 1 1 [0.916, 0.526, 0.351]\n17 14 0 0 [0.833, 0.473, 0.391]\n18 24 1 1 [0.958, 0.563, 0.393]\n19 12 0 0 [0.926, 0.495, 0.399]\n20 11 1 0 [1.044, 0.475, 0.313]\n21 23 1 0 [0.964, 0.535, 0.41]\n22 14 0 0 [0.864, 0.47, 0.333]\n23 21 3 3 [0.85, 0.468, 0.354]\n24 13 0 0 [0.808, 0.38, 0.322]\n25 14 0 0 [0.553, 0.472, 0.371]\n26 17 1 1 [0.851, 0.46, 0.331]\n27 16 0 0 [0.879, 0.527, 0.39]\n\nMost mutually aligned direct/H5/H1 MLP covectors:\nneuron 14 cluster 3 pairwise cosines [0.748 0.655 0.76 ] norms [0.444, 0.405, 0.248] center cosines [0.619, 0.37, 0.265]\nneuron 251 cluster 1 pairwise cosines [0.668 0.645 0.656] norms [0.48, 0.391, 0.221] center cosines [0.73, 0.59, 0.591]\nneuron 1206 cluster 21 pairwise cosines [0.6 0.768 0.592] norms [0.577, 0.444, 0.326] center cosines [0.787, 0.642, 0.561]\nneuron 129 cluster 7 pairwise cosines [0.586 0.654 0.619] norms [0.604, 0.497, 0.363] center cosines [0.604, 0.435, 0.637]\nneuron 753 cluster 6 pairwise cosines [0.71 0.586 0.655] norms [0.418, 0.42, 0.311] center cosines [0.675, 0.575, 0.418]\nneuron 1265 cluster 3 pairwise cosines [0.693 0.679 0.582] norms [0.599, 0.441, 0.363] center cosines [0.572, 0.397, 0.532]\nneuron 795 cluster 27 pairwise cosines [0.665 0.581 0.66 ] norms [0.549, 0.5, 0.371] center cosines [0.654, 0.478, 0.533]\nneuron 85 cluster 23 pairwise cosines [0.596 0.638 0.573] norms [0.315, 0.354, 0.199] center cosines [0.402, 0.231, 0.29]\nneuron 1065 cluster 1 pairwise cosines [0.706 0.57 0.641] norms [0.519, 0.365, 0.32] center cosines [0.501, 0.383, 0.345]\nneuron 26 cluster 14 pairwise cosines [0.532 0.532 0.656] norms [0.424, 0.406, 0.378] center cosines [0.477, 0.231, 0.445]\nneuron 310 cluster 1 pairwise cosines [0.518 0.552 0.69 ] norms [0.542, 0.588, 0.251] center cosines [0.649, 0.485, 0.513]\nneuron 570 cluster 12 pairwise cosines [0.68 0.511 0.506] norms [0.495, 0.475, 0.21] center cosines [0.639, 0.369, 0.394]\nneuron 581 cluster 1 pairwise cosines [0.735 0.505 0.782] norms [0.499, 0.557, 0.324] center cosines [0.397, 0.243, 0.377]\nneuron 311 cluster 26 pairwise cosines [0.531 0.691 0.501] norms [0.548, 0.447, 0.394] center cosines [0.696, 0.642, 0.629]\nneuron 325 cluster 10 pairwise cosines [0.499 0.607 0.61 ] norms [0.64, 0.324, 0.316] center cosines [0.71, 0.664, 0.83]\nneuron 1240 cluster 3 pairwise cosines [0.536 0.489 0.733] norms [0.395, 0.449, 0.306] center cosines [0.603, 0.404, 0.489]\nneuron 1006 cluster 5 pairwise cosines [0.47 0.543 0.679] norms [0.378, 0.264, 0.16] center cosines [0.712, 0.552, 0.437]\nneuron 799 cluster 27 pairwise cosines [0.563 0.467 0.669] norms [0.538, 0.539, 0.402] center cosines [0.561, 0.451, 0.298]\nneuron 1354 cluster 23 pairwise cosines [0.458 0.472 0.538] norms [0.591, 0.313, 0.287] center cosines [0.798, 0.673, 0.741]\nneuron 642 cluster 1 pairwise cosines [0.448 0.703 0.633] norms [0.731, 0.49, 0.29] center cosines [0.641, 0.516, 0.724]\n\nL0H0 token-restricted key singular values [0.059 0.031 0.006 0.002 0.002]\nkey direction 0 [(1, 0.558), (6, 0.41), (0, 0.379), (22, 0.338), (10, 0.331), (3, 0.283)]\nkey direction 1 [(4, -0.469), (6, -0.39), (15, 0.327), (24, 0.321), (9, -0.31), (25, -0.288)]\n\nL0H2 token-restricted key singular values [0.058 0.025 0.021 0.019 0.016]\nkey direction 0 [(10, 0.546), (26, 0.376), (23, -0.3), (11, -0.299), (22, 0.259), (24, 0.251)]\nkey direction 1 [(0, 0.49), (3, 0.399), (14, 0.363), (17, 0.313), (2, -0.301), (23, 0.266)]\n\nL0H3 token-restricted key singular values [0.083 0.051 0.027 0.017 0.013]\nkey direction 0 [(15, 0.607), (1, -0.487), (25, 0.37), (17, 0.284), (20, 0.276), (18, -0.273)]\nkey direction 1 [(15, -0.415), (0, -0.41), (18, -0.352), (3, -0.29), (22, -0.29), (14, -0.288)]\n",
"seconds": 1.453,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "9327b43ddc6d887274c161f49ced2eadb975f257025853045fbb0de4026336e0",
"weights_sha256": "77ed8f7d418619c9390da0abea19de2a05d6e2200d8484473deafee15a6fd43f"
}Position-orthogonalized H5 OV singular values: [1.049 1.031 0.948 0.936 0.899 0.879 0.841 0.829 0.808 0.751 0.743 0.716 0.698 0.657 0.645 0.613 0.591 0.537 0.504 0.44 0.385 0.3 0.272 0.263 0.238 0.209 0.186 0.178 0.163 0.157 0.142 0.132 0.128 0.118 0.111 0.103 0.101 0.098 0.085 0.083] Best cluster cosine quantiles, direct / H5-transported / H1-transported [0.526 0.622 0.736 0.855 0.934] [0.283 0.375 0.463 0.562 0.675] [0.302 0.37 0.464 0.564 0.672] Cluster: direct size; repeated pair count; repeated triple count; mean direct/H5/H1 covector norm for direct members 0 7 0 0 [0.933, 0.615, 0.368] 1 65 10 1 [0.956, 0.545, 0.417] 2 25 0 0 [0.855, 0.436, 0.33] 3 21 0 0 [0.952, 0.51, 0.394] 4 14 0 0 [0.56, 0.401, 0.342] 5 18 0 0 [0.807, 0.41, 0.314] 6 11 1 0 [0.663, 0.441, 0.302] 7 20 0 0 [1.007, 0.573, 0.358] 8 14 3 2 [0.817, 0.399, 0.303] 9 9 0 0 [0.843, 0.364, 0.293] 10 25 2 0 [1.037, 0.402, 0.318] 11 10 0 0 [1.042, 0.433, 0.275] 12 16 1 1 [0.857, 0.45, 0.317] 13 12 1 1 [0.987, 0.491, 0.331] 14 23 2 1 [0.963, 0.502, 0.416] 15 26 1 0 [0.924, 0.48, 0.307] 16 23 1 1 [0.916, 0.526, 0.351] 17 14 0 0 [0.833, 0.473, 0.391] 18 24 1 1 [0.958, 0.563, 0.393] 19 12 0 0 [0.926, 0.495, 0.399] 20 11 1 0 [1.044, 0.475, 0.313] 21 23 1 0 [0.964, 0.535, 0.41] 22 14 0 0 [0.864, 0.47, 0.333] 23 21 3 3 [0.85, 0.468, 0.354] 24 13 0 0 [0.808, 0.38, 0.322] 25 14 0 0 [0.553, 0.472, 0.371] 26 17 1 1 [0.851, 0.46, 0.331] 27 16 0 0 [0.879, 0.527, 0.39] Most mutually aligned direct/H5/H1 MLP covectors: neuron 14 cluster 3 pairwise cosines [0.748 0.655 0.76 ] norms [0.444, 0.405, 0.248] center cosines [0.619, 0.37, 0.265] neuron 251 cluster 1 pairwise cosines [0.668 0.645 0.656] norms [0.48, 0.391, 0.221] center cosines [0.73, 0.59, 0.591] neuron 1206 cluster 21 pairwise cosines [0.6 0.768 0.592] norms [0.577, 0.444, 0.326] center cosines [0.787, 0.642, 0.561] neuron 129 cluster 7 pairwise cosines [0.586 0.654 0.619] norms [0.604, 0.497, 0.363] center cosines [0.604, 0.435, 0.637] neuron 753 cluster 6 pairwise cosines [0.71 0.586 0.655] norms [0.418, 0.42, 0.311] center cosines [0.675, 0.575, 0.418] neuron 1265 cluster 3 pairwise cosines [0.693 0.679 0.582] norms [0.599, 0.441, 0.363] center cosines [0.572, 0.397, 0.532] neuron 795 cluster 27 pairwise cosines [0.665 0.581 0.66 ] norms [0.549, 0.5, 0.371] center cosines [0.654, 0.478, 0.533] neuron 85 cluster 23 pairwise cosines [0.596 0.638 0.573] norms [0.315, 0.354, 0.199] center cosines [0.402, 0.231, 0.29] neuron 1065 cluster 1 pairwise cosines [0.706 0.57 0.641] norms [0.519, 0.365, 0.32] center cosines [0.501, 0.383, 0.345] neuron 26 cluster 14 pairwise cosines [0.532 0.532 0.656] norms [0.424, 0.406, 0.378] center cosines [0.477, 0.231, 0.445] neuron 310 cluster 1 pairwise cosines [0.518 0.552 0.69 ] norms [0.542, 0.588, 0.251] center cosines [0.649, 0.485, 0.513] neuron 570 cluster 12 pairwise cosines [0.68 0.511 0.506] norms [0.495, 0.475, 0.21] center cosines [0.639, 0.369, 0.394] neuron 581 cluster 1 pairwise cosines [0.735 0.505 0.782] norms [0.499, 0.557, 0.324] center cosines [0.397, 0.243, 0.377] neuron 311 cluster 26 pairwise cosines [0.531 0.691 0.501] norms [0.548, 0.447, 0.394] center cosines [0.696, 0.642, 0.629] neuron 325 cluster 10 pairwise cosines [0.499 0.607 0.61 ] norms [0.64, 0.324, 0.316] center cosines [0.71, 0.664, 0.83] neuron 1240 cluster 3 pairwise cosines [0.536 0.489 0.733] norms [0.395, 0.449, 0.306] center cosines [0.603, 0.404, 0.489] neuron 1006 cluster 5 pairwise cosines [0.47 0.543 0.679] norms [0.378, 0.264, 0.16] center cosines [0.712, 0.552, 0.437] neuron 799 cluster 27 pairwise cosines [0.563 0.467 0.669] norms [0.538, 0.539, 0.402] center cosines [0.561, 0.451, 0.298] neuron 1354 cluster 23 pairwise cosines [0.458 0.472 0.538] norms [0.591, 0.313, 0.287] center cosines [0.798, 0.673, 0.741] neuron 642 cluster 1 pairwise cosines [0.448 0.703 0.633] norms [0.731, 0.49, 0.29] center cosines [0.641, 0.516, 0.724] L0H0 token-restricted key singular values [0.059 0.031 0.006 0.002 0.002] key direction 0 [(1, 0.558), (6, 0.41), (0, 0.379), (22, 0.338), (10, 0.331), (3, 0.283)] key direction 1 [(4, -0.469), (6, -0.39), (15, 0.327), (24, 0.321), (9, -0.31), (25, -0.288)] L0H2 token-restricted key singular values [0.058 0.025 0.021 0.019 0.016] key direction 0 [(10, 0.546), (26, 0.376), (23, -0.3), (11, -0.299), (22, 0.259), (24, 0.251)] key direction 1 [(0, 0.49), (3, 0.399), (14, 0.363), (17, 0.313), (2, -0.301), (23, 0.266)] L0H3 token-restricted key singular values [0.083 0.051 0.027 0.017 0.013] key direction 0 [(15, 0.607), (1, -0.487), (25, 0.37), (17, 0.284), (20, 0.276), (18, -0.273)] key direction 1 [(15, -0.415), (0, -0.41), (18, -0.352), (3, -0.29), (22, -0.29), (14, -0.288)]
Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 10 of at most 100: ``` Position-orthogonalized H5 OV singular values: [1.049 1.031 0.948 0.936 0.899 0.879 0.841 0.829 0.808 0.751 0.743 0.716 0.698 0.657 0.645 0.613 0.591 0.537 0.504 0.44 0.385 0.3 0.272 0.263 0.238 0.209 0.186 0.178 0.163 0.157 0.142 0.132 0.128 0.118 0.111 0.103 0.101 0.098 0.085 0.083] Best cluster cosine quantiles, direct / H5-transported / H1-transported [0.526 0.622 0.736 0.855 0.934] [0.283 0.375 0.463 0.562 0.675] [0.302 0.37 0.464 0.564 0.672] Cluster: direct size; repeated pair count; repeated triple count; mean direct/H5/H1 covector norm for direct members 0 7 0 0 [0.933, 0.615, 0.368] 1 65 10 1 [0.956, 0.545, 0.417] 2 25 0 0 [0.855, 0.436, 0.33] 3 21 0 0 [0.952, 0.51, 0.394] 4 14 0 0 [0.56, 0.401, 0.342] 5 18 0 0 [0.807, 0.41, 0.314] 6 11 1 0 [0.663, 0.441, 0.302] 7 20 0 0 [1.007, 0.573, 0.358] 8 14 3 2 [0.817, 0.399, 0.303] 9 9 0 0 [0.843, 0.364, 0.293] 10 25 2 0 [1.037, 0.402, 0.318] 11 10 0 0 [1.042, 0.433, 0.275] 12 16 1 1 [0.857, 0.45, 0.317] 13 12 1 1 [0.987, 0.491, 0.331] 14 23 2 1 [0.963, 0.502, 0.416] 15 26 1 0 [0.924, 0.48, 0.307] 16 23 1 1 [0.916, 0.526, 0.351] 17 14 0 0 [0.833, 0.473, 0.391] 18 24 1 1 [0.958, 0.563, 0.393] 19 12 0 0 [0.926, 0.495, 0.399] 20 11 1 0 [1.044, 0.475, 0.313] 21 23 1 0 [0.964, 0.535, 0.41] 22 14 0 0 [0.864, 0.47, 0.333] 23 21 3 3 [0.85, 0.468, 0.354] 24 13 0 0 [0.808, 0.38, 0.322] 25 14 0 0 [0.553, 0.472, 0.371] 26 17 1 1 [0.851, 0.46, 0.331] 27 16 0 0 [0.879, 0.527, 0.39] Most mutually aligned direct/H5/H1 MLP covectors: neuron 14 cluster 3 pairwise cosines [0.748 0.655 0.76 ] norms [0.444, 0.405, 0.248] center cosines [0.619, 0.37, 0.265] neuron 251 cluster 1 pairwise cosines [0.668 0.645 0.656] norms [0.48, 0.391, 0.221] center cosines [0.73, 0.59, 0.591] neuron 1206 cluster 21 pairwise cosines [0.6 0.768 0.592] norms [0.577, 0.444, 0.326] center cosines [0.787, 0.642, 0.561] neuron 129 cluster 7 pairwise cosines [0.586 0.654 0.619] norms [0.604, 0.497, 0.363] center cosines [0.604, 0.435, 0.637] neuron 753 cluster 6 pairwise cosines [0.71 0.586 0.655] norms [0.418, 0.42, 0.311] center cosines [0.675, 0.575, 0.418] neuron 1265 cluster 3 pairwise cosines [0.693 0.679 0.582] norms [0.599, 0.441, 0.363] center cosines [0.572, 0.397, 0.532] neuron 795 cluster 27 pairwise cosines [0.665 0.581 0.66 ] norms [0.549, 0.5, 0.371] center cosines [0.654, 0.478, 0.533] neuron 85 cluster 23 pairwise cosines [0.596 0.638 0.573] norms [0.315, 0.354, 0.199] center cosines [0.402, 0.231, 0.29] neuron 1065 cluster 1 pairwise cosines [0.706 0.57 0.641] norms [0.519, 0.365, 0.32] center cosines [0.501, 0.383, 0.345] neuron 26 cluster 14 pairwise cosines [0.532 0.532 0.656] norms [0.424, 0.406, 0.378] center cosines [0.477, 0.231, 0.445] neuron 310 cluster 1 pairwise cosines [0.518 0.552 0.69 ] norms [0.542, 0.588, 0.251] center cosines [0.649, 0.485, 0.513] neuron 570 cluster 12 pairwise cosines [0.68 0.511 0.506] norms [0.495, 0.475, 0.21] center cosines [0.639, 0.369, 0.394] neuron 581 cluster 1 pairwise cosines [0.735 0.505 0.782] norms [0.499, 0.557, 0.324] center cosines [0.397, 0.243, 0.377] neuron 311 cluster 26 pairwise cosines [0.531 0.691 0.501] norms [0.548, 0.447, 0.394] center cosines [0.696, 0.642, 0.629] neuron 325 cluster 10 pairwise cosines [0.499 0.607 0.61 ] norms [0.64, 0.324, 0.316] center cosines [0.71, 0.664, 0.83] neuron 1240 cluster 3 pairwise cosines [0.536 0.489 0.733] norms [0.395, 0.449, 0.306] center cosines [0.603, 0.404, 0.489] neuron 1006 cluster 5 pairwise cosines [0.47 0.543 0.679] norms [0.378, 0.264, 0.16] center cosines [0.712, 0.552, 0.437] neuron 799 cluster 27 pairwise cosines [0.563 0.467 0.669] norms [0.538, 0.539, 0.402] center cosines [0.561, 0.451, 0.298] neuron 1354 cluster 23 pairwise cosines [0.458 0.472 0.538] norms [0.591, 0.313, 0.287] center cosines [0.798, 0.673, 0.741] neuron 642 cluster 1 pairwise cosines [0.448 0.703 0.633] norms [0.731, 0.49, 0.29] center cosines [0.641, 0.516, 0.724] L0H0 token-restricted key singular values [0.059 0.031 0.006 0.002 0.002] key direction 0 [(1, 0.558), (6, 0.41), (0, 0.379), (22, 0.338), (10, 0.331), (3, 0.283)] key direction 1 [(4, -0.469), (6, -0.39), (15, 0.327), (24, 0.321), (9, -0.31), (25, -0.288)] L0H2 token-restricted key singular values [0.058 0.025 0.021 0.019 0.016] key direction 0 [(10, 0.546), (26, 0.376), (23, -0.3), (11, -0.299), (22, 0.259), (24, 0.251)] key direction 1 [(0, 0.49), (3, 0.399), (14, 0.363), (17, 0.313), (2, -0.301), (23, 0.266)] L0H3 token-restricted key singular values [0.083 0.051 0.027 0.017 0.013] key direction 0 [(15, 0.607), (1, -0.487), (25, 0.37), (17, 0.284), (20, 0.276), (18, -0.273)] key direction 1 [(15, -0.415), (0, -0.41), (18, -0.352), (3, -0.29), (22, -0.29), (14, -0.288)] ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
## Observations These are **weight-based hypotheses, not measured attention patterns**. Missing embeddings prevent reliable character identification and exact distance calibration. - **Two related positional heads:** `W_q.0`/`W_k.0` for H1 and H5 share a strong subspace. Their matrix-pencil phases satisfy approximately **phase(H1) = 2 × phase(H5)**, with agreement **0.98–0.99** across several truncation ranks. This strongly suggests two different short offsets—most plausibly two characters back and one character back. - **A broad aggregation head:** L0H4’s LayerNorm-scale-adjusted QK product has Frobenius norm **0.0093**, versus **0.9704** for L0H5. Its adjusted OV product has leading singular values **13.877, 9.644, 2.507**. Thus it combines weak selection with strong, low-dimensional information transport. - **Later heads read that aggregate:** The top-three write directions of L0H4 capture **76.5–93.1%** of the bias-derived key-selection coefficient energy in the six L1 heads. Together with its overlap with the positional subspace, this suggests a shared positional or recency signal. - **Induction-like connections:** Products of later QK matrices with earlier OV matrices show substantial positive matching structure. L2H1, L3H1 and L3H5 consistently stand out. - **Corpus identification is weaker:** `W_proj.3` has 25 singular values above 10, and early weight-direction clustering exposes substantial differentiated structure. Neither result identifies letters or proves a particular language. For the head evidence below: - **R** is the fraction of the bias-derived key-selection coefficient lying in L0H4’s top-three write directions. - **I(h←g)** is the trace/Frobenius-norm ratio of the projected QK–OV composition, using L0H5’s top-24 OV read directions. Positive values support matching circuitry; they are **not probabilities**. ## Heads L0H0: **unclear** — QK is nearly rank two; suggests specialized selection, but neither its target nor its distance is identifiable. L0H1: **local** — strong shared positional structure with H5; approximately double its phase, suggesting two characters back. L0H2: **unclear** — mixed positional and content-read structure; no stable offset or induction signature. L0H3: **unclear** — low-rank selection, but its apparent offset changes with the chosen subspace. L0H4: **broad** — exceptionally weak QK product and strong low-rank OV product favor broad aggregation. L0H5: **previous** — strongest candidate for the fundamental short-offset transport head; supports subsequent induction-like circuits. L1H0: **local** — R = **0.765**; shares the common positional-selection direction, though less strongly than its neighbors. L1H1: **local** — R = **0.905**; strongly coupled to the shared positional aggregate. L1H2: **local** — R = **0.908**; strong shared positional-selection structure. L1H3: **local** — R = **0.931**, the largest L1 value; likely recency-oriented selection. L1H4: **local** — R = **0.919**; closely aligned with L1H3’s selection coefficient. L1H5: **local** — R = **0.825**; same routing family, with more additional structure. L2H0: **local** — tentative: R = **0.465** and alignment with L1 selection directions; no strong induction composition. L2H1: **copy** — I(L2H1←L0H5) = **2.82**, robust across alternative read subspaces; strong induction candidate. L2H2: **local** — R = **0.858**; its OV product supplies a strong matching connection to later copy candidates. Exact offset unresolved. L2H3: **local** — R = **0.804**; strong continuation of the earlier positional-selection family. L2H4: **local** — R = **0.672**; substantial shared recency-related structure, without a clear induction signature. L2H5: **unclear** — weaker positional coupling, R = **0.270**, and only modest matching compositions. L3H0: **unclear** — some induction-like alignment, I(L3H0←L0H5) = **1.26**, but appreciably weaker than the clear candidates. L3H1: **copy** — I values **2.66** through L0H5 and **2.23** through L2H2; convergent matching evidence. L3H2: **unclear** — distributed QK structure; neither locality nor copying is strongly established. L3H3: **unclear** — relatively weak QK and strong OV suggest aggregation, but do not establish broad attention. L3H4: **unclear** — mixed connections, without a sufficiently distinctive positional or induction signature. L3H5: **copy** — I values approximately **2.77** through both L0H5 and L2H2; strongest combined induction evidence. ## Reasoning The paired short-offset heads and later matching circuits strongly favor **as-is** text. Line shuffling preserves composition and line structure, but largely removes the predictable ordered continuations that make these circuits useful. My weaker corpus preference is **Python**: several matching circuits and substantial position-sensitive machinery fit repeated identifiers and structured source text. English remains quite plausible; these features are not code-specific. Romaji is less favored, but the unlabeled weight geometry does not decisively exclude it. I have **not** identified indentation, particular characters, or keywords. The probabilities below are subjective judgments, not calibrated estimates. corpus: python order: as is probabilities: python=0.55 romaji=0.07 english=0.38; as is=0.98 shuffled=0.02
## Observations These are **weight-based hypotheses, not measured attention patterns**. Missing embeddings prevent reliable character identification and exact distance calibration. - **Two related positional heads:** `W_q.0`/`W_k.0` for H1 and H5 share a strong subspace. Their matrix-pencil phases satisfy approximately **phase(H1) = 2 × phase(H5)**, with agreement **0.98–0.99** across several truncation ranks. This strongly suggests two different short offsets—most plausibly two characters back and one character back. - **A broad aggregation head:** L0H4’s LayerNorm-scale-adjusted QK product has Frobenius norm **0.0093**, versus **0.9704** for L0H5. Its adjusted OV product has leading singular values **13.877, 9.644, 2.507**. Thus it combines weak selection with strong, low-dimensional information transport. - **Later heads read that aggregate:** The top-three write directions of L0H4 capture **76.5–93.1%** of the bias-derived key-selection coefficient energy in the six L1 heads. Together with its overlap with the positional subspace, this suggests a shared positional or recency signal. - **Induction-like connections:** Products of later QK matrices with earlier OV matrices show substantial positive matching structure. L2H1, L3H1 and L3H5 consistently stand out. - **Corpus identification is weaker:** `W_proj.3` has 25 singular values above 10, and early weight-direction clustering exposes substantial differentiated structure. Neither result identifies letters or proves a particular language. For the head evidence below: - **R** is the fraction of the bias-derived key-selection coefficient lying in L0H4’s top-three write directions. - **I(h←g)** is the trace/Frobenius-norm ratio of the projected QK–OV composition, using L0H5’s top-24 OV read directions. Positive values support matching circuitry; they are **not probabilities**. ## Heads L0H0: **unclear** — QK is nearly rank two; suggests specialized selection, but neither its target nor its distance is identifiable. L0H1: **local** — strong shared positional structure with H5; approximately double its phase, suggesting two characters back. L0H2: **unclear** — mixed positional and content-read structure; no stable offset or induction signature. L0H3: **unclear** — low-rank selection, but its apparent offset changes with the chosen subspace. L0H4: **broad** — exceptionally weak QK product and strong low-rank OV product favor broad aggregation. L0H5: **previous** — strongest candidate for the fundamental short-offset transport head; supports subsequent induction-like circuits. L1H0: **local** — R = **0.765**; shares the common positional-selection direction, though less strongly than its neighbors. L1H1: **local** — R = **0.905**; strongly coupled to the shared positional aggregate. L1H2: **local** — R = **0.908**; strong shared positional-selection structure. L1H3: **local** — R = **0.931**, the largest L1 value; likely recency-oriented selection. L1H4: **local** — R = **0.919**; closely aligned with L1H3’s selection coefficient. L1H5: **local** — R = **0.825**; same routing family, with more additional structure. L2H0: **local** — tentative: R = **0.465** and alignment with L1 selection directions; no strong induction composition. L2H1: **copy** — I(L2H1←L0H5) = **2.82**, robust across alternative read subspaces; strong induction candidate. L2H2: **local** — R = **0.858**; its OV product supplies a strong matching connection to later copy candidates. Exact offset unresolved. L2H3: **local** — R = **0.804**; strong continuation of the earlier positional-selection family. L2H4: **local** — R = **0.672**; substantial shared recency-related structure, without a clear induction signature. L2H5: **unclear** — weaker positional coupling, R = **0.270**, and only modest matching compositions. L3H0: **unclear** — some induction-like alignment, I(L3H0←L0H5) = **1.26**, but appreciably weaker than the clear candidates. L3H1: **copy** — I values **2.66** through L0H5 and **2.23** through L2H2; convergent matching evidence. L3H2: **unclear** — distributed QK structure; neither locality nor copying is strongly established. L3H3: **unclear** — relatively weak QK and strong OV suggest aggregation, but do not establish broad attention. L3H4: **unclear** — mixed connections, without a sufficiently distinctive positional or induction signature. L3H5: **copy** — I values approximately **2.77** through both L0H5 and L2H2; strongest combined induction evidence. ## Reasoning The paired short-offset heads and later matching circuits strongly favor **as-is** text. Line shuffling preserves composition and line structure, but largely removes the predictable ordered continuations that make these circuits useful. My weaker corpus preference is **Python**: several matching circuits and substantial position-sensitive machinery fit repeated identifiers and structured source text. English remains quite plausible; these features are not code-specific. Romaji is less favored, but the unlabeled weight geometry does not decisively exclude it. I have **not** identified indentation, particular characters, or keywords. The probabilities below are subjective judgments, not calibrated estimates. corpus: python order: as is probabilities: python=0.55 romaji=0.07 english=0.38; as is=0.98 shuffled=0.02