Built-in Codex tools were disabled. Harness execution is separately permitted only in Experiments 3 and 6. Text below is taken from saved records; it is not a new model run. Reasoning blocks are the summaries actually emitted by the CLI.
Answer the user's request.
Exact prompt · Raw CLI events · Settings and audit
Below is a small character-level language model: a 4-layer transformer with 6 attention heads per layer, which reads up to 128 characters and outputs scores (logits) for the next character. This is exactly how it computes (numpy):
```python
import numpy as np
VOCAB = "\n " + "abcdefghijklmnopqrstuvwxyz" # index 0 = newline, 1 = space, 2..27 = a..z
HEADS, LAYERS = 6, 4
def layer_norm(x, w, b): # over the last axis, eps 1e-5
mu = x.mean(-1, keepdims=True)
var = ((x - mu) ** 2).mean(-1, keepdims=True)
return (x - mu) / np.sqrt(var + 1e-5) * w + b
def next_char_logits(text, p):
"""text: up to 128 characters (only the last 128 are used); p: dict of the weight arrays.
Returns the 28 logits for the character that comes next."""
ids = [VOCAB.index(c) for c in text[-128:]]
n, d = len(ids), p["tok_emb"].shape[1]
hd = d // HEADS
x = p["tok_emb"][ids] + p["pos_emb"][:n] # (n, d); positions 0..n-1 from the start of the text
for l in range(LAYERS): # layer l uses the tensors named <name>.<l>
h = layer_norm(x, p[f"ln1_w.{l}"], p[f"ln1_b.{l}"])
q, k, v = h @ p[f"W_q.{l}"].T, h @ p[f"W_k.{l}"].T, h @ p[f"W_v.{l}"].T
out = np.zeros((n, d))
for j in range(HEADS): # head j uses dimensions j*hd .. (j+1)*hd - 1
s = slice(j * hd, (j + 1) * hd)
att = q[:, s] @ k[:, s].T / np.sqrt(hd) # (n, n)
att = att + np.triu(np.full((n, n), -np.inf), 1) # causal: position i sees positions 0..i
att = np.exp(att - att.max(-1, keepdims=True))
att = att / att.sum(-1, keepdims=True)
out[:, s] = att @ v[:, s]
x = x + out @ p[f"W_o.{l}"].T
h = layer_norm(x, p[f"ln2_w.{l}"], p[f"ln2_b.{l}"])
x = x + np.maximum(0.0, h @ p[f"W_fc.{l}"].T) @ p[f"W_proj.{l}"].T # ReLU MLP, hidden 4 d
h = layer_norm(x, p["lnf_w"], p["lnf_b"])
return h[-1] @ p["W_out"].T + p["b_out"] # logits of the next character (28)
```
Training: from random initialization (every matrix entry drawn from a normal distribution with standard deviation 0.02, or 0.007071 for W_o and W_proj; LayerNorm weights 1 and biases 0; b_out 0), next-character prediction with AdamW (30,517 steps of 256 random 128-character windows, 1,000,000,000 characters in total; learning-rate warm-up then cosine decay; weight decay 0.01 on the matrices).
The training text was ONE of these three corpora:
- python: Python source code (about 36,600 files of Python libraries, docstrings and comments removed; 320 million characters)
- romaji: Japanese literature (the Aozora Bunko collection, about 17,000 works) written in romaji (Latin letters, Hepburn-style romanization; 564 million characters)
- english: English text (English Wikipedia articles; 500 million characters)
and it was used EITHER as is OR with the characters of every line randomly shuffled (each line keeps its letters and its length, the order of the characters inside the line is random, the order of the lines is kept).
Normalization (the same for all three corpora, before anything else): lowercased; in each line the leading spaces (indentation) were kept, every other character that is not a letter a-z became a space, runs of spaces were collapsed to one and trailing spaces removed. So the text contains only newlines, spaces and the letters a-z (the 28 characters of VOCAB).
WHAT YOU CAN USE. The weights inside the four transformer blocks, exactly as trained (float32): for layer l = 0, 1, 2 and 3, ln1_w.l, ln1_b.l, W_q.l, W_k.l, W_v.l, W_o.l (attention) and ln2_w.l, ln2_b.l, W_fc.l, W_proj.l (MLP). The input layer (tok_emb, the 28 x d character embeddings, and pos_emb, the 128 x d position embeddings) and the output layer (lnf_w, lnf_b, W_out, b_out) are NOT available. The d = 384 dimensions of the residual stream carry no labels. Head j of layer l uses rows 64j .. 64j+63 of W_q.l, W_k.l and W_v.l and columns 64j .. 64j+63 of W_o.l.
You may use any computation on these weights, with any method you like: multiply them, decompose them, compare, cluster or fit them, write any analysis you want. Two things are NOT allowed:
1. A forward pass: do not run the network or any of its layers (LayerNorm, attention with its softmax, the MLP with its ReLU) on any input, whether real, random or constructed, in code, in your head or on paper. Treating the weight matrices as linear-algebra objects (products, eigenvalues, singular vectors, applying one weight matrix to directions taken from other weights) is fine; simulating what the network does to an input is not.
2. Search: do not look anything up (there is no internet access in any case).
HOW YOU COMPUTE. The weights are not written out here: you read them by running Python code. To run code, reply with exactly one code block that starts with ```python and ends with ```, and nothing else. We run it and reply with its printed output. Available: numpy, scipy, scikit-learn, PyTorch (CPU), pandas, sympy, networkx. Before your code runs, numpy is imported as np and W is a dict of the weight arrays by name: W['W_q.0'] is 384 x 384, W['W_fc.0'] is 1536 x 384, W['W_proj.0'] is 384 x 1536, W['ln1_w.0'] has 384 entries, and so on; matrices are (out, in), as used in the code above. There is nothing else to read (no other files, no network). Each run is a fresh process (nothing is kept between runs), may take up to 600 seconds, and its output is cut at 20,000 characters. You may run code up to 100 times. When you are done, reply WITHOUT a code block, with your final answer in the order given in the question.
QUESTION. What was this network trained on, and how does it use the context? Answer in this order:
Observations: <what you see in the shown weights and what it suggests; quote the weights you rely on>
Heads: <one line per head, L0H0 .. L0H5, L1H0 .. L1H5, L2H0 .. L2H5, L3H0 .. L3H5: what it most likely does, as one of: 'previous' (attends to the previous character), 'local' (a few characters back), 'broad' (spread far back), 'copy' (induction: attends to the character right after an earlier occurrence of the current character or context, so the network can repeat earlier text), or 'unclear'; then your evidence>
Reasoning: <which training text fits these observations, and why>
corpus: <python | romaji | english>
order: <as is | shuffled>
probabilities: python=<p> romaji=<p> english=<p>; as is=<p> shuffled=<p>
Equal probabilities are not allowed. Even if the evidence is weak, give your best judgement.Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(2)
np.set_printoptions(precision=4, suppress=True)
print("WEIGHT-ONLY ANALYSIS: no inputs, activations, or forward passes.")
print("Keys and shapes:", [(k, tuple(v.shape)) for k, v in sorted(W.items())])
def spectrum_summary(a):
s = np.linalg.svd(a.astype(np.float64), compute_uv=False)
e = np.cumsum(s*s) / np.sum(s*s)
return np.linalg.norm(a), s[0], int(np.searchsorted(e, .5)+1), int(np.searchsorted(e, .9)+1)
for l in range(4):
print("\nLAYER", l)
for name in ["ln1_w", "ln1_b", "ln2_w", "ln2_b"]:
a = W[f"{name}.{l}"]
print(name, "min/mean/max/std/norm:",
np.round([a.min(), a.mean(), a.max(), a.std(), np.linalg.norm(a)], 5))
for name in ["W_q", "W_k", "W_v", "W_o", "W_fc", "W_proj"]:
a = W[f"{name}.{l}"]
norm, top, r50, r90 = spectrum_summary(a)
print(name, "norm/topSV/r50/r90:", round(float(norm), 4),
round(float(top), 4), r50, r90)
print("\nHEAD PRODUCTS")
print("QK=Q.T@K/sqrt(64); OV=O@V.")
print("sym=<A,A.T>/||A||F^2; trN=trace(A)/nuclear_norm(A).")
print("Columns: head; QK(norm,topSV,r50,r90,sym,trN); "
"OV(norm,topSV,r50,r90,sym,trN); leading nonzero eigenvalues by magnitude.")
for l in range(4):
for h in range(6):
sl = slice(64*h, 64*(h+1))
q = W[f"W_q.{l}"][sl].astype(np.float64)
k = W[f"W_k.{l}"][sl].astype(np.float64)
v = W[f"W_v.{l}"][sl].astype(np.float64)
o = W[f"W_o.{l}"][:, sl].astype(np.float64)
summaries, eigenvalues = [], []
for a, small in [(q.T @ k / 8, k @ q.T / 8), (o @ v, v @ o)]:
s = np.linalg.svd(a, compute_uv=False)
norm = np.linalg.norm(a)
e = np.cumsum(s*s) / np.sum(s*s)
summaries.append([
round(norm, 4), round(s[0], 4),
int(np.searchsorted(e, .5)+1), int(np.searchsorted(e, .9)+1),
round(np.sum(a*a.T)/(norm*norm), 4),
round(np.trace(a)/s.sum(), 4)
])
ev = np.linalg.eigvals(small)
ev = ev[np.argsort(-np.abs(ev))][:5]
eigenvalues.append(np.round(ev, 3))
print(f"L{l}H{h}", summaries[0], summaries[1],
"QKev", eigenvalues[0], "OVev", eigenvalues[1])
```{
"run": 1,
"turn": "t001",
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(2)\n\nnp.set_printoptions(precision=4, suppress=True)\nprint(\"WEIGHT-ONLY ANALYSIS: no inputs, activations, or forward passes.\")\nprint(\"Keys and shapes:\", [(k, tuple(v.shape)) for k, v in sorted(W.items())])\n\ndef spectrum_summary(a):\n s = np.linalg.svd(a.astype(np.float64), compute_uv=False)\n e = np.cumsum(s*s) / np.sum(s*s)\n return np.linalg.norm(a), s[0], int(np.searchsorted(e, .5)+1), int(np.searchsorted(e, .9)+1)\n\nfor l in range(4):\n print(\"\\nLAYER\", l)\n for name in [\"ln1_w\", \"ln1_b\", \"ln2_w\", \"ln2_b\"]:\n a = W[f\"{name}.{l}\"]\n print(name, \"min/mean/max/std/norm:\",\n np.round([a.min(), a.mean(), a.max(), a.std(), np.linalg.norm(a)], 5))\n for name in [\"W_q\", \"W_k\", \"W_v\", \"W_o\", \"W_fc\", \"W_proj\"]:\n a = W[f\"{name}.{l}\"]\n norm, top, r50, r90 = spectrum_summary(a)\n print(name, \"norm/topSV/r50/r90:\", round(float(norm), 4),\n round(float(top), 4), r50, r90)\n\nprint(\"\\nHEAD PRODUCTS\")\nprint(\"QK=Q.T@K/sqrt(64); OV=O@V.\")\nprint(\"sym=<A,A.T>/||A||F^2; trN=trace(A)/nuclear_norm(A).\")\nprint(\"Columns: head; QK(norm,topSV,r50,r90,sym,trN); \"\n \"OV(norm,topSV,r50,r90,sym,trN); leading nonzero eigenvalues by magnitude.\")\n\nfor l in range(4):\n for h in range(6):\n sl = slice(64*h, 64*(h+1))\n q = W[f\"W_q.{l}\"][sl].astype(np.float64)\n k = W[f\"W_k.{l}\"][sl].astype(np.float64)\n v = W[f\"W_v.{l}\"][sl].astype(np.float64)\n o = W[f\"W_o.{l}\"][:, sl].astype(np.float64)\n summaries, eigenvalues = [], []\n for a, small in [(q.T @ k / 8, k @ q.T / 8), (o @ v, v @ o)]:\n s = np.linalg.svd(a, compute_uv=False)\n norm = np.linalg.norm(a)\n e = np.cumsum(s*s) / np.sum(s*s)\n summaries.append([\n round(norm, 4), round(s[0], 4),\n int(np.searchsorted(e, .5)+1), int(np.searchsorted(e, .9)+1),\n round(np.sum(a*a.T)/(norm*norm), 4),\n round(np.trace(a)/s.sum(), 4)\n ])\n ev = np.linalg.eigvals(small)\n ev = ev[np.argsort(-np.abs(ev))][:5]\n eigenvalues.append(np.round(ev, 3))\n print(f\"L{l}H{h}\", summaries[0], summaries[1],\n \"QKev\", eigenvalues[0], \"OVev\", eigenvalues[1])",
"out": "WEIGHT-ONLY ANALYSIS: no inputs, activations, or forward passes.\nKeys and shapes: [('W_fc.0', (1536, 384)), ('W_fc.1', (1536, 384)), ('W_fc.2', (1536, 384)), ('W_fc.3', (1536, 384)), ('W_k.0', (384, 384)), ('W_k.1', (384, 384)), ('W_k.2', (384, 384)), ('W_k.3', (384, 384)), ('W_o.0', (384, 384)), ('W_o.1', (384, 384)), ('W_o.2', (384, 384)), ('W_o.3', (384, 384)), ('W_proj.0', (384, 1536)), ('W_proj.1', (384, 1536)), ('W_proj.2', (384, 1536)), ('W_proj.3', (384, 1536)), ('W_q.0', (384, 384)), ('W_q.1', (384, 384)), ('W_q.2', (384, 384)), ('W_q.3', (384, 384)), ('W_v.0', (384, 384)), ('W_v.1', (384, 384)), ('W_v.2', (384, 384)), ('W_v.3', (384, 384)), ('ln1_b.0', (384,)), ('ln1_b.1', (384,)), ('ln1_b.2', (384,)), ('ln1_b.3', (384,)), ('ln1_w.0', (384,)), ('ln1_w.1', (384,)), ('ln1_w.2', (384,)), ('ln1_w.3', (384,)), ('ln2_b.0', (384,)), ('ln2_b.1', (384,)), ('ln2_b.2', (384,)), ('ln2_b.3', (384,)), ('ln2_w.0', (384,)), ('ln2_w.1', (384,)), ('ln2_w.2', (384,)), ('ln2_w.3', (384,))]\n\nLAYER 0\nln1_w min/mean/max/std/norm: [-0.001 0.2403 1.0341 0.2436 6.7051]\nln1_b min/mean/max/std/norm: [-0.0959 0.0008 0.091 0.0287 0.5635]\nln2_w min/mean/max/std/norm: [ 0.0044 0.8006 1.5072 0.3295 16.9648]\nln2_b min/mean/max/std/norm: [-0.4986 0.0022 0.3331 0.0683 1.3392]\nW_q norm/topSV/r50/r90: 32.2912 8.2519 9 30\nW_k norm/topSV/r50/r90: 30.3909 7.6464 10 35\nW_v norm/topSV/r50/r90: 14.836 4.339 14 97\nW_o norm/topSV/r50/r90: 16.3713 7.1033 7 49\nW_fc norm/topSV/r50/r90: 46.2485 18.1759 16 79\nW_proj norm/topSV/r50/r90: 39.3073 9.8257 27 99\n\nLAYER 1\nln1_w min/mean/max/std/norm: [0.1361 0.4575 1.1239 0.1022 9.1868]\nln1_b min/mean/max/std/norm: [-1.4713 -0.005 1.1544 0.1672 3.2771]\nln2_w min/mean/max/std/norm: [-0.0019 1.0429 1.4856 0.2003 20.8093]\nln2_b min/mean/max/std/norm: [-0.3597 0.0005 0.3686 0.1364 2.6732]\nW_q norm/topSV/r50/r90: 32.3126 9.9032 15 54\nW_k norm/topSV/r50/r90: 29.2506 13.3072 8 53\nW_v norm/topSV/r50/r90: 27.3529 5.2845 22 76\nW_o norm/topSV/r50/r90: 25.9233 4.9657 42 113\nW_fc norm/topSV/r50/r90: 79.1264 21.4881 24 98\nW_proj norm/topSV/r50/r90: 75.7102 8.1935 105 279\n\nLAYER 2\nln1_w min/mean/max/std/norm: [ 0.2078 0.7052 1.0144 0.0749 13.8969]\nln1_b min/mean/max/std/norm: [-1.3819 -0.001 1.0527 0.1525 2.9893]\nln2_w min/mean/max/std/norm: [ 0.0011 1.3123 1.4655 0.1255 25.8329]\nln2_b min/mean/max/std/norm: [-0.3761 0.005 0.3886 0.1537 3.0128]\nW_q norm/topSV/r50/r90: 39.3455 5.7962 53 169\nW_k norm/topSV/r50/r90: 37.7688 10.6627 46 164\nW_v norm/topSV/r50/r90: 33.3048 3.7062 62 179\nW_o norm/topSV/r50/r90: 32.8397 4.1025 67 189\nW_fc norm/topSV/r50/r90: 78.9667 15.3634 82 256\nW_proj norm/topSV/r50/r90: 76.831 9.4036 87 266\n\nLAYER 3\nln1_w min/mean/max/std/norm: [ 0.2944 0.7546 0.8701 0.0542 14.8247]\nln1_b min/mean/max/std/norm: [-0.6069 0.0121 0.6502 0.0849 1.6801]\nln2_w min/mean/max/std/norm: [ 0.4364 1.2778 1.4306 0.0808 25.0891]\nln2_b min/mean/max/std/norm: [-0.6728 0.0043 0.3068 0.1276 2.5016]\nW_q norm/topSV/r50/r90: 32.9482 4.5598 56 176\nW_k norm/topSV/r50/r90: 32.2094 6.0903 61 184\nW_v norm/topSV/r50/r90: 39.7941 5.0598 52 170\nW_o norm/topSV/r50/r90: 40.4549 9.5668 41 157\nW_fc norm/topSV/r50/r90: 77.7757 14.1624 97 277\nW_proj norm/topSV/r50/r90: 82.395 26.3094 9 24\n\nHEAD PRODUCTS\nQK=Q.T@K/sqrt(64); OV=O@V.\nsym=<A,A.T>/||A||F^2; trN=trace(A)/nuclear_norm(A).\nColumns: head; QK(norm,topSV,r50,r90,sym,trN); OV(norm,topSV,r50,r90,sym,trN); leading nonzero eigenvalues by magnitude.\nL0H0 [np.float64(0.381), np.float64(0.3137), 1, 12, np.float64(0.0561), np.float64(-0.1312)] [np.float64(12.9041), np.float64(10.2524), 1, 4, np.float64(0.0078), np.float64(-0.0051)] QKev [-0.068+0.j -0.041+0.008j -0.041-0.008j -0.016+0.j -0.003+0.01j ] OVev [ 1.173+0.j -0.432+0.575j -0.432-0.575j -0.38 +0.j 0.274+0.244j]\nL0H1 [np.float64(0.2297), np.float64(0.1371), 2, 27, np.float64(0.1165), np.float64(-0.1169)] [np.float64(10.1485), np.float64(7.662), 1, 4, np.float64(0.0184), np.float64(-0.0264)] QKev [-0.07 +0.j -0.034+0.j -0.009+0.01j -0.009-0.01j 0.012+0.j ] OVev [ 0.981+0.j -0.924+0.j -0.229+0.315j -0.229-0.315j 0.388+0.j ]\nL0H2 [np.float64(22.8044), np.float64(7.4848), 6, 14, np.float64(-0.0304), np.float64(0.0382)] [np.float64(13.223), np.float64(4.0009), 7, 16, np.float64(0.0098), np.float64(-0.069)] QKev [-0.867+5.787j -0.867-5.787j 1.92 +4.997j 1.92 -4.997j 4.415+2.676j] OVev [-0.698+0.122j -0.698-0.122j -0.473+0.j -0.32 +0.318j -0.32 -0.318j]\nL0H3 [np.float64(1.4189), np.float64(0.821), 2, 9, np.float64(0.1641), np.float64(-0.169)] [np.float64(7.3079), np.float64(4.305), 2, 5, np.float64(0.0159), np.float64(-0.0816)] QKev [ 0.304+0.j -0.211+0.064j -0.211-0.064j 0.194+0.048j 0.194-0.048j] OVev [-0.691+0.j -0.332+0.027j -0.332-0.027j 0.262+0.j -0.204+0.j ]\nL0H4 [np.float64(0.8259), np.float64(0.7527), 1, 2, np.float64(0.1641), np.float64(-0.2512)] [np.float64(7.3117), np.float64(4.8786), 2, 6, np.float64(0.0082), np.float64(-0.0381)] QKev [-0.309+0.j -0.103+0.j -0.07 +0.j -0.021+0.j -0.014+0.009j] OVev [-0.467+0.j 0.36 +0.j -0.264+0.j -0.221+0.j 0.107+0.141j]\nL0H5 [np.float64(0.3223), np.float64(0.2737), 1, 15, np.float64(0.0624), np.float64(-0.0847)] [np.float64(13.0384), np.float64(10.3142), 1, 4, np.float64(0.0039), np.float64(-0.0023)] QKev [-0.079+0.j -0.016+0.j -0.012+0.j 0.006+0.008j 0.006-0.008j] OVev [ 0.827+0.j -0.41 +0.498j -0.41 -0.498j 0.423+0.j 0.104+0.189j]\nL1H0 [np.float64(2.9582), np.float64(1.5768), 2, 10, np.float64(0.0806), np.float64(-0.2659)] [np.float64(21.7066), np.float64(8.0989), 5, 15, np.float64(0.0885), np.float64(-0.2035)] QKev [-0.383+0.j -0.356+0.105j -0.356-0.105j -0.24 +0.059j -0.24 -0.059j] OVev [-2.776+0.j -2.372+0.124j -2.372-0.124j -2.193+0.345j -2.193-0.345j]\nL1H1 [np.float64(4.4391), np.float64(2.4561), 2, 11, np.float64(0.1022), np.float64(-0.3042)] [np.float64(22.226), np.float64(6.5833), 9, 22, np.float64(0.0682), np.float64(-0.2227)] QKev [-0.684+0.j -0.515+0.096j -0.515-0.096j -0.478+0.019j -0.478-0.019j] OVev [-2.052+0.j -1.972+0.206j -1.972-0.206j -1.509-0.28j -1.509+0.28j ]\nL1H2 [np.float64(5.2226), np.float64(2.9227), 2, 11, np.float64(0.1095), np.float64(-0.3463)] [np.float64(19.5165), np.float64(6.1346), 8, 19, np.float64(0.0494), np.float64(-0.2067)] QKev [-0.769+0.j -0.621+0.287j -0.621-0.287j -0.656+0.j -0.608+0.j ] OVev [-1.708+0.229j -1.708-0.229j -1.42 +0.326j -1.42 -0.326j -1.325+0.j ]\nL1H3 [np.float64(4.6046), np.float64(2.6384), 2, 11, np.float64(0.0956), np.float64(-0.2981)] [np.float64(21.4133), np.float64(7.7675), 6, 15, np.float64(0.0452), np.float64(-0.1952)] QKev [-0.822+0.j -0.576+0.j -0.488+0.012j -0.488-0.012j -0.405+0.085j] OVev [-1.88 +0.j -1.706+0.128j -1.706-0.128j -1.526+0.j -1.438+0.j ]\nL1H4 [np.float64(4.6727), np.float64(2.8698), 2, 11, np.float64(0.0851), np.float64(-0.3096)] [np.float64(22.0658), np.float64(6.4984), 8, 18, np.float64(0.0675), np.float64(-0.228)] QKev [-0.566+0.06j -0.566-0.06j -0.55 +0.j -0.451+0.174j -0.451-0.174j] OVev [-2.065+0.j -2.045+0.265j -2.045-0.265j -1.945+0.j -1.467+0.482j]\nL1H5 [np.float64(4.5264), np.float64(2.6178), 2, 11, np.float64(0.0644), np.float64(-0.2713)] [np.float64(22.4638), np.float64(6.2081), 9, 20, np.float64(0.0683), np.float64(-0.2384)] QKev [-0.52 +0.114j -0.52 -0.114j -0.524+0.j -0.377+0.287j -0.377-0.287j] OVev [-2.115+0.j -2.009+0.j -1.932+0.409j -1.932-0.409j -1.638+0.108j]\nL2H0 [np.float64(4.8218), np.float64(1.4573), 11, 33, np.float64(-0.0144), np.float64(0.0075)] [np.float64(22.8142), np.float64(4.8981), 15, 41, np.float64(0.1187), np.float64(-0.3236)] QKev [0.111+0.4j 0.111-0.4j 0.118+0.375j 0.118-0.375j 0.237+0.157j] OVev [-2.112+0.j -2.02 +0.513j -2.02 -0.513j -2.002+0.082j -2.002-0.082j]\nL2H1 [np.float64(4.1385), np.float64(1.1546), 13, 39, np.float64(0.0163), np.float64(0.0398)] [np.float64(24.8446), np.float64(5.4684), 16, 42, np.float64(0.0796), np.float64(-0.2716)] QKev [ 0.264+0.j 0.233+0.087j 0.233-0.087j 0.245+0.j -0.167+0.139j] OVev [-2.182+0.269j -2.182-0.269j -1.599+1.285j -1.599-1.285j -1.913+0.j ]\nL2H2 [np.float64(3.1536), np.float64(1.11), 11, 39, np.float64(0.0121), np.float64(-0.0914)] [np.float64(28.7074), np.float64(7.8628), 12, 38, np.float64(0.0723), np.float64(-0.2283)] QKev [ 0.031+0.213j 0.031-0.213j 0.174+0.113j 0.174-0.113j -0.196+0.j ] OVev [-2.963+0.j -2.564+0.636j -2.564-0.636j -2.556+0.j -2.024+0.j ]\nL2H3 [np.float64(4.1027), np.float64(1.0661), 13, 39, np.float64(-0.0033), np.float64(-0.0502)] [np.float64(24.5055), np.float64(6.1803), 14, 41, np.float64(0.0956), np.float64(-0.2784)] QKev [ 0.11 +0.404j 0.11 -0.404j -0.234+0.j -0.222+0.061j -0.222-0.061j] OVev [-2.719+0.j -2.27 +0.499j -2.27 -0.499j -1.906+0.032j -1.906-0.032j]\nL2H4 [np.float64(4.6034), np.float64(1.1879), 13, 37, np.float64(-0.0074), np.float64(-0.0285)] [np.float64(23.2786), np.float64(5.0699), 16, 41, np.float64(0.08), np.float64(-0.2651)] QKev [ 0.105+0.409j 0.105-0.409j -0.253+0.j 0.161+0.192j 0.161-0.192j] OVev [-1.945+0.j -1.822+0.26j -1.822-0.26j -1.585+0.715j -1.585-0.715j]\nL2H5 [np.float64(4.4193), np.float64(1.2007), 12, 36, np.float64(-0.008), np.float64(-0.0415)] [np.float64(23.3416), np.float64(4.8404), 16, 42, np.float64(0.0994), np.float64(-0.2923)] QKev [ 0.083+0.407j 0.083-0.407j -0.274+0.j -0.158+0.216j -0.158-0.216j] OVev [-2.304+0.j -1.972+0.j -1.941+0.17j -1.941-0.17j -1.757+0.454j]\nL3H0 [np.float64(3.9054), np.float64(0.7905), 18, 46, np.float64(0.0512), np.float64(0.2014)] [np.float64(29.6831), np.float64(8.3838), 13, 41, np.float64(0.1181), np.float64(-0.3382)] QKev [0.321+0.j 0.289+0.028j 0.289-0.028j 0.243+0.137j 0.243-0.137j] OVev [-3.166+0.235j -3.166-0.235j -2.676+0.j -2.512+0.j -2.339+0.41j ]\nL3H1 [np.float64(3.2236), np.float64(0.5714), 21, 50, np.float64(0.2114), np.float64(0.4453)] [np.float64(53.3976), np.float64(13.8537), 10, 23, np.float64(0.0622), np.float64(-0.2166)] QKev [0.301+0.13j 0.301-0.13j 0.309-0.096j 0.309+0.096j 0.304+0.j ] OVev [-5.342+0.j -4.693+0.548j -4.693-0.548j -3.938+0.j -3.693+0.482j]\nL3H2 [np.float64(2.6302), np.float64(0.808), 12, 42, np.float64(-0.0016), np.float64(0.0211)] [np.float64(35.2432), np.float64(11.8908), 9, 33, np.float64(0.0488), np.float64(-0.1901)] QKev [ 0.053+0.216j 0.053-0.216j 0.148+0.075j 0.148-0.075j -0.063+0.142j] OVev [-3.171+0.559j -3.171-0.559j -2.821+0.468j -2.821-0.468j -2.168+1.06j ]\nL3H3 [np.float64(2.7937), np.float64(0.8563), 14, 45, np.float64(-0.0148), np.float64(0.0092)] [np.float64(35.9947), np.float64(12.4344), 9, 37, np.float64(0.0483), np.float64(-0.2096)] QKev [ 0.041+0.26j 0.041-0.26j -0.036+0.163j -0.036-0.163j 0.155+0.024j] OVev [-3.108+0.j -2.921+1.018j -2.921-1.018j -2.591+0.j -2.092+1.046j]\nL3H4 [np.float64(3.2493), np.float64(0.919), 15, 45, np.float64(0.0008), np.float64(0.0666)] [np.float64(31.7775), np.float64(9.5258), 12, 39, np.float64(0.0392), np.float64(-0.1946)] QKev [0.115+0.169j 0.115-0.169j 0.175+0.059j 0.175-0.059j 0.18 +0.j ] OVev [-2.931+0.j -2.449+0.j -2.121+0.j -1.369+1.511j -1.369-1.511j]\nL3H5 [np.float64(1.4813), np.float64(0.4831), 10, 40, np.float64(0.0244), np.float64(0.0166)] [np.float64(59.2935), np.float64(21.9791), 6, 18, np.float64(0.0647), np.float64(-0.1821)] QKev [0.135+0.058j 0.135-0.058j 0.134+0.j 0.042+0.087j 0.042-0.087j] OVev [-8.508+0.j -6.584+0.j -5.094+0.838j -5.094-0.838j -3.835+1.273j]\n",
"seconds": 1.102,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "e916317d4e66f8b04024e042a9bbded66f2dc57ff1d51cba9b7e8f59b3b25710",
"weights_sha256": "130135d2b27829df2734d78fa3af27b00b1cc1bf34c41b01a09b350e6c09a7de"
}WEIGHT-ONLY ANALYSIS: no inputs, activations, or forward passes.
Keys and shapes: [('W_fc.0', (1536, 384)), ('W_fc.1', (1536, 384)), ('W_fc.2', (1536, 384)), ('W_fc.3', (1536, 384)), ('W_k.0', (384, 384)), ('W_k.1', (384, 384)), ('W_k.2', (384, 384)), ('W_k.3', (384, 384)), ('W_o.0', (384, 384)), ('W_o.1', (384, 384)), ('W_o.2', (384, 384)), ('W_o.3', (384, 384)), ('W_proj.0', (384, 1536)), ('W_proj.1', (384, 1536)), ('W_proj.2', (384, 1536)), ('W_proj.3', (384, 1536)), ('W_q.0', (384, 384)), ('W_q.1', (384, 384)), ('W_q.2', (384, 384)), ('W_q.3', (384, 384)), ('W_v.0', (384, 384)), ('W_v.1', (384, 384)), ('W_v.2', (384, 384)), ('W_v.3', (384, 384)), ('ln1_b.0', (384,)), ('ln1_b.1', (384,)), ('ln1_b.2', (384,)), ('ln1_b.3', (384,)), ('ln1_w.0', (384,)), ('ln1_w.1', (384,)), ('ln1_w.2', (384,)), ('ln1_w.3', (384,)), ('ln2_b.0', (384,)), ('ln2_b.1', (384,)), ('ln2_b.2', (384,)), ('ln2_b.3', (384,)), ('ln2_w.0', (384,)), ('ln2_w.1', (384,)), ('ln2_w.2', (384,)), ('ln2_w.3', (384,))]
LAYER 0
ln1_w min/mean/max/std/norm: [-0.001 0.2403 1.0341 0.2436 6.7051]
ln1_b min/mean/max/std/norm: [-0.0959 0.0008 0.091 0.0287 0.5635]
ln2_w min/mean/max/std/norm: [ 0.0044 0.8006 1.5072 0.3295 16.9648]
ln2_b min/mean/max/std/norm: [-0.4986 0.0022 0.3331 0.0683 1.3392]
W_q norm/topSV/r50/r90: 32.2912 8.2519 9 30
W_k norm/topSV/r50/r90: 30.3909 7.6464 10 35
W_v norm/topSV/r50/r90: 14.836 4.339 14 97
W_o norm/topSV/r50/r90: 16.3713 7.1033 7 49
W_fc norm/topSV/r50/r90: 46.2485 18.1759 16 79
W_proj norm/topSV/r50/r90: 39.3073 9.8257 27 99
LAYER 1
ln1_w min/mean/max/std/norm: [0.1361 0.4575 1.1239 0.1022 9.1868]
ln1_b min/mean/max/std/norm: [-1.4713 -0.005 1.1544 0.1672 3.2771]
ln2_w min/mean/max/std/norm: [-0.0019 1.0429 1.4856 0.2003 20.8093]
ln2_b min/mean/max/std/norm: [-0.3597 0.0005 0.3686 0.1364 2.6732]
W_q norm/topSV/r50/r90: 32.3126 9.9032 15 54
W_k norm/topSV/r50/r90: 29.2506 13.3072 8 53
W_v norm/topSV/r50/r90: 27.3529 5.2845 22 76
W_o norm/topSV/r50/r90: 25.9233 4.9657 42 113
W_fc norm/topSV/r50/r90: 79.1264 21.4881 24 98
W_proj norm/topSV/r50/r90: 75.7102 8.1935 105 279
LAYER 2
ln1_w min/mean/max/std/norm: [ 0.2078 0.7052 1.0144 0.0749 13.8969]
ln1_b min/mean/max/std/norm: [-1.3819 -0.001 1.0527 0.1525 2.9893]
ln2_w min/mean/max/std/norm: [ 0.0011 1.3123 1.4655 0.1255 25.8329]
ln2_b min/mean/max/std/norm: [-0.3761 0.005 0.3886 0.1537 3.0128]
W_q norm/topSV/r50/r90: 39.3455 5.7962 53 169
W_k norm/topSV/r50/r90: 37.7688 10.6627 46 164
W_v norm/topSV/r50/r90: 33.3048 3.7062 62 179
W_o norm/topSV/r50/r90: 32.8397 4.1025 67 189
W_fc norm/topSV/r50/r90: 78.9667 15.3634 82 256
W_proj norm/topSV/r50/r90: 76.831 9.4036 87 266
LAYER 3
ln1_w min/mean/max/std/norm: [ 0.2944 0.7546 0.8701 0.0542 14.8247]
ln1_b min/mean/max/std/norm: [-0.6069 0.0121 0.6502 0.0849 1.6801]
ln2_w min/mean/max/std/norm: [ 0.4364 1.2778 1.4306 0.0808 25.0891]
ln2_b min/mean/max/std/norm: [-0.6728 0.0043 0.3068 0.1276 2.5016]
W_q norm/topSV/r50/r90: 32.9482 4.5598 56 176
W_k norm/topSV/r50/r90: 32.2094 6.0903 61 184
W_v norm/topSV/r50/r90: 39.7941 5.0598 52 170
W_o norm/topSV/r50/r90: 40.4549 9.5668 41 157
W_fc norm/topSV/r50/r90: 77.7757 14.1624 97 277
W_proj norm/topSV/r50/r90: 82.395 26.3094 9 24
HEAD PRODUCTS
QK=Q.T@K/sqrt(64); OV=O@V.
sym=<A,A.T>/||A||F^2; trN=trace(A)/nuclear_norm(A).
Columns: head; QK(norm,topSV,r50,r90,sym,trN); OV(norm,topSV,r50,r90,sym,trN); leading nonzero eigenvalues by magnitude.
L0H0 [np.float64(0.381), np.float64(0.3137), 1, 12, np.float64(0.0561), np.float64(-0.1312)] [np.float64(12.9041), np.float64(10.2524), 1, 4, np.float64(0.0078), np.float64(-0.0051)] QKev [-0.068+0.j -0.041+0.008j -0.041-0.008j -0.016+0.j -0.003+0.01j ] OVev [ 1.173+0.j -0.432+0.575j -0.432-0.575j -0.38 +0.j 0.274+0.244j]
L0H1 [np.float64(0.2297), np.float64(0.1371), 2, 27, np.float64(0.1165), np.float64(-0.1169)] [np.float64(10.1485), np.float64(7.662), 1, 4, np.float64(0.0184), np.float64(-0.0264)] QKev [-0.07 +0.j -0.034+0.j -0.009+0.01j -0.009-0.01j 0.012+0.j ] OVev [ 0.981+0.j -0.924+0.j -0.229+0.315j -0.229-0.315j 0.388+0.j ]
L0H2 [np.float64(22.8044), np.float64(7.4848), 6, 14, np.float64(-0.0304), np.float64(0.0382)] [np.float64(13.223), np.float64(4.0009), 7, 16, np.float64(0.0098), np.float64(-0.069)] QKev [-0.867+5.787j -0.867-5.787j 1.92 +4.997j 1.92 -4.997j 4.415+2.676j] OVev [-0.698+0.122j -0.698-0.122j -0.473+0.j -0.32 +0.318j -0.32 -0.318j]
L0H3 [np.float64(1.4189), np.float64(0.821), 2, 9, np.float64(0.1641), np.float64(-0.169)] [np.float64(7.3079), np.float64(4.305), 2, 5, np.float64(0.0159), np.float64(-0.0816)] QKev [ 0.304+0.j -0.211+0.064j -0.211-0.064j 0.194+0.048j 0.194-0.048j] OVev [-0.691+0.j -0.332+0.027j -0.332-0.027j 0.262+0.j -0.204+0.j ]
L0H4 [np.float64(0.8259), np.float64(0.7527), 1, 2, np.float64(0.1641), np.float64(-0.2512)] [np.float64(7.3117), np.float64(4.8786), 2, 6, np.float64(0.0082), np.float64(-0.0381)] QKev [-0.309+0.j -0.103+0.j -0.07 +0.j -0.021+0.j -0.014+0.009j] OVev [-0.467+0.j 0.36 +0.j -0.264+0.j -0.221+0.j 0.107+0.141j]
L0H5 [np.float64(0.3223), np.float64(0.2737), 1, 15, np.float64(0.0624), np.float64(-0.0847)] [np.float64(13.0384), np.float64(10.3142), 1, 4, np.float64(0.0039), np.float64(-0.0023)] QKev [-0.079+0.j -0.016+0.j -0.012+0.j 0.006+0.008j 0.006-0.008j] OVev [ 0.827+0.j -0.41 +0.498j -0.41 -0.498j 0.423+0.j 0.104+0.189j]
L1H0 [np.float64(2.9582), np.float64(1.5768), 2, 10, np.float64(0.0806), np.float64(-0.2659)] [np.float64(21.7066), np.float64(8.0989), 5, 15, np.float64(0.0885), np.float64(-0.2035)] QKev [-0.383+0.j -0.356+0.105j -0.356-0.105j -0.24 +0.059j -0.24 -0.059j] OVev [-2.776+0.j -2.372+0.124j -2.372-0.124j -2.193+0.345j -2.193-0.345j]
L1H1 [np.float64(4.4391), np.float64(2.4561), 2, 11, np.float64(0.1022), np.float64(-0.3042)] [np.float64(22.226), np.float64(6.5833), 9, 22, np.float64(0.0682), np.float64(-0.2227)] QKev [-0.684+0.j -0.515+0.096j -0.515-0.096j -0.478+0.019j -0.478-0.019j] OVev [-2.052+0.j -1.972+0.206j -1.972-0.206j -1.509-0.28j -1.509+0.28j ]
L1H2 [np.float64(5.2226), np.float64(2.9227), 2, 11, np.float64(0.1095), np.float64(-0.3463)] [np.float64(19.5165), np.float64(6.1346), 8, 19, np.float64(0.0494), np.float64(-0.2067)] QKev [-0.769+0.j -0.621+0.287j -0.621-0.287j -0.656+0.j -0.608+0.j ] OVev [-1.708+0.229j -1.708-0.229j -1.42 +0.326j -1.42 -0.326j -1.325+0.j ]
L1H3 [np.float64(4.6046), np.float64(2.6384), 2, 11, np.float64(0.0956), np.float64(-0.2981)] [np.float64(21.4133), np.float64(7.7675), 6, 15, np.float64(0.0452), np.float64(-0.1952)] QKev [-0.822+0.j -0.576+0.j -0.488+0.012j -0.488-0.012j -0.405+0.085j] OVev [-1.88 +0.j -1.706+0.128j -1.706-0.128j -1.526+0.j -1.438+0.j ]
L1H4 [np.float64(4.6727), np.float64(2.8698), 2, 11, np.float64(0.0851), np.float64(-0.3096)] [np.float64(22.0658), np.float64(6.4984), 8, 18, np.float64(0.0675), np.float64(-0.228)] QKev [-0.566+0.06j -0.566-0.06j -0.55 +0.j -0.451+0.174j -0.451-0.174j] OVev [-2.065+0.j -2.045+0.265j -2.045-0.265j -1.945+0.j -1.467+0.482j]
L1H5 [np.float64(4.5264), np.float64(2.6178), 2, 11, np.float64(0.0644), np.float64(-0.2713)] [np.float64(22.4638), np.float64(6.2081), 9, 20, np.float64(0.0683), np.float64(-0.2384)] QKev [-0.52 +0.114j -0.52 -0.114j -0.524+0.j -0.377+0.287j -0.377-0.287j] OVev [-2.115+0.j -2.009+0.j -1.932+0.409j -1.932-0.409j -1.638+0.108j]
L2H0 [np.float64(4.8218), np.float64(1.4573), 11, 33, np.float64(-0.0144), np.float64(0.0075)] [np.float64(22.8142), np.float64(4.8981), 15, 41, np.float64(0.1187), np.float64(-0.3236)] QKev [0.111+0.4j 0.111-0.4j 0.118+0.375j 0.118-0.375j 0.237+0.157j] OVev [-2.112+0.j -2.02 +0.513j -2.02 -0.513j -2.002+0.082j -2.002-0.082j]
L2H1 [np.float64(4.1385), np.float64(1.1546), 13, 39, np.float64(0.0163), np.float64(0.0398)] [np.float64(24.8446), np.float64(5.4684), 16, 42, np.float64(0.0796), np.float64(-0.2716)] QKev [ 0.264+0.j 0.233+0.087j 0.233-0.087j 0.245+0.j -0.167+0.139j] OVev [-2.182+0.269j -2.182-0.269j -1.599+1.285j -1.599-1.285j -1.913+0.j ]
L2H2 [np.float64(3.1536), np.float64(1.11), 11, 39, np.float64(0.0121), np.float64(-0.0914)] [np.float64(28.7074), np.float64(7.8628), 12, 38, np.float64(0.0723), np.float64(-0.2283)] QKev [ 0.031+0.213j 0.031-0.213j 0.174+0.113j 0.174-0.113j -0.196+0.j ] OVev [-2.963+0.j -2.564+0.636j -2.564-0.636j -2.556+0.j -2.024+0.j ]
L2H3 [np.float64(4.1027), np.float64(1.0661), 13, 39, np.float64(-0.0033), np.float64(-0.0502)] [np.float64(24.5055), np.float64(6.1803), 14, 41, np.float64(0.0956), np.float64(-0.2784)] QKev [ 0.11 +0.404j 0.11 -0.404j -0.234+0.j -0.222+0.061j -0.222-0.061j] OVev [-2.719+0.j -2.27 +0.499j -2.27 -0.499j -1.906+0.032j -1.906-0.032j]
L2H4 [np.float64(4.6034), np.float64(1.1879), 13, 37, np.float64(-0.0074), np.float64(-0.0285)] [np.float64(23.2786), np.float64(5.0699), 16, 41, np.float64(0.08), np.float64(-0.2651)] QKev [ 0.105+0.409j 0.105-0.409j -0.253+0.j 0.161+0.192j 0.161-0.192j] OVev [-1.945+0.j -1.822+0.26j -1.822-0.26j -1.585+0.715j -1.585-0.715j]
L2H5 [np.float64(4.4193), np.float64(1.2007), 12, 36, np.float64(-0.008), np.float64(-0.0415)] [np.float64(23.3416), np.float64(4.8404), 16, 42, np.float64(0.0994), np.float64(-0.2923)] QKev [ 0.083+0.407j 0.083-0.407j -0.274+0.j -0.158+0.216j -0.158-0.216j] OVev [-2.304+0.j -1.972+0.j -1.941+0.17j -1.941-0.17j -1.757+0.454j]
L3H0 [np.float64(3.9054), np.float64(0.7905), 18, 46, np.float64(0.0512), np.float64(0.2014)] [np.float64(29.6831), np.float64(8.3838), 13, 41, np.float64(0.1181), np.float64(-0.3382)] QKev [0.321+0.j 0.289+0.028j 0.289-0.028j 0.243+0.137j 0.243-0.137j] OVev [-3.166+0.235j -3.166-0.235j -2.676+0.j -2.512+0.j -2.339+0.41j ]
L3H1 [np.float64(3.2236), np.float64(0.5714), 21, 50, np.float64(0.2114), np.float64(0.4453)] [np.float64(53.3976), np.float64(13.8537), 10, 23, np.float64(0.0622), np.float64(-0.2166)] QKev [0.301+0.13j 0.301-0.13j 0.309-0.096j 0.309+0.096j 0.304+0.j ] OVev [-5.342+0.j -4.693+0.548j -4.693-0.548j -3.938+0.j -3.693+0.482j]
L3H2 [np.float64(2.6302), np.float64(0.808), 12, 42, np.float64(-0.0016), np.float64(0.0211)] [np.float64(35.2432), np.float64(11.8908), 9, 33, np.float64(0.0488), np.float64(-0.1901)] QKev [ 0.053+0.216j 0.053-0.216j 0.148+0.075j 0.148-0.075j -0.063+0.142j] OVev [-3.171+0.559j -3.171-0.559j -2.821+0.468j -2.821-0.468j -2.168+1.06j ]
L3H3 [np.float64(2.7937), np.float64(0.8563), 14, 45, np.float64(-0.0148), np.float64(0.0092)] [np.float64(35.9947), np.float64(12.4344), 9, 37, np.float64(0.0483), np.float64(-0.2096)] QKev [ 0.041+0.26j 0.041-0.26j -0.036+0.163j -0.036-0.163j 0.155+0.024j] OVev [-3.108+0.j -2.921+1.018j -2.921-1.018j -2.591+0.j -2.092+1.046j]
L3H4 [np.float64(3.2493), np.float64(0.919), 15, 45, np.float64(0.0008), np.float64(0.0666)] [np.float64(31.7775), np.float64(9.5258), 12, 39, np.float64(0.0392), np.float64(-0.1946)] QKev [0.115+0.169j 0.115-0.169j 0.175+0.059j 0.175-0.059j 0.18 +0.j ] OVev [-2.931+0.j -2.449+0.j -2.121+0.j -1.369+1.511j -1.369-1.511j]
L3H5 [np.float64(1.4813), np.float64(0.4831), 10, 40, np.float64(0.0244), np.float64(0.0166)] [np.float64(59.2935), np.float64(21.9791), 6, 18, np.float64(0.0647), np.float64(-0.1821)] QKev [0.135+0.058j 0.135-0.058j 0.134+0.j 0.042+0.087j 0.042-0.087j] OVev [-8.508+0.j -6.584+0.j -5.094+0.838j -5.094-0.838j -3.835+1.273j]
Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 1 of at most 100:
```
WEIGHT-ONLY ANALYSIS: no inputs, activations, or forward passes.
Keys and shapes: [('W_fc.0', (1536, 384)), ('W_fc.1', (1536, 384)), ('W_fc.2', (1536, 384)), ('W_fc.3', (1536, 384)), ('W_k.0', (384, 384)), ('W_k.1', (384, 384)), ('W_k.2', (384, 384)), ('W_k.3', (384, 384)), ('W_o.0', (384, 384)), ('W_o.1', (384, 384)), ('W_o.2', (384, 384)), ('W_o.3', (384, 384)), ('W_proj.0', (384, 1536)), ('W_proj.1', (384, 1536)), ('W_proj.2', (384, 1536)), ('W_proj.3', (384, 1536)), ('W_q.0', (384, 384)), ('W_q.1', (384, 384)), ('W_q.2', (384, 384)), ('W_q.3', (384, 384)), ('W_v.0', (384, 384)), ('W_v.1', (384, 384)), ('W_v.2', (384, 384)), ('W_v.3', (384, 384)), ('ln1_b.0', (384,)), ('ln1_b.1', (384,)), ('ln1_b.2', (384,)), ('ln1_b.3', (384,)), ('ln1_w.0', (384,)), ('ln1_w.1', (384,)), ('ln1_w.2', (384,)), ('ln1_w.3', (384,)), ('ln2_b.0', (384,)), ('ln2_b.1', (384,)), ('ln2_b.2', (384,)), ('ln2_b.3', (384,)), ('ln2_w.0', (384,)), ('ln2_w.1', (384,)), ('ln2_w.2', (384,)), ('ln2_w.3', (384,))]
LAYER 0
ln1_w min/mean/max/std/norm: [-0.001 0.2403 1.0341 0.2436 6.7051]
ln1_b min/mean/max/std/norm: [-0.0959 0.0008 0.091 0.0287 0.5635]
ln2_w min/mean/max/std/norm: [ 0.0044 0.8006 1.5072 0.3295 16.9648]
ln2_b min/mean/max/std/norm: [-0.4986 0.0022 0.3331 0.0683 1.3392]
W_q norm/topSV/r50/r90: 32.2912 8.2519 9 30
W_k norm/topSV/r50/r90: 30.3909 7.6464 10 35
W_v norm/topSV/r50/r90: 14.836 4.339 14 97
W_o norm/topSV/r50/r90: 16.3713 7.1033 7 49
W_fc norm/topSV/r50/r90: 46.2485 18.1759 16 79
W_proj norm/topSV/r50/r90: 39.3073 9.8257 27 99
LAYER 1
ln1_w min/mean/max/std/norm: [0.1361 0.4575 1.1239 0.1022 9.1868]
ln1_b min/mean/max/std/norm: [-1.4713 -0.005 1.1544 0.1672 3.2771]
ln2_w min/mean/max/std/norm: [-0.0019 1.0429 1.4856 0.2003 20.8093]
ln2_b min/mean/max/std/norm: [-0.3597 0.0005 0.3686 0.1364 2.6732]
W_q norm/topSV/r50/r90: 32.3126 9.9032 15 54
W_k norm/topSV/r50/r90: 29.2506 13.3072 8 53
W_v norm/topSV/r50/r90: 27.3529 5.2845 22 76
W_o norm/topSV/r50/r90: 25.9233 4.9657 42 113
W_fc norm/topSV/r50/r90: 79.1264 21.4881 24 98
W_proj norm/topSV/r50/r90: 75.7102 8.1935 105 279
LAYER 2
ln1_w min/mean/max/std/norm: [ 0.2078 0.7052 1.0144 0.0749 13.8969]
ln1_b min/mean/max/std/norm: [-1.3819 -0.001 1.0527 0.1525 2.9893]
ln2_w min/mean/max/std/norm: [ 0.0011 1.3123 1.4655 0.1255 25.8329]
ln2_b min/mean/max/std/norm: [-0.3761 0.005 0.3886 0.1537 3.0128]
W_q norm/topSV/r50/r90: 39.3455 5.7962 53 169
W_k norm/topSV/r50/r90: 37.7688 10.6627 46 164
W_v norm/topSV/r50/r90: 33.3048 3.7062 62 179
W_o norm/topSV/r50/r90: 32.8397 4.1025 67 189
W_fc norm/topSV/r50/r90: 78.9667 15.3634 82 256
W_proj norm/topSV/r50/r90: 76.831 9.4036 87 266
LAYER 3
ln1_w min/mean/max/std/norm: [ 0.2944 0.7546 0.8701 0.0542 14.8247]
ln1_b min/mean/max/std/norm: [-0.6069 0.0121 0.6502 0.0849 1.6801]
ln2_w min/mean/max/std/norm: [ 0.4364 1.2778 1.4306 0.0808 25.0891]
ln2_b min/mean/max/std/norm: [-0.6728 0.0043 0.3068 0.1276 2.5016]
W_q norm/topSV/r50/r90: 32.9482 4.5598 56 176
W_k norm/topSV/r50/r90: 32.2094 6.0903 61 184
W_v norm/topSV/r50/r90: 39.7941 5.0598 52 170
W_o norm/topSV/r50/r90: 40.4549 9.5668 41 157
W_fc norm/topSV/r50/r90: 77.7757 14.1624 97 277
W_proj norm/topSV/r50/r90: 82.395 26.3094 9 24
HEAD PRODUCTS
QK=Q.T@K/sqrt(64); OV=O@V.
sym=<A,A.T>/||A||F^2; trN=trace(A)/nuclear_norm(A).
Columns: head; QK(norm,topSV,r50,r90,sym,trN); OV(norm,topSV,r50,r90,sym,trN); leading nonzero eigenvalues by magnitude.
L0H0 [np.float64(0.381), np.float64(0.3137), 1, 12, np.float64(0.0561), np.float64(-0.1312)] [np.float64(12.9041), np.float64(10.2524), 1, 4, np.float64(0.0078), np.float64(-0.0051)] QKev [-0.068+0.j -0.041+0.008j -0.041-0.008j -0.016+0.j -0.003+0.01j ] OVev [ 1.173+0.j -0.432+0.575j -0.432-0.575j -0.38 +0.j 0.274+0.244j]
L0H1 [np.float64(0.2297), np.float64(0.1371), 2, 27, np.float64(0.1165), np.float64(-0.1169)] [np.float64(10.1485), np.float64(7.662), 1, 4, np.float64(0.0184), np.float64(-0.0264)] QKev [-0.07 +0.j -0.034+0.j -0.009+0.01j -0.009-0.01j 0.012+0.j ] OVev [ 0.981+0.j -0.924+0.j -0.229+0.315j -0.229-0.315j 0.388+0.j ]
L0H2 [np.float64(22.8044), np.float64(7.4848), 6, 14, np.float64(-0.0304), np.float64(0.0382)] [np.float64(13.223), np.float64(4.0009), 7, 16, np.float64(0.0098), np.float64(-0.069)] QKev [-0.867+5.787j -0.867-5.787j 1.92 +4.997j 1.92 -4.997j 4.415+2.676j] OVev [-0.698+0.122j -0.698-0.122j -0.473+0.j -0.32 +0.318j -0.32 -0.318j]
L0H3 [np.float64(1.4189), np.float64(0.821), 2, 9, np.float64(0.1641), np.float64(-0.169)] [np.float64(7.3079), np.float64(4.305), 2, 5, np.float64(0.0159), np.float64(-0.0816)] QKev [ 0.304+0.j -0.211+0.064j -0.211-0.064j 0.194+0.048j 0.194-0.048j] OVev [-0.691+0.j -0.332+0.027j -0.332-0.027j 0.262+0.j -0.204+0.j ]
L0H4 [np.float64(0.8259), np.float64(0.7527), 1, 2, np.float64(0.1641), np.float64(-0.2512)] [np.float64(7.3117), np.float64(4.8786), 2, 6, np.float64(0.0082), np.float64(-0.0381)] QKev [-0.309+0.j -0.103+0.j -0.07 +0.j -0.021+0.j -0.014+0.009j] OVev [-0.467+0.j 0.36 +0.j -0.264+0.j -0.221+0.j 0.107+0.141j]
L0H5 [np.float64(0.3223), np.float64(0.2737), 1, 15, np.float64(0.0624), np.float64(-0.0847)] [np.float64(13.0384), np.float64(10.3142), 1, 4, np.float64(0.0039), np.float64(-0.0023)] QKev [-0.079+0.j -0.016+0.j -0.012+0.j 0.006+0.008j 0.006-0.008j] OVev [ 0.827+0.j -0.41 +0.498j -0.41 -0.498j 0.423+0.j 0.104+0.189j]
L1H0 [np.float64(2.9582), np.float64(1.5768), 2, 10, np.float64(0.0806), np.float64(-0.2659)] [np.float64(21.7066), np.float64(8.0989), 5, 15, np.float64(0.0885), np.float64(-0.2035)] QKev [-0.383+0.j -0.356+0.105j -0.356-0.105j -0.24 +0.059j -0.24 -0.059j] OVev [-2.776+0.j -2.372+0.124j -2.372-0.124j -2.193+0.345j -2.193-0.345j]
L1H1 [np.float64(4.4391), np.float64(2.4561), 2, 11, np.float64(0.1022), np.float64(-0.3042)] [np.float64(22.226), np.float64(6.5833), 9, 22, np.float64(0.0682), np.float64(-0.2227)] QKev [-0.684+0.j -0.515+0.096j -0.515-0.096j -0.478+0.019j -0.478-0.019j] OVev [-2.052+0.j -1.972+0.206j -1.972-0.206j -1.509-0.28j -1.509+0.28j ]
L1H2 [np.float64(5.2226), np.float64(2.9227), 2, 11, np.float64(0.1095), np.float64(-0.3463)] [np.float64(19.5165), np.float64(6.1346), 8, 19, np.float64(0.0494), np.float64(-0.2067)] QKev [-0.769+0.j -0.621+0.287j -0.621-0.287j -0.656+0.j -0.608+0.j ] OVev [-1.708+0.229j -1.708-0.229j -1.42 +0.326j -1.42 -0.326j -1.325+0.j ]
L1H3 [np.float64(4.6046), np.float64(2.6384), 2, 11, np.float64(0.0956), np.float64(-0.2981)] [np.float64(21.4133), np.float64(7.7675), 6, 15, np.float64(0.0452), np.float64(-0.1952)] QKev [-0.822+0.j -0.576+0.j -0.488+0.012j -0.488-0.012j -0.405+0.085j] OVev [-1.88 +0.j -1.706+0.128j -1.706-0.128j -1.526+0.j -1.438+0.j ]
L1H4 [np.float64(4.6727), np.float64(2.8698), 2, 11, np.float64(0.0851), np.float64(-0.3096)] [np.float64(22.0658), np.float64(6.4984), 8, 18, np.float64(0.0675), np.float64(-0.228)] QKev [-0.566+0.06j -0.566-0.06j -0.55 +0.j -0.451+0.174j -0.451-0.174j] OVev [-2.065+0.j -2.045+0.265j -2.045-0.265j -1.945+0.j -1.467+0.482j]
L1H5 [np.float64(4.5264), np.float64(2.6178), 2, 11, np.float64(0.0644), np.float64(-0.2713)] [np.float64(22.4638), np.float64(6.2081), 9, 20, np.float64(0.0683), np.float64(-0.2384)] QKev [-0.52 +0.114j -0.52 -0.114j -0.524+0.j -0.377+0.287j -0.377-0.287j] OVev [-2.115+0.j -2.009+0.j -1.932+0.409j -1.932-0.409j -1.638+0.108j]
L2H0 [np.float64(4.8218), np.float64(1.4573), 11, 33, np.float64(-0.0144), np.float64(0.0075)] [np.float64(22.8142), np.float64(4.8981), 15, 41, np.float64(0.1187), np.float64(-0.3236)] QKev [0.111+0.4j 0.111-0.4j 0.118+0.375j 0.118-0.375j 0.237+0.157j] OVev [-2.112+0.j -2.02 +0.513j -2.02 -0.513j -2.002+0.082j -2.002-0.082j]
L2H1 [np.float64(4.1385), np.float64(1.1546), 13, 39, np.float64(0.0163), np.float64(0.0398)] [np.float64(24.8446), np.float64(5.4684), 16, 42, np.float64(0.0796), np.float64(-0.2716)] QKev [ 0.264+0.j 0.233+0.087j 0.233-0.087j 0.245+0.j -0.167+0.139j] OVev [-2.182+0.269j -2.182-0.269j -1.599+1.285j -1.599-1.285j -1.913+0.j ]
L2H2 [np.float64(3.1536), np.float64(1.11), 11, 39, np.float64(0.0121), np.float64(-0.0914)] [np.float64(28.7074), np.float64(7.8628), 12, 38, np.float64(0.0723), np.float64(-0.2283)] QKev [ 0.031+0.213j 0.031-0.213j 0.174+0.113j 0.174-0.113j -0.196+0.j ] OVev [-2.963+0.j -2.564+0.636j -2.564-0.636j -2.556+0.j -2.024+0.j ]
L2H3 [np.float64(4.1027), np.float64(1.0661), 13, 39, np.float64(-0.0033), np.float64(-0.0502)] [np.float64(24.5055), np.float64(6.1803), 14, 41, np.float64(0.0956), np.float64(-0.2784)] QKev [ 0.11 +0.404j 0.11 -0.404j -0.234+0.j -0.222+0.061j -0.222-0.061j] OVev [-2.719+0.j -2.27 +0.499j -2.27 -0.499j -1.906+0.032j -1.906-0.032j]
L2H4 [np.float64(4.6034), np.float64(1.1879), 13, 37, np.float64(-0.0074), np.float64(-0.0285)] [np.float64(23.2786), np.float64(5.0699), 16, 41, np.float64(0.08), np.float64(-0.2651)] QKev [ 0.105+0.409j 0.105-0.409j -0.253+0.j 0.161+0.192j 0.161-0.192j] OVev [-1.945+0.j -1.822+0.26j -1.822-0.26j -1.585+0.715j -1.585-0.715j]
L2H5 [np.float64(4.4193), np.float64(1.2007), 12, 36, np.float64(-0.008), np.float64(-0.0415)] [np.float64(23.3416), np.float64(4.8404), 16, 42, np.float64(0.0994), np.float64(-0.2923)] QKev [ 0.083+0.407j 0.083-0.407j -0.274+0.j -0.158+0.216j -0.158-0.216j] OVev [-2.304+0.j -1.972+0.j -1.941+0.17j -1.941-0.17j -1.757+0.454j]
L3H0 [np.float64(3.9054), np.float64(0.7905), 18, 46, np.float64(0.0512), np.float64(0.2014)] [np.float64(29.6831), np.float64(8.3838), 13, 41, np.float64(0.1181), np.float64(-0.3382)] QKev [0.321+0.j 0.289+0.028j 0.289-0.028j 0.243+0.137j 0.243-0.137j] OVev [-3.166+0.235j -3.166-0.235j -2.676+0.j -2.512+0.j -2.339+0.41j ]
L3H1 [np.float64(3.2236), np.float64(0.5714), 21, 50, np.float64(0.2114), np.float64(0.4453)] [np.float64(53.3976), np.float64(13.8537), 10, 23, np.float64(0.0622), np.float64(-0.2166)] QKev [0.301+0.13j 0.301-0.13j 0.309-0.096j 0.309+0.096j 0.304+0.j ] OVev [-5.342+0.j -4.693+0.548j -4.693-0.548j -3.938+0.j -3.693+0.482j]
L3H2 [np.float64(2.6302), np.float64(0.808), 12, 42, np.float64(-0.0016), np.float64(0.0211)] [np.float64(35.2432), np.float64(11.8908), 9, 33, np.float64(0.0488), np.float64(-0.1901)] QKev [ 0.053+0.216j 0.053-0.216j 0.148+0.075j 0.148-0.075j -0.063+0.142j] OVev [-3.171+0.559j -3.171-0.559j -2.821+0.468j -2.821-0.468j -2.168+1.06j ]
L3H3 [np.float64(2.7937), np.float64(0.8563), 14, 45, np.float64(-0.0148), np.float64(0.0092)] [np.float64(35.9947), np.float64(12.4344), 9, 37, np.float64(0.0483), np.float64(-0.2096)] QKev [ 0.041+0.26j 0.041-0.26j -0.036+0.163j -0.036-0.163j 0.155+0.024j] OVev [-3.108+0.j -2.921+1.018j -2.921-1.018j -2.591+0.j -2.092+1.046j]
L3H4 [np.float64(3.2493), np.float64(0.919), 15, 45, np.float64(0.0008), np.float64(0.0666)] [np.float64(31.7775), np.float64(9.5258), 12, 39, np.float64(0.0392), np.float64(-0.1946)] QKev [0.115+0.169j 0.115-0.169j 0.175+0.059j 0.175-0.059j 0.18 +0.j ] OVev [-2.931+0.j -2.449+0.j -2.121+0.j -1.369+1.511j -1.369-1.511j]
L3H5 [np.float64(1.4813), np.float64(0.4831), 10, 40, np.float64(0.0244), np.float64(0.0166)] [np.float64(59.2935), np.float64(21.9791), 6, 18, np.float64(0.0647), np.float64(-0.1821)] QKev [0.135+0.058j 0.135-0.058j 0.134+0.j 0.042+0.087j 0.042-0.087j] OVev [-8.508+0.j -6.584+0.j -5.094+0.838j -5.094-0.838j -3.835+1.273j]
```Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(2)
np.set_printoptions(precision=4, suppress=True, linewidth=180)
d = 384
P = np.eye(d) - np.ones((d, d))/d
A, B = {}, {}
print("All calculations below are matrix products or decompositions, not layer evaluations.")
for l in range(4):
g = W[f"ln1_w.{l}"].astype(float)
b = W[f"ln1_b.{l}"].astype(float)
print("\nLAYER", l)
for h in range(6):
sl = slice(64*h, 64*(h+1))
q, k, v = [W[f"W_{name}.{l}"][sl].astype(float) for name in ["q", "k", "v"]]
o = W[f"W_o.{l}"][:, sl].astype(float)
qg, kg, vg = [(m*g) @ P for m in [q, k, v]]
a = qg.T @ kg / 8
bb = P @ o @ vg
A[l,h], B[l,h] = a, bb
s = np.linalg.svd(a, compute_uv=False)
ev = np.linalg.eigvals(kg @ qg.T / 8)
ev = ev[np.argsort(-abs(ev))]
key_bias = kg.T @ (q @ b) / 8
print(f"L{l}H{h}",
"Q/K/V/O norms", np.round([np.linalg.norm(m) for m in [q,k,v,o]],3),
"effective QK norm/top/sym/trace",
np.round([np.linalg.norm(a),s[0],np.sum(a*a.T)/np.sum(a*a),np.trace(a)],4),
"query_bias_ratio", round(np.linalg.norm(q@b)/np.linalg.norm(qg),4),
"key_bias_norm", round(np.linalg.norm(key_bias),4),
"QKev", np.round(ev[:8],3))
print("\nSINGULAR VALUES: first 45 for selected matrices")
for name in ["W_q.0","W_k.0","W_v.0","W_o.0","W_fc.0","W_proj.0",
"W_proj.1","W_proj.2","W_proj.3"]:
s = np.linalg.svd(W[name].astype(float),compute_uv=False)
print(name, np.round(s[:45],3))
# Positional candidate subspace: the dominant left/right singular spaces
# of L0H2's QK product. This is not an embedding or a constructed input.
u,s,vt = np.linalg.svd(A[0,2],full_matrices=False)
joined = np.concatenate([u[:,:16],vt[:16].T],axis=1)
up,sp,_ = np.linalg.svd(joined,full_matrices=False)
print("\nL0H2 left/right-space joint singular values",np.round(sp,4))
for r in [16,24,32]:
T=up[:,:r]
print("Candidate position-space dimension",r)
for l in range(4):
vals=[]
for h in range(6):
a=A[l,h]
vals.append([np.linalg.norm(T.T@a)**2/np.linalg.norm(a)**2,
np.linalg.norm(a@T)**2/np.linalg.norm(a)**2,
np.linalg.norm(T.T@a@T)**2/np.linalg.norm(a)**2])
print("layer",l,"Q/K/both energy:",np.round(vals,3))
print("\nKEY-COMPOSITION CANDIDATES: QK_l @ OV_earlier")
print("For each head: strongest positive trace/Frobenius scores; "
"entries=(earlier head, trace/Fnorm, transpose symmetry, Fnorm).")
for l in range(1,4):
for h in range(6):
results=[]
for j in range(l):
for k in range(6):
c=A[l,h]@B[j,k]
fn=np.linalg.norm(c)
results.append((f"L{j}H{k}",np.trace(c)/fn,np.sum(c*c.T)/(fn*fn),fn))
results.sort(key=lambda t:t[1],reverse=True)
print(f"L{l}H{h}",[(n,round(t,3),round(y,3),round(f,3)) for n,t,y,f in results[:5]])
print("\nPairwise cosine similarities of effective QK products, within each layer")
for l in range(4):
z=np.array([A[l,h].ravel()/np.linalg.norm(A[l,h]) for h in range(6)])
print("Layer",l,"\n",np.round(z@z.T,3))
```{
"run": 2,
"turn": "t002",
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(2)\nnp.set_printoptions(precision=4, suppress=True, linewidth=180)\n\nd = 384\nP = np.eye(d) - np.ones((d, d))/d\nA, B = {}, {}\nprint(\"All calculations below are matrix products or decompositions, not layer evaluations.\")\n\nfor l in range(4):\n g = W[f\"ln1_w.{l}\"].astype(float)\n b = W[f\"ln1_b.{l}\"].astype(float)\n print(\"\\nLAYER\", l)\n for h in range(6):\n sl = slice(64*h, 64*(h+1))\n q, k, v = [W[f\"W_{name}.{l}\"][sl].astype(float) for name in [\"q\", \"k\", \"v\"]]\n o = W[f\"W_o.{l}\"][:, sl].astype(float)\n qg, kg, vg = [(m*g) @ P for m in [q, k, v]]\n a = qg.T @ kg / 8\n bb = P @ o @ vg\n A[l,h], B[l,h] = a, bb\n s = np.linalg.svd(a, compute_uv=False)\n ev = np.linalg.eigvals(kg @ qg.T / 8)\n ev = ev[np.argsort(-abs(ev))]\n key_bias = kg.T @ (q @ b) / 8\n print(f\"L{l}H{h}\",\n \"Q/K/V/O norms\", np.round([np.linalg.norm(m) for m in [q,k,v,o]],3),\n \"effective QK norm/top/sym/trace\",\n np.round([np.linalg.norm(a),s[0],np.sum(a*a.T)/np.sum(a*a),np.trace(a)],4),\n \"query_bias_ratio\", round(np.linalg.norm(q@b)/np.linalg.norm(qg),4),\n \"key_bias_norm\", round(np.linalg.norm(key_bias),4),\n \"QKev\", np.round(ev[:8],3))\n\nprint(\"\\nSINGULAR VALUES: first 45 for selected matrices\")\nfor name in [\"W_q.0\",\"W_k.0\",\"W_v.0\",\"W_o.0\",\"W_fc.0\",\"W_proj.0\",\n \"W_proj.1\",\"W_proj.2\",\"W_proj.3\"]:\n s = np.linalg.svd(W[name].astype(float),compute_uv=False)\n print(name, np.round(s[:45],3))\n\n# Positional candidate subspace: the dominant left/right singular spaces\n# of L0H2's QK product. This is not an embedding or a constructed input.\nu,s,vt = np.linalg.svd(A[0,2],full_matrices=False)\njoined = np.concatenate([u[:,:16],vt[:16].T],axis=1)\nup,sp,_ = np.linalg.svd(joined,full_matrices=False)\nprint(\"\\nL0H2 left/right-space joint singular values\",np.round(sp,4))\nfor r in [16,24,32]:\n T=up[:,:r]\n print(\"Candidate position-space dimension\",r)\n for l in range(4):\n vals=[]\n for h in range(6):\n a=A[l,h]\n vals.append([np.linalg.norm(T.T@a)**2/np.linalg.norm(a)**2,\n np.linalg.norm(a@T)**2/np.linalg.norm(a)**2,\n np.linalg.norm(T.T@a@T)**2/np.linalg.norm(a)**2])\n print(\"layer\",l,\"Q/K/both energy:\",np.round(vals,3))\n\nprint(\"\\nKEY-COMPOSITION CANDIDATES: QK_l @ OV_earlier\")\nprint(\"For each head: strongest positive trace/Frobenius scores; \"\n \"entries=(earlier head, trace/Fnorm, transpose symmetry, Fnorm).\")\nfor l in range(1,4):\n for h in range(6):\n results=[]\n for j in range(l):\n for k in range(6):\n c=A[l,h]@B[j,k]\n fn=np.linalg.norm(c)\n results.append((f\"L{j}H{k}\",np.trace(c)/fn,np.sum(c*c.T)/(fn*fn),fn))\n results.sort(key=lambda t:t[1],reverse=True)\n print(f\"L{l}H{h}\",[(n,round(t,3),round(y,3),round(f,3)) for n,t,y,f in results[:5]])\n\nprint(\"\\nPairwise cosine similarities of effective QK products, within each layer\")\nfor l in range(4):\n z=np.array([A[l,h].ravel()/np.linalg.norm(A[l,h]) for h in range(6)])\n print(\"Layer\",l,\"\\n\",np.round(z@z.T,3))",
"out": "All calculations below are matrix products or decompositions, not layer evaluations.\n\nLAYER 0\nL0H0 Q/K/V/O norms [3.305 4.18 6.039 6.666] effective QK norm/top/sym/trace [ 0.0546 0.0514 0.0233 -0.0047] query_bias_ratio 0.214 key_bias_norm 0.0136 QKev [ 0.007+0.j -0.004+0.j -0.002+0.j -0. +0.001j -0. -0.001j 0. +0.001j 0. -0.001j -0.001+0.001j]\nL0H1 Q/K/V/O norms [3.09 3.658 5.382 6.025] effective QK norm/top/sym/trace [ 0.0295 0.025 0.0475 -0.014 ] query_bias_ratio 0.1243 key_bias_norm 0.0033 QKev [-0.004+0.001j -0.004-0.001j 0.002+0.j -0.001+0.001j -0.001-0.001j -0.001+0.j -0.001+0.001j -0.001-0.001j]\nL0H2 Q/K/V/O norms [30.818 28.3 7.765 8.915] effective QK norm/top/sym/trace [ 4.0688 1.4079 -0.0446 1.199 ] query_bias_ratio 0.0589 key_bias_norm 0.1428 QKev [ 0.632+0.813j 0.632-0.813j -0.231+0.927j -0.231-0.927j 0.332+0.871j 0.332-0.871j -0.777+0.334j -0.777-0.334j]\nL0H3 Q/K/V/O norms [6.821 7.243 5.434 5.029] effective QK norm/top/sym/trace [0.3945 0.3588 0.4782 0.1968] query_bias_ratio 0.1771 key_bias_norm 0.0535 QKev [ 0.262+0.j 0.078+0.j 0.01 +0.031j 0.01 -0.031j -0.017+0.005j -0.017-0.005j -0.017+0.j -0.015+0.003j]\nL0H4 Q/K/V/O norms [3.971 4.995 5.208 5.587] effective QK norm/top/sym/trace [ 0.1062 0.089 0.004 -0.0707] query_bias_ratio 0.1505 key_bias_norm 0.0112 QKev [-0.028+0.028j -0.028-0.028j -0.008+0.j -0.002+0.j -0.001+0.001j -0.001-0.001j -0.002+0.j 0.001+0.001j]\nL0H5 Q/K/V/O norms [3.19 3.802 6.143 7.162] effective QK norm/top/sym/trace [ 0.0775 0.0759 0.1603 -0.0347] query_bias_ratio 0.1849 key_bias_norm 0.0162 QKev [-0.031+0.j 0.001+0.002j 0.001-0.002j -0.001+0.j -0.001-0.j -0.001+0.001j -0.001-0.001j 0.001+0.001j]\n\nLAYER 1\nL1H0 Q/K/V/O norms [10.716 9.827 11.156 9.984] effective QK norm/top/sym/trace [ 0.8864 0.5969 0.0086 -0.6436] query_bias_ratio 0.7821 key_bias_norm 1.3572 QKev [ 0.028+0.106j 0.028-0.106j -0.078+0.j -0.066+0.028j -0.066-0.028j -0.055+0.007j -0.055-0.007j -0.047+0.j ]\nL1H1 Q/K/V/O norms [13.292 12.068 11.72 11.303] effective QK norm/top/sym/trace [ 1.3146 0.8811 0.0186 -1.1603] query_bias_ratio 0.8034 key_bias_norm 2.2848 QKev [ 0.07 +0.191j 0.07 -0.191j -0.153+0.j -0.107+0.024j -0.107-0.024j -0.108+0.006j -0.108-0.006j -0.086+0.j ]\nL1H2 Q/K/V/O norms [14.476 13.092 10.668 10.272] effective QK norm/top/sym/trace [ 1.5491 0.9943 0.0179 -1.6165] query_bias_ratio 0.6959 key_bias_norm 2.5879 QKev [ 0.097+0.246j 0.097-0.246j -0.168+0.j -0.141+0.064j -0.141-0.064j -0.135+0.009j -0.135-0.009j -0.104+0.03j ]\nL1H3 Q/K/V/O norms [13.358 12.153 10.904 9.928] effective QK norm/top/sym/trace [ 1.3742 0.9188 0.0172 -1.1551] query_bias_ratio 0.7616 key_bias_norm 2.3511 QKev [ 0.094+0.203j 0.094-0.203j -0.175+0.j -0.113+0.008j -0.113-0.008j -0.107+0.j -0.04 +0.091j -0.04 -0.091j]\nL1H4 Q/K/V/O norms [13.518 12.129 11.148 10.794] effective QK norm/top/sym/trace [ 1.4497 0.9476 0.0108 -1.2432] query_bias_ratio 0.711 key_bias_norm 2.2902 QKev [ 0.062+0.214j 0.062-0.214j -0.135+0.01j -0.135-0.01j -0.127+0.j -0.099+0.j -0.081+0.053j -0.081-0.053j]\nL1H5 Q/K/V/O norms [13.486 12.132 11.376 11.136] effective QK norm/top/sym/trace [ 1.3363 0.8925 -0.0027 -1.0127] query_bias_ratio 0.6733 key_bias_norm 1.7526 QKev [ 0.088+0.221j 0.088-0.221j -0.121+0.012j -0.121-0.012j -0.1 +0.004j -0.1 -0.004j -0.074+0.055j -0.074-0.055j]\n\nLAYER 2\nL2H0 Q/K/V/O norms [16.822 16.208 13.115 12.942] effective QK norm/top/sym/trace [ 2.4375 0.6943 -0.0335 -0.0216] query_bias_ratio 0.3088 key_bias_norm 1.6991 QKev [ 0.053+0.306j 0.053-0.306j 0.036+0.183j 0.036-0.183j 0.118+0.08j 0.118-0.08j -0.077+0.098j -0.077-0.098j]\nL2H1 Q/K/V/O norms [15.963 15.287 13.768 13.525] effective QK norm/top/sym/trace [2.0677 0.5267 0.0101 0.53 ] query_bias_ratio 0.2076 key_bias_norm 0.7446 QKev [ 0.027+0.144j 0.027-0.144j 0.126+0.011j 0.126-0.011j 0.109+0.037j 0.109-0.037j -0.084+0.071j -0.084-0.071j]\nL2H2 Q/K/V/O norms [13.97 13.239 14.567 14.24 ] effective QK norm/top/sym/trace [ 1.6127 0.6069 -0.0046 -1.1051] query_bias_ratio 0.2312 key_bias_norm 0.7554 QKev [ 0.037+0.175j 0.037-0.175j -0.106+0.j 0.067+0.055j 0.067-0.055j -0.084+0.015j -0.084-0.015j -0.076-0.014j]\nL2H3 Q/K/V/O norms [16.121 15.554 13.498 13.391] effective QK norm/top/sym/trace [ 2.0862 0.5616 -0.0268 -0.7372] query_bias_ratio 0.2623 key_bias_norm 1.4625 QKev [ 0.072+0.307j 0.072-0.307j -0.076+0.084j -0.076-0.084j -0.102+0.042j -0.102-0.042j -0.104+0.005j -0.104-0.005j]\nL2H4 Q/K/V/O norms [16.876 16.188 13.241 13.143] effective QK norm/top/sym/trace [ 2.3438 0.6278 -0.0248 -0.4747] query_bias_ratio 0.2685 key_bias_norm 1.5774 QKev [ 0.061+0.303j 0.061-0.303j 0.077+0.102j 0.077-0.102j -0.081+0.091j -0.081-0.091j -0.118+0.j -0.045+0.105j]\nL2H5 Q/K/V/O norms [16.445 15.839 13.339 13.161] effective QK norm/top/sym/trace [ 2.2503 0.6375 -0.0283 -0.6957] query_bias_ratio 0.2779 key_bias_norm 1.5086 QKev [ 0.046+0.302j 0.046-0.302j -0.141+0.j -0.071-0.108j -0.071+0.108j -0.119+0.037j -0.119-0.037j -0.068+0.099j]\n\nLAYER 3\nL3H0 Q/K/V/O norms [15.747 15.258 14.874 14.87 ] effective QK norm/top/sym/trace [2.2322 0.4514 0.0503 3.4122] query_bias_ratio 0.1057 key_bias_norm 0.2581 QKev [0.182+0.j 0.141+0.078j 0.141-0.078j 0.158+0.019j 0.158-0.019j 0.143+0.056j 0.143-0.056j 0.141+0.036j]\nL3H1 Q/K/V/O norms [14.22 14.197 17.885 18.331] effective QK norm/top/sym/trace [1.8295 0.3275 0.2114 6.3101] query_bias_ratio 0.1322 key_bias_norm 0.2408 QKev [0.174-0.06j 0.174+0.06j 0.171-0.067j 0.171+0.067j 0.149-0.082j 0.149+0.082j 0.164-0.031j 0.164+0.031j]\nL3H2 Q/K/V/O norms [12.743 12.446 15.523 15.585] effective QK norm/top/sym/trace [ 1.505 0.4702 -0.0036 0.2216] query_bias_ratio 0.124 key_bias_norm 0.2895 QKev [ 0.03 +0.134j 0.03 -0.134j 0.081+0.038j 0.081-0.038j -0.035+0.082j -0.035-0.082j 0.082+0.j -0.077+0.j ]\nL3H3 Q/K/V/O norms [13.283 13.057 15.732 16.118] effective QK norm/top/sym/trace [ 1.5985 0.4973 -0.0166 0.1116] query_bias_ratio 0.121 key_bias_norm 0.3603 QKev [ 0.023+0.159j 0.023-0.159j 0.099+0.j -0.023+0.095j -0.023-0.095j 0.076+0.036j 0.076-0.036j 0.078+0.j ]\nL3H4 Q/K/V/O norms [14.356 13.874 15.175 15.3 ] effective QK norm/top/sym/trace [ 1.8555 0.5287 -0.0005 0.9035] query_bias_ratio 0.1217 key_bias_norm 0.3072 QKev [0.068+0.092j 0.068-0.092j 0.002+0.108j 0.002-0.108j 0.098+0.04j 0.098-0.04j 0.102+0.j 0.07 +0.066j]\nL3H5 Q/K/V/O norms [ 9.514 9.231 17.993 18.512] effective QK norm/top/sym/trace [0.8339 0.2591 0.0242 0.1007] query_bias_ratio 0.1537 key_bias_norm 0.1645 QKev [0.075+0.035j 0.075-0.035j 0.078+0.j 0.02 +0.048j 0.02 -0.048j 0.023+0.036j 0.023-0.036j 0.038+0.011j]\n\nSINGULAR VALUES: first 45 for selected matrices\nW_q.0 [8.252 8.223 7.963 7.819 7.68 7.642 7.519 7.205 6.948 6.819 6.388 6.165 5.801 5.555 5.13 4.947 4.854 4.52 4.268 3.951 3.591 3.527 3.285 3.125 3.026 2.928 2.707 2.511 2.471\n 2.336 2.249 2.136 2.089 2.058 1.914 1.792 1.755 1.679 1.536 1.513 1.475 1.455 1.327 1.31 1.282]\nW_k.0 [7.646 7.436 7.33 6.912 6.829 6.672 6.472 6.446 6.393 6.118 6.009 5.943 5.629 5.505 4.99 4.901 4.698 4.293 4.066 4.018 3.708 3.529 3.343 3.068 2.987 2.874 2.696 2.586 2.412\n 2.262 2.216 2.11 2.032 1.876 1.744 1.733 1.697 1.666 1.547 1.514 1.441 1.402 1.386 1.334 1.29 ]\nW_v.0 [4.339 3.529 3.27 3.166 2.982 2.647 2.589 2.552 2.436 2.381 2.256 2.162 2.13 2.103 2.042 2.017 2.003 1.973 1.928 1.897 1.843 1.787 1.711 1.665 1.628 1.564 1.558 1.441 1.4\n 1.377 1.347 1.267 1.222 1.159 1.086 1.046 1.016 0.999 0.945 0.918 0.902 0.894 0.875 0.87 0.841]\nW_o.0 [7.103 6.607 4.268 2.77 2.503 2.285 2.264 2.218 2.151 2.139 2.105 2.08 2.059 2.026 2.003 1.987 1.928 1.89 1.836 1.825 1.804 1.731 1.72 1.694 1.685 1.581 1.543 1.518 1.507\n 1.428 1.4 1.349 1.31 1.299 1.248 1.223 1.202 1.187 1.17 1.163 1.142 1.109 1.081 1.068 1.036]\nW_fc.0 [18.176 11.018 8.737 7.452 7.212 7.062 7.011 6.62 6.57 6.42 6.343 6.29 6.137 6.043 5.974 5.918 5.807 5.779 5.583 5.513 5.5 5.36 5.16 5.12 5.086\n 5.032 4.899 4.879 4.819 4.732 4.601 4.533 4.479 4.441 4.268 4.198 4.031 3.974 3.869 3.824 3.72 3.683 3.646 3.574 3.516]\nW_proj.0 [9.826 7.351 6.265 5.869 5.797 5.657 5.516 5.45 5.333 5.242 5.197 5.094 4.98 4.904 4.863 4.819 4.793 4.689 4.627 4.595 4.557 4.523 4.411 4.378 4.283 4.216 4.143 4.125 4.048\n 4.025 3.894 3.847 3.808 3.792 3.764 3.711 3.685 3.636 3.595 3.525 3.499 3.479 3.469 3.422 3.39 ]\nW_proj.1 [8.193 7.894 7.199 6.997 6.671 6.433 6.274 6.133 6.099 6.021 5.942 5.865 5.829 5.789 5.698 5.675 5.646 5.629 5.581 5.564 5.553 5.538 5.528 5.477 5.456 5.447 5.438 5.403 5.399\n 5.38 5.367 5.351 5.332 5.327 5.314 5.296 5.252 5.235 5.219 5.208 5.197 5.183 5.174 5.17 5.146]\nW_proj.2 [9.404 9.187 8.723 8.47 8.04 7.795 7.614 7.45 7.392 7.253 7.223 7.148 7.082 6.972 6.829 6.733 6.637 6.547 6.46 6.404 6.374 6.307 6.209 6.181 6.12 6.091 5.989 5.9 5.88\n 5.833 5.758 5.703 5.685 5.645 5.588 5.559 5.491 5.478 5.467 5.429 5.406 5.4 5.378 5.351 5.335]\nW_proj.3 [26.309 20.685 19.699 19.261 18.594 18.197 17.862 17.667 17.044 16.749 16.039 15.151 14.756 14.326 14.05 13.5 13.293 12.531 12.401 11.958 11.74 11.594 10.367 8.351 8.064\n 7.914 7.461 7.166 5.833 4.072 3.221 3.139 2.842 2.8 2.687 2.571 2.403 2.328 2.244 2.185 2.181 2.116 2.081 2.053 1.996]\n\nL0H2 left/right-space joint singular values [1.406 1.4046 1.4043 1.4001 1.3966 1.395 1.3932 1.3834 1.3806 1.373 1.3689 1.3616 1.3366 1.2648 1.2248 1.0401 0.9582 0.7071 0.6326 0.4621 0.382 0.3551 0.3389 0.3066 0.2937\n 0.2429 0.2324 0.2225 0.1996 0.1674 0.1643 0.1521]\nCandidate position-space dimension 16\nlayer 0 Q/K/both energy: [[0.7 0.064 0.042]\n [0.491 0.202 0.109]\n [0.902 0.897 0.83 ]\n [0.637 0.813 0.554]\n [0.196 0.135 0.026]\n [0.782 0.164 0.13 ]]\nlayer 1 Q/K/both energy: [[0.054 0.136 0.006]\n [0.062 0.12 0.007]\n [0.074 0.121 0.01 ]\n [0.068 0.129 0.009]\n [0.074 0.137 0.01 ]\n [0.071 0.125 0.009]]\nlayer 2 Q/K/both energy: [[0.037 0.052 0.002]\n [0.035 0.043 0.001]\n [0.043 0.063 0.003]\n [0.04 0.052 0.003]\n [0.041 0.054 0.002]\n [0.042 0.052 0.002]]\nlayer 3 Q/K/both energy: [[0.041 0.041 0.002]\n [0.036 0.043 0.003]\n [0.042 0.051 0.002]\n [0.041 0.05 0.003]\n [0.041 0.044 0.002]\n [0.036 0.047 0.001]]\nCandidate position-space dimension 24\nlayer 0 Q/K/both energy: [[0.745 0.159 0.118]\n [0.56 0.326 0.202]\n [0.969 0.97 0.947]\n [0.805 0.837 0.706]\n [0.487 0.334 0.172]\n [0.84 0.473 0.398]]\nlayer 1 Q/K/both energy: [[0.076 0.229 0.018]\n [0.082 0.205 0.019]\n [0.097 0.214 0.024]\n [0.098 0.22 0.025]\n [0.099 0.242 0.027]\n [0.095 0.216 0.022]]\nlayer 2 Q/K/both energy: [[0.059 0.082 0.007]\n [0.052 0.065 0.004]\n [0.065 0.106 0.008]\n [0.061 0.082 0.007]\n [0.066 0.084 0.007]\n [0.064 0.081 0.007]]\nlayer 3 Q/K/both energy: [[0.059 0.06 0.004]\n [0.053 0.062 0.005]\n [0.064 0.072 0.005]\n [0.064 0.073 0.006]\n [0.06 0.062 0.004]\n [0.06 0.07 0.004]]\nCandidate position-space dimension 32\nlayer 0 Q/K/both energy: [[0.821 0.433 0.363]\n [0.618 0.468 0.308]\n [0.985 0.985 0.976]\n [0.851 0.865 0.756]\n [0.598 0.464 0.288]\n [0.895 0.562 0.506]]\nlayer 1 Q/K/both energy: [[0.102 0.328 0.033]\n [0.121 0.295 0.039]\n [0.127 0.311 0.046]\n [0.127 0.32 0.045]\n [0.125 0.346 0.047]\n [0.125 0.318 0.042]]\nlayer 2 Q/K/both energy: [[0.083 0.112 0.013]\n [0.069 0.09 0.008]\n [0.09 0.15 0.017]\n [0.084 0.117 0.014]\n [0.087 0.115 0.014]\n [0.087 0.113 0.014]]\nlayer 3 Q/K/both energy: [[0.078 0.081 0.007]\n [0.069 0.083 0.008]\n [0.085 0.101 0.01 ]\n [0.087 0.101 0.011]\n [0.082 0.087 0.008]\n [0.077 0.089 0.007]]\n\nKEY-COMPOSITION CANDIDATES: QK_l @ OV_earlier\nFor each head: strongest positive trace/Frobenius scores; entries=(earlier head, trace/Fnorm, transpose symmetry, Fnorm).\nL1H0 [('L0H0', np.float64(0.072), np.float64(0.007), np.float64(4.379)), ('L0H4', np.float64(0.062), np.float64(0.004), np.float64(1.866)), ('L0H1', np.float64(0.032), np.float64(0.006), np.float64(3.264)), ('L0H5', np.float64(0.007), np.float64(0.001), np.float64(4.613)), ('L0H2', np.float64(-0.113), np.float64(0.01), np.float64(0.108))]\nL1H1 [('L0H4', np.float64(0.102), np.float64(0.007), np.float64(2.573)), ('L0H0', np.float64(0.092), np.float64(0.012), np.float64(5.94)), ('L0H1', np.float64(0.066), np.float64(0.009), np.float64(4.607)), ('L0H5', np.float64(0.033), np.float64(0.002), np.float64(6.86)), ('L0H2', np.float64(-0.003), np.float64(0.036), np.float64(0.213))]\nL1H2 [('L0H2', np.float64(0.253), np.float64(0.012), np.float64(0.227)), ('L0H4', np.float64(0.112), np.float64(0.013), np.float64(3.017)), ('L0H0', np.float64(0.101), np.float64(0.017), np.float64(6.982)), ('L0H1', np.float64(0.084), np.float64(0.016), np.float64(5.425)), ('L0H5', np.float64(0.04), np.float64(0.007), np.float64(7.968))]\nL1H3 [('L0H2', np.float64(0.233), np.float64(0.014), np.float64(0.22)), ('L0H4', np.float64(0.123), np.float64(0.013), np.float64(2.721)), ('L0H0', np.float64(0.116), np.float64(0.018), np.float64(6.348)), ('L0H1', np.float64(0.094), np.float64(0.013), np.float64(4.85)), ('L0H5', np.float64(0.05), np.float64(0.006), np.float64(7.229))]\nL1H4 [('L0H2', np.float64(0.251), np.float64(0.007), np.float64(0.196)), ('L0H4', np.float64(0.163), np.float64(0.02), np.float64(3.015)), ('L0H0', np.float64(0.149), np.float64(0.023), np.float64(7.072)), ('L0H1', np.float64(0.134), np.float64(0.018), np.float64(5.351)), ('L0H5', np.float64(0.081), np.float64(0.006), np.float64(7.691))]\nL1H5 [('L0H0', np.float64(0.078), np.float64(0.019), np.float64(6.069)), ('L0H2', np.float64(0.077), np.float64(0.004), np.float64(0.267)), ('L0H4', np.float64(0.076), np.float64(0.014), np.float64(2.605)), ('L0H1', np.float64(0.036), np.float64(0.019), np.float64(4.682)), ('L0H5', np.float64(-0.02), np.float64(0.012), np.float64(6.959))]\nL2H0 [('L0H4', np.float64(0.096), np.float64(0.006), np.float64(1.879)), ('L0H0', np.float64(0.08), np.float64(0.007), np.float64(4.368)), ('L0H5', np.float64(0.071), np.float64(0.002), np.float64(5.026)), ('L0H1', np.float64(0.062), np.float64(0.003), np.float64(3.425)), ('L0H2', np.float64(-0.08), np.float64(-0.001), np.float64(0.388))]\nL2H1 [('L0H4', np.float64(0.076), np.float64(0.003), np.float64(1.145)), ('L0H5', np.float64(0.071), np.float64(0.0), np.float64(3.117)), ('L0H0', np.float64(0.071), np.float64(0.001), np.float64(2.597)), ('L1H3', np.float64(0.063), np.float64(0.005), np.float64(1.019)), ('L0H1', np.float64(0.046), np.float64(0.002), np.float64(2.064))]\nL2H2 [('L1H4', np.float64(0.174), np.float64(0.009), np.float64(0.748)), ('L0H4', np.float64(0.136), np.float64(0.013), np.float64(1.71)), ('L0H0', np.float64(0.11), np.float64(0.013), np.float64(3.936)), ('L1H3', np.float64(0.091), np.float64(0.001), np.float64(0.713)), ('L0H1', np.float64(0.087), np.float64(0.003), np.float64(3.13))]\nL2H3 [('L0H4', np.float64(0.116), np.float64(0.008), np.float64(1.681)), ('L0H0', np.float64(0.111), np.float64(0.009), np.float64(3.904)), ('L0H5', np.float64(0.099), np.float64(0.003), np.float64(4.258)), ('L0H1', np.float64(0.08), np.float64(0.004), np.float64(3.015)), ('L0H2', np.float64(0.054), np.float64(-0.001), np.float64(0.286))]\nL2H4 [('L0H4', np.float64(0.152), np.float64(0.016), np.float64(1.865)), ('L0H0', np.float64(0.144), np.float64(0.019), np.float64(4.344)), ('L0H1', np.float64(0.108), np.float64(0.006), np.float64(3.348)), ('L0H5', np.float64(0.1), np.float64(-0.001), np.float64(4.602)), ('L1H3', np.float64(-0.059), np.float64(-0.005), np.float64(1.194))]\nL2H5 [('L0H0', np.float64(0.112), np.float64(0.008), np.float64(4.078)), ('L0H4', np.float64(0.11), np.float64(0.007), np.float64(1.759)), ('L0H5', np.float64(0.078), np.float64(-0.002), np.float64(4.482)), ('L0H1', np.float64(0.073), np.float64(0.001), np.float64(3.157)), ('L1H2', np.float64(0.007), np.float64(0.005), np.float64(1.03))]\nL3H0 [('L0H4', np.float64(0.088), np.float64(0.006), np.float64(0.725)), ('L0H5', np.float64(0.078), np.float64(0.004), np.float64(1.848)), ('L0H1', np.float64(0.06), np.float64(0.003), np.float64(1.256)), ('L0H0', np.float64(0.057), np.float64(0.001), np.float64(1.653)), ('L0H2', np.float64(-0.042), np.float64(0.001), np.float64(0.309))]\nL3H1 [('L2H1', np.float64(1.25), np.float64(0.041), np.float64(2.105)), ('L0H2', np.float64(1.173), np.float64(0.097), np.float64(0.376)), ('L1H2', np.float64(0.879), np.float64(0.033), np.float64(1.061)), ('L1H3', np.float64(0.768), np.float64(0.049), np.float64(1.36)), ('L1H1', np.float64(0.63), np.float64(0.012), np.float64(1.087))]\nL3H2 [('L0H5', np.float64(0.101), np.float64(0.008), np.float64(3.049)), ('L0H0', np.float64(0.087), np.float64(0.006), np.float64(2.532)), ('L0H4', np.float64(0.082), np.float64(0.003), np.float64(1.052)), ('L0H2', np.float64(0.063), np.float64(0.0), np.float64(0.18)), ('L0H1', np.float64(0.059), np.float64(0.002), np.float64(1.86))]\nL3H3 [('L0H5', np.float64(0.07), np.float64(0.002), np.float64(3.087)), ('L0H0', np.float64(0.06), np.float64(0.003), np.float64(2.578)), ('L0H4', np.float64(0.05), np.float64(0.002), np.float64(1.107)), ('L0H1', np.float64(0.03), np.float64(-0.001), np.float64(1.951)), ('L0H2', np.float64(0.024), np.float64(-0.004), np.float64(0.195))]\nL3H4 [('L0H5', np.float64(0.164), np.float64(0.024), np.float64(2.969)), ('L0H4', np.float64(0.16), np.float64(0.019), np.float64(1.065)), ('L0H1', np.float64(0.142), np.float64(0.016), np.float64(1.879)), ('L0H0', np.float64(0.134), np.float64(0.015), np.float64(2.471)), ('L2H0', np.float64(0.015), np.float64(0.002), np.float64(1.46))]\nL3H5 [('L2H3', np.float64(0.318), np.float64(0.009), np.float64(0.773)), ('L2H4', np.float64(0.228), np.float64(0.007), np.float64(0.678)), ('L2H1', np.float64(0.217), np.float64(0.005), np.float64(0.786)), ('L2H5', np.float64(0.198), np.float64(0.005), np.float64(0.689)), ('L1H0', np.float64(0.186), np.float64(0.006), np.float64(0.506))]\n\nPairwise cosine similarities of effective QK products, within each layer\nLayer 0 \n [[ 1. 0.209 -0.016 -0.014 0.115 0.018]\n [ 0.209 1. -0.004 -0.036 0.21 -0.008]\n [-0.016 -0.004 1. 0.24 -0.006 -0.026]\n [-0.014 -0.036 0.24 1. -0.054 -0.058]\n [ 0.115 0.21 -0.006 -0.054 1. 0.172]\n [ 0.018 -0.008 -0.026 -0.058 0.172 1. ]]\nLayer 1 \n [[1. 0.578 0.416 0.448 0.455 0.464]\n [0.578 1. 0.55 0.488 0.556 0.528]\n [0.416 0.55 1. 0.608 0.667 0.564]\n [0.448 0.488 0.608 1. 0.644 0.563]\n [0.455 0.556 0.667 0.644 1. 0.577]\n [0.464 0.528 0.564 0.563 0.577 1. ]]\nLayer 2 \n [[1. 0.078 0.133 0.103 0.112 0.106]\n [0.078 1. 0.093 0.086 0.077 0.07 ]\n [0.133 0.093 1. 0.118 0.126 0.125]\n [0.103 0.086 0.118 1. 0.12 0.113]\n [0.112 0.077 0.126 0.12 1. 0.107]\n [0.106 0.07 0.125 0.113 0.107 1. ]]\nLayer 3 \n [[1. 0.071 0.044 0.036 0.075 0.03 ]\n [0.071 1. 0.021 0.024 0.03 0.045]\n [0.044 0.021 1. 0.094 0.103 0.054]\n [0.036 0.024 0.094 1. 0.084 0.065]\n [0.075 0.03 0.103 0.084 1. 0.049]\n [0.03 0.045 0.054 0.065 0.049 1. ]]\n",
"seconds": 1.128,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "39791d98be7841ea69ac9be79e9ba37873049afa0781e7ac2e452b3ba48465fd",
"weights_sha256": "130135d2b27829df2734d78fa3af27b00b1cc1bf34c41b01a09b350e6c09a7de"
}All calculations below are matrix products or decompositions, not layer evaluations.
LAYER 0
L0H0 Q/K/V/O norms [3.305 4.18 6.039 6.666] effective QK norm/top/sym/trace [ 0.0546 0.0514 0.0233 -0.0047] query_bias_ratio 0.214 key_bias_norm 0.0136 QKev [ 0.007+0.j -0.004+0.j -0.002+0.j -0. +0.001j -0. -0.001j 0. +0.001j 0. -0.001j -0.001+0.001j]
L0H1 Q/K/V/O norms [3.09 3.658 5.382 6.025] effective QK norm/top/sym/trace [ 0.0295 0.025 0.0475 -0.014 ] query_bias_ratio 0.1243 key_bias_norm 0.0033 QKev [-0.004+0.001j -0.004-0.001j 0.002+0.j -0.001+0.001j -0.001-0.001j -0.001+0.j -0.001+0.001j -0.001-0.001j]
L0H2 Q/K/V/O norms [30.818 28.3 7.765 8.915] effective QK norm/top/sym/trace [ 4.0688 1.4079 -0.0446 1.199 ] query_bias_ratio 0.0589 key_bias_norm 0.1428 QKev [ 0.632+0.813j 0.632-0.813j -0.231+0.927j -0.231-0.927j 0.332+0.871j 0.332-0.871j -0.777+0.334j -0.777-0.334j]
L0H3 Q/K/V/O norms [6.821 7.243 5.434 5.029] effective QK norm/top/sym/trace [0.3945 0.3588 0.4782 0.1968] query_bias_ratio 0.1771 key_bias_norm 0.0535 QKev [ 0.262+0.j 0.078+0.j 0.01 +0.031j 0.01 -0.031j -0.017+0.005j -0.017-0.005j -0.017+0.j -0.015+0.003j]
L0H4 Q/K/V/O norms [3.971 4.995 5.208 5.587] effective QK norm/top/sym/trace [ 0.1062 0.089 0.004 -0.0707] query_bias_ratio 0.1505 key_bias_norm 0.0112 QKev [-0.028+0.028j -0.028-0.028j -0.008+0.j -0.002+0.j -0.001+0.001j -0.001-0.001j -0.002+0.j 0.001+0.001j]
L0H5 Q/K/V/O norms [3.19 3.802 6.143 7.162] effective QK norm/top/sym/trace [ 0.0775 0.0759 0.1603 -0.0347] query_bias_ratio 0.1849 key_bias_norm 0.0162 QKev [-0.031+0.j 0.001+0.002j 0.001-0.002j -0.001+0.j -0.001-0.j -0.001+0.001j -0.001-0.001j 0.001+0.001j]
LAYER 1
L1H0 Q/K/V/O norms [10.716 9.827 11.156 9.984] effective QK norm/top/sym/trace [ 0.8864 0.5969 0.0086 -0.6436] query_bias_ratio 0.7821 key_bias_norm 1.3572 QKev [ 0.028+0.106j 0.028-0.106j -0.078+0.j -0.066+0.028j -0.066-0.028j -0.055+0.007j -0.055-0.007j -0.047+0.j ]
L1H1 Q/K/V/O norms [13.292 12.068 11.72 11.303] effective QK norm/top/sym/trace [ 1.3146 0.8811 0.0186 -1.1603] query_bias_ratio 0.8034 key_bias_norm 2.2848 QKev [ 0.07 +0.191j 0.07 -0.191j -0.153+0.j -0.107+0.024j -0.107-0.024j -0.108+0.006j -0.108-0.006j -0.086+0.j ]
L1H2 Q/K/V/O norms [14.476 13.092 10.668 10.272] effective QK norm/top/sym/trace [ 1.5491 0.9943 0.0179 -1.6165] query_bias_ratio 0.6959 key_bias_norm 2.5879 QKev [ 0.097+0.246j 0.097-0.246j -0.168+0.j -0.141+0.064j -0.141-0.064j -0.135+0.009j -0.135-0.009j -0.104+0.03j ]
L1H3 Q/K/V/O norms [13.358 12.153 10.904 9.928] effective QK norm/top/sym/trace [ 1.3742 0.9188 0.0172 -1.1551] query_bias_ratio 0.7616 key_bias_norm 2.3511 QKev [ 0.094+0.203j 0.094-0.203j -0.175+0.j -0.113+0.008j -0.113-0.008j -0.107+0.j -0.04 +0.091j -0.04 -0.091j]
L1H4 Q/K/V/O norms [13.518 12.129 11.148 10.794] effective QK norm/top/sym/trace [ 1.4497 0.9476 0.0108 -1.2432] query_bias_ratio 0.711 key_bias_norm 2.2902 QKev [ 0.062+0.214j 0.062-0.214j -0.135+0.01j -0.135-0.01j -0.127+0.j -0.099+0.j -0.081+0.053j -0.081-0.053j]
L1H5 Q/K/V/O norms [13.486 12.132 11.376 11.136] effective QK norm/top/sym/trace [ 1.3363 0.8925 -0.0027 -1.0127] query_bias_ratio 0.6733 key_bias_norm 1.7526 QKev [ 0.088+0.221j 0.088-0.221j -0.121+0.012j -0.121-0.012j -0.1 +0.004j -0.1 -0.004j -0.074+0.055j -0.074-0.055j]
LAYER 2
L2H0 Q/K/V/O norms [16.822 16.208 13.115 12.942] effective QK norm/top/sym/trace [ 2.4375 0.6943 -0.0335 -0.0216] query_bias_ratio 0.3088 key_bias_norm 1.6991 QKev [ 0.053+0.306j 0.053-0.306j 0.036+0.183j 0.036-0.183j 0.118+0.08j 0.118-0.08j -0.077+0.098j -0.077-0.098j]
L2H1 Q/K/V/O norms [15.963 15.287 13.768 13.525] effective QK norm/top/sym/trace [2.0677 0.5267 0.0101 0.53 ] query_bias_ratio 0.2076 key_bias_norm 0.7446 QKev [ 0.027+0.144j 0.027-0.144j 0.126+0.011j 0.126-0.011j 0.109+0.037j 0.109-0.037j -0.084+0.071j -0.084-0.071j]
L2H2 Q/K/V/O norms [13.97 13.239 14.567 14.24 ] effective QK norm/top/sym/trace [ 1.6127 0.6069 -0.0046 -1.1051] query_bias_ratio 0.2312 key_bias_norm 0.7554 QKev [ 0.037+0.175j 0.037-0.175j -0.106+0.j 0.067+0.055j 0.067-0.055j -0.084+0.015j -0.084-0.015j -0.076-0.014j]
L2H3 Q/K/V/O norms [16.121 15.554 13.498 13.391] effective QK norm/top/sym/trace [ 2.0862 0.5616 -0.0268 -0.7372] query_bias_ratio 0.2623 key_bias_norm 1.4625 QKev [ 0.072+0.307j 0.072-0.307j -0.076+0.084j -0.076-0.084j -0.102+0.042j -0.102-0.042j -0.104+0.005j -0.104-0.005j]
L2H4 Q/K/V/O norms [16.876 16.188 13.241 13.143] effective QK norm/top/sym/trace [ 2.3438 0.6278 -0.0248 -0.4747] query_bias_ratio 0.2685 key_bias_norm 1.5774 QKev [ 0.061+0.303j 0.061-0.303j 0.077+0.102j 0.077-0.102j -0.081+0.091j -0.081-0.091j -0.118+0.j -0.045+0.105j]
L2H5 Q/K/V/O norms [16.445 15.839 13.339 13.161] effective QK norm/top/sym/trace [ 2.2503 0.6375 -0.0283 -0.6957] query_bias_ratio 0.2779 key_bias_norm 1.5086 QKev [ 0.046+0.302j 0.046-0.302j -0.141+0.j -0.071-0.108j -0.071+0.108j -0.119+0.037j -0.119-0.037j -0.068+0.099j]
LAYER 3
L3H0 Q/K/V/O norms [15.747 15.258 14.874 14.87 ] effective QK norm/top/sym/trace [2.2322 0.4514 0.0503 3.4122] query_bias_ratio 0.1057 key_bias_norm 0.2581 QKev [0.182+0.j 0.141+0.078j 0.141-0.078j 0.158+0.019j 0.158-0.019j 0.143+0.056j 0.143-0.056j 0.141+0.036j]
L3H1 Q/K/V/O norms [14.22 14.197 17.885 18.331] effective QK norm/top/sym/trace [1.8295 0.3275 0.2114 6.3101] query_bias_ratio 0.1322 key_bias_norm 0.2408 QKev [0.174-0.06j 0.174+0.06j 0.171-0.067j 0.171+0.067j 0.149-0.082j 0.149+0.082j 0.164-0.031j 0.164+0.031j]
L3H2 Q/K/V/O norms [12.743 12.446 15.523 15.585] effective QK norm/top/sym/trace [ 1.505 0.4702 -0.0036 0.2216] query_bias_ratio 0.124 key_bias_norm 0.2895 QKev [ 0.03 +0.134j 0.03 -0.134j 0.081+0.038j 0.081-0.038j -0.035+0.082j -0.035-0.082j 0.082+0.j -0.077+0.j ]
L3H3 Q/K/V/O norms [13.283 13.057 15.732 16.118] effective QK norm/top/sym/trace [ 1.5985 0.4973 -0.0166 0.1116] query_bias_ratio 0.121 key_bias_norm 0.3603 QKev [ 0.023+0.159j 0.023-0.159j 0.099+0.j -0.023+0.095j -0.023-0.095j 0.076+0.036j 0.076-0.036j 0.078+0.j ]
L3H4 Q/K/V/O norms [14.356 13.874 15.175 15.3 ] effective QK norm/top/sym/trace [ 1.8555 0.5287 -0.0005 0.9035] query_bias_ratio 0.1217 key_bias_norm 0.3072 QKev [0.068+0.092j 0.068-0.092j 0.002+0.108j 0.002-0.108j 0.098+0.04j 0.098-0.04j 0.102+0.j 0.07 +0.066j]
L3H5 Q/K/V/O norms [ 9.514 9.231 17.993 18.512] effective QK norm/top/sym/trace [0.8339 0.2591 0.0242 0.1007] query_bias_ratio 0.1537 key_bias_norm 0.1645 QKev [0.075+0.035j 0.075-0.035j 0.078+0.j 0.02 +0.048j 0.02 -0.048j 0.023+0.036j 0.023-0.036j 0.038+0.011j]
SINGULAR VALUES: first 45 for selected matrices
W_q.0 [8.252 8.223 7.963 7.819 7.68 7.642 7.519 7.205 6.948 6.819 6.388 6.165 5.801 5.555 5.13 4.947 4.854 4.52 4.268 3.951 3.591 3.527 3.285 3.125 3.026 2.928 2.707 2.511 2.471
2.336 2.249 2.136 2.089 2.058 1.914 1.792 1.755 1.679 1.536 1.513 1.475 1.455 1.327 1.31 1.282]
W_k.0 [7.646 7.436 7.33 6.912 6.829 6.672 6.472 6.446 6.393 6.118 6.009 5.943 5.629 5.505 4.99 4.901 4.698 4.293 4.066 4.018 3.708 3.529 3.343 3.068 2.987 2.874 2.696 2.586 2.412
2.262 2.216 2.11 2.032 1.876 1.744 1.733 1.697 1.666 1.547 1.514 1.441 1.402 1.386 1.334 1.29 ]
W_v.0 [4.339 3.529 3.27 3.166 2.982 2.647 2.589 2.552 2.436 2.381 2.256 2.162 2.13 2.103 2.042 2.017 2.003 1.973 1.928 1.897 1.843 1.787 1.711 1.665 1.628 1.564 1.558 1.441 1.4
1.377 1.347 1.267 1.222 1.159 1.086 1.046 1.016 0.999 0.945 0.918 0.902 0.894 0.875 0.87 0.841]
W_o.0 [7.103 6.607 4.268 2.77 2.503 2.285 2.264 2.218 2.151 2.139 2.105 2.08 2.059 2.026 2.003 1.987 1.928 1.89 1.836 1.825 1.804 1.731 1.72 1.694 1.685 1.581 1.543 1.518 1.507
1.428 1.4 1.349 1.31 1.299 1.248 1.223 1.202 1.187 1.17 1.163 1.142 1.109 1.081 1.068 1.036]
W_fc.0 [18.176 11.018 8.737 7.452 7.212 7.062 7.011 6.62 6.57 6.42 6.343 6.29 6.137 6.043 5.974 5.918 5.807 5.779 5.583 5.513 5.5 5.36 5.16 5.12 5.086
5.032 4.899 4.879 4.819 4.732 4.601 4.533 4.479 4.441 4.268 4.198 4.031 3.974 3.869 3.824 3.72 3.683 3.646 3.574 3.516]
W_proj.0 [9.826 7.351 6.265 5.869 5.797 5.657 5.516 5.45 5.333 5.242 5.197 5.094 4.98 4.904 4.863 4.819 4.793 4.689 4.627 4.595 4.557 4.523 4.411 4.378 4.283 4.216 4.143 4.125 4.048
4.025 3.894 3.847 3.808 3.792 3.764 3.711 3.685 3.636 3.595 3.525 3.499 3.479 3.469 3.422 3.39 ]
W_proj.1 [8.193 7.894 7.199 6.997 6.671 6.433 6.274 6.133 6.099 6.021 5.942 5.865 5.829 5.789 5.698 5.675 5.646 5.629 5.581 5.564 5.553 5.538 5.528 5.477 5.456 5.447 5.438 5.403 5.399
5.38 5.367 5.351 5.332 5.327 5.314 5.296 5.252 5.235 5.219 5.208 5.197 5.183 5.174 5.17 5.146]
W_proj.2 [9.404 9.187 8.723 8.47 8.04 7.795 7.614 7.45 7.392 7.253 7.223 7.148 7.082 6.972 6.829 6.733 6.637 6.547 6.46 6.404 6.374 6.307 6.209 6.181 6.12 6.091 5.989 5.9 5.88
5.833 5.758 5.703 5.685 5.645 5.588 5.559 5.491 5.478 5.467 5.429 5.406 5.4 5.378 5.351 5.335]
W_proj.3 [26.309 20.685 19.699 19.261 18.594 18.197 17.862 17.667 17.044 16.749 16.039 15.151 14.756 14.326 14.05 13.5 13.293 12.531 12.401 11.958 11.74 11.594 10.367 8.351 8.064
7.914 7.461 7.166 5.833 4.072 3.221 3.139 2.842 2.8 2.687 2.571 2.403 2.328 2.244 2.185 2.181 2.116 2.081 2.053 1.996]
L0H2 left/right-space joint singular values [1.406 1.4046 1.4043 1.4001 1.3966 1.395 1.3932 1.3834 1.3806 1.373 1.3689 1.3616 1.3366 1.2648 1.2248 1.0401 0.9582 0.7071 0.6326 0.4621 0.382 0.3551 0.3389 0.3066 0.2937
0.2429 0.2324 0.2225 0.1996 0.1674 0.1643 0.1521]
Candidate position-space dimension 16
layer 0 Q/K/both energy: [[0.7 0.064 0.042]
[0.491 0.202 0.109]
[0.902 0.897 0.83 ]
[0.637 0.813 0.554]
[0.196 0.135 0.026]
[0.782 0.164 0.13 ]]
layer 1 Q/K/both energy: [[0.054 0.136 0.006]
[0.062 0.12 0.007]
[0.074 0.121 0.01 ]
[0.068 0.129 0.009]
[0.074 0.137 0.01 ]
[0.071 0.125 0.009]]
layer 2 Q/K/both energy: [[0.037 0.052 0.002]
[0.035 0.043 0.001]
[0.043 0.063 0.003]
[0.04 0.052 0.003]
[0.041 0.054 0.002]
[0.042 0.052 0.002]]
layer 3 Q/K/both energy: [[0.041 0.041 0.002]
[0.036 0.043 0.003]
[0.042 0.051 0.002]
[0.041 0.05 0.003]
[0.041 0.044 0.002]
[0.036 0.047 0.001]]
Candidate position-space dimension 24
layer 0 Q/K/both energy: [[0.745 0.159 0.118]
[0.56 0.326 0.202]
[0.969 0.97 0.947]
[0.805 0.837 0.706]
[0.487 0.334 0.172]
[0.84 0.473 0.398]]
layer 1 Q/K/both energy: [[0.076 0.229 0.018]
[0.082 0.205 0.019]
[0.097 0.214 0.024]
[0.098 0.22 0.025]
[0.099 0.242 0.027]
[0.095 0.216 0.022]]
layer 2 Q/K/both energy: [[0.059 0.082 0.007]
[0.052 0.065 0.004]
[0.065 0.106 0.008]
[0.061 0.082 0.007]
[0.066 0.084 0.007]
[0.064 0.081 0.007]]
layer 3 Q/K/both energy: [[0.059 0.06 0.004]
[0.053 0.062 0.005]
[0.064 0.072 0.005]
[0.064 0.073 0.006]
[0.06 0.062 0.004]
[0.06 0.07 0.004]]
Candidate position-space dimension 32
layer 0 Q/K/both energy: [[0.821 0.433 0.363]
[0.618 0.468 0.308]
[0.985 0.985 0.976]
[0.851 0.865 0.756]
[0.598 0.464 0.288]
[0.895 0.562 0.506]]
layer 1 Q/K/both energy: [[0.102 0.328 0.033]
[0.121 0.295 0.039]
[0.127 0.311 0.046]
[0.127 0.32 0.045]
[0.125 0.346 0.047]
[0.125 0.318 0.042]]
layer 2 Q/K/both energy: [[0.083 0.112 0.013]
[0.069 0.09 0.008]
[0.09 0.15 0.017]
[0.084 0.117 0.014]
[0.087 0.115 0.014]
[0.087 0.113 0.014]]
layer 3 Q/K/both energy: [[0.078 0.081 0.007]
[0.069 0.083 0.008]
[0.085 0.101 0.01 ]
[0.087 0.101 0.011]
[0.082 0.087 0.008]
[0.077 0.089 0.007]]
KEY-COMPOSITION CANDIDATES: QK_l @ OV_earlier
For each head: strongest positive trace/Frobenius scores; entries=(earlier head, trace/Fnorm, transpose symmetry, Fnorm).
L1H0 [('L0H0', np.float64(0.072), np.float64(0.007), np.float64(4.379)), ('L0H4', np.float64(0.062), np.float64(0.004), np.float64(1.866)), ('L0H1', np.float64(0.032), np.float64(0.006), np.float64(3.264)), ('L0H5', np.float64(0.007), np.float64(0.001), np.float64(4.613)), ('L0H2', np.float64(-0.113), np.float64(0.01), np.float64(0.108))]
L1H1 [('L0H4', np.float64(0.102), np.float64(0.007), np.float64(2.573)), ('L0H0', np.float64(0.092), np.float64(0.012), np.float64(5.94)), ('L0H1', np.float64(0.066), np.float64(0.009), np.float64(4.607)), ('L0H5', np.float64(0.033), np.float64(0.002), np.float64(6.86)), ('L0H2', np.float64(-0.003), np.float64(0.036), np.float64(0.213))]
L1H2 [('L0H2', np.float64(0.253), np.float64(0.012), np.float64(0.227)), ('L0H4', np.float64(0.112), np.float64(0.013), np.float64(3.017)), ('L0H0', np.float64(0.101), np.float64(0.017), np.float64(6.982)), ('L0H1', np.float64(0.084), np.float64(0.016), np.float64(5.425)), ('L0H5', np.float64(0.04), np.float64(0.007), np.float64(7.968))]
L1H3 [('L0H2', np.float64(0.233), np.float64(0.014), np.float64(0.22)), ('L0H4', np.float64(0.123), np.float64(0.013), np.float64(2.721)), ('L0H0', np.float64(0.116), np.float64(0.018), np.float64(6.348)), ('L0H1', np.float64(0.094), np.float64(0.013), np.float64(4.85)), ('L0H5', np.float64(0.05), np.float64(0.006), np.float64(7.229))]
L1H4 [('L0H2', np.float64(0.251), np.float64(0.007), np.float64(0.196)), ('L0H4', np.float64(0.163), np.float64(0.02), np.float64(3.015)), ('L0H0', np.float64(0.149), np.float64(0.023), np.float64(7.072)), ('L0H1', np.float64(0.134), np.float64(0.018), np.float64(5.351)), ('L0H5', np.float64(0.081), np.float64(0.006), np.float64(7.691))]
L1H5 [('L0H0', np.float64(0.078), np.float64(0.019), np.float64(6.069)), ('L0H2', np.float64(0.077), np.float64(0.004), np.float64(0.267)), ('L0H4', np.float64(0.076), np.float64(0.014), np.float64(2.605)), ('L0H1', np.float64(0.036), np.float64(0.019), np.float64(4.682)), ('L0H5', np.float64(-0.02), np.float64(0.012), np.float64(6.959))]
L2H0 [('L0H4', np.float64(0.096), np.float64(0.006), np.float64(1.879)), ('L0H0', np.float64(0.08), np.float64(0.007), np.float64(4.368)), ('L0H5', np.float64(0.071), np.float64(0.002), np.float64(5.026)), ('L0H1', np.float64(0.062), np.float64(0.003), np.float64(3.425)), ('L0H2', np.float64(-0.08), np.float64(-0.001), np.float64(0.388))]
L2H1 [('L0H4', np.float64(0.076), np.float64(0.003), np.float64(1.145)), ('L0H5', np.float64(0.071), np.float64(0.0), np.float64(3.117)), ('L0H0', np.float64(0.071), np.float64(0.001), np.float64(2.597)), ('L1H3', np.float64(0.063), np.float64(0.005), np.float64(1.019)), ('L0H1', np.float64(0.046), np.float64(0.002), np.float64(2.064))]
L2H2 [('L1H4', np.float64(0.174), np.float64(0.009), np.float64(0.748)), ('L0H4', np.float64(0.136), np.float64(0.013), np.float64(1.71)), ('L0H0', np.float64(0.11), np.float64(0.013), np.float64(3.936)), ('L1H3', np.float64(0.091), np.float64(0.001), np.float64(0.713)), ('L0H1', np.float64(0.087), np.float64(0.003), np.float64(3.13))]
L2H3 [('L0H4', np.float64(0.116), np.float64(0.008), np.float64(1.681)), ('L0H0', np.float64(0.111), np.float64(0.009), np.float64(3.904)), ('L0H5', np.float64(0.099), np.float64(0.003), np.float64(4.258)), ('L0H1', np.float64(0.08), np.float64(0.004), np.float64(3.015)), ('L0H2', np.float64(0.054), np.float64(-0.001), np.float64(0.286))]
L2H4 [('L0H4', np.float64(0.152), np.float64(0.016), np.float64(1.865)), ('L0H0', np.float64(0.144), np.float64(0.019), np.float64(4.344)), ('L0H1', np.float64(0.108), np.float64(0.006), np.float64(3.348)), ('L0H5', np.float64(0.1), np.float64(-0.001), np.float64(4.602)), ('L1H3', np.float64(-0.059), np.float64(-0.005), np.float64(1.194))]
L2H5 [('L0H0', np.float64(0.112), np.float64(0.008), np.float64(4.078)), ('L0H4', np.float64(0.11), np.float64(0.007), np.float64(1.759)), ('L0H5', np.float64(0.078), np.float64(-0.002), np.float64(4.482)), ('L0H1', np.float64(0.073), np.float64(0.001), np.float64(3.157)), ('L1H2', np.float64(0.007), np.float64(0.005), np.float64(1.03))]
L3H0 [('L0H4', np.float64(0.088), np.float64(0.006), np.float64(0.725)), ('L0H5', np.float64(0.078), np.float64(0.004), np.float64(1.848)), ('L0H1', np.float64(0.06), np.float64(0.003), np.float64(1.256)), ('L0H0', np.float64(0.057), np.float64(0.001), np.float64(1.653)), ('L0H2', np.float64(-0.042), np.float64(0.001), np.float64(0.309))]
L3H1 [('L2H1', np.float64(1.25), np.float64(0.041), np.float64(2.105)), ('L0H2', np.float64(1.173), np.float64(0.097), np.float64(0.376)), ('L1H2', np.float64(0.879), np.float64(0.033), np.float64(1.061)), ('L1H3', np.float64(0.768), np.float64(0.049), np.float64(1.36)), ('L1H1', np.float64(0.63), np.float64(0.012), np.float64(1.087))]
L3H2 [('L0H5', np.float64(0.101), np.float64(0.008), np.float64(3.049)), ('L0H0', np.float64(0.087), np.float64(0.006), np.float64(2.532)), ('L0H4', np.float64(0.082), np.float64(0.003), np.float64(1.052)), ('L0H2', np.float64(0.063), np.float64(0.0), np.float64(0.18)), ('L0H1', np.float64(0.059), np.float64(0.002), np.float64(1.86))]
L3H3 [('L0H5', np.float64(0.07), np.float64(0.002), np.float64(3.087)), ('L0H0', np.float64(0.06), np.float64(0.003), np.float64(2.578)), ('L0H4', np.float64(0.05), np.float64(0.002), np.float64(1.107)), ('L0H1', np.float64(0.03), np.float64(-0.001), np.float64(1.951)), ('L0H2', np.float64(0.024), np.float64(-0.004), np.float64(0.195))]
L3H4 [('L0H5', np.float64(0.164), np.float64(0.024), np.float64(2.969)), ('L0H4', np.float64(0.16), np.float64(0.019), np.float64(1.065)), ('L0H1', np.float64(0.142), np.float64(0.016), np.float64(1.879)), ('L0H0', np.float64(0.134), np.float64(0.015), np.float64(2.471)), ('L2H0', np.float64(0.015), np.float64(0.002), np.float64(1.46))]
L3H5 [('L2H3', np.float64(0.318), np.float64(0.009), np.float64(0.773)), ('L2H4', np.float64(0.228), np.float64(0.007), np.float64(0.678)), ('L2H1', np.float64(0.217), np.float64(0.005), np.float64(0.786)), ('L2H5', np.float64(0.198), np.float64(0.005), np.float64(0.689)), ('L1H0', np.float64(0.186), np.float64(0.006), np.float64(0.506))]
Pairwise cosine similarities of effective QK products, within each layer
Layer 0
[[ 1. 0.209 -0.016 -0.014 0.115 0.018]
[ 0.209 1. -0.004 -0.036 0.21 -0.008]
[-0.016 -0.004 1. 0.24 -0.006 -0.026]
[-0.014 -0.036 0.24 1. -0.054 -0.058]
[ 0.115 0.21 -0.006 -0.054 1. 0.172]
[ 0.018 -0.008 -0.026 -0.058 0.172 1. ]]
Layer 1
[[1. 0.578 0.416 0.448 0.455 0.464]
[0.578 1. 0.55 0.488 0.556 0.528]
[0.416 0.55 1. 0.608 0.667 0.564]
[0.448 0.488 0.608 1. 0.644 0.563]
[0.455 0.556 0.667 0.644 1. 0.577]
[0.464 0.528 0.564 0.563 0.577 1. ]]
Layer 2
[[1. 0.078 0.133 0.103 0.112 0.106]
[0.078 1. 0.093 0.086 0.077 0.07 ]
[0.133 0.093 1. 0.118 0.126 0.125]
[0.103 0.086 0.118 1. 0.12 0.113]
[0.112 0.077 0.126 0.12 1. 0.107]
[0.106 0.07 0.125 0.113 0.107 1. ]]
Layer 3
[[1. 0.071 0.044 0.036 0.075 0.03 ]
[0.071 1. 0.021 0.024 0.03 0.045]
[0.044 0.021 1. 0.094 0.103 0.054]
[0.036 0.024 0.094 1. 0.084 0.065]
[0.075 0.03 0.103 0.084 1. 0.049]
[0.03 0.045 0.054 0.065 0.049 1. ]]
Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 2 of at most 100:
```
All calculations below are matrix products or decompositions, not layer evaluations.
LAYER 0
L0H0 Q/K/V/O norms [3.305 4.18 6.039 6.666] effective QK norm/top/sym/trace [ 0.0546 0.0514 0.0233 -0.0047] query_bias_ratio 0.214 key_bias_norm 0.0136 QKev [ 0.007+0.j -0.004+0.j -0.002+0.j -0. +0.001j -0. -0.001j 0. +0.001j 0. -0.001j -0.001+0.001j]
L0H1 Q/K/V/O norms [3.09 3.658 5.382 6.025] effective QK norm/top/sym/trace [ 0.0295 0.025 0.0475 -0.014 ] query_bias_ratio 0.1243 key_bias_norm 0.0033 QKev [-0.004+0.001j -0.004-0.001j 0.002+0.j -0.001+0.001j -0.001-0.001j -0.001+0.j -0.001+0.001j -0.001-0.001j]
L0H2 Q/K/V/O norms [30.818 28.3 7.765 8.915] effective QK norm/top/sym/trace [ 4.0688 1.4079 -0.0446 1.199 ] query_bias_ratio 0.0589 key_bias_norm 0.1428 QKev [ 0.632+0.813j 0.632-0.813j -0.231+0.927j -0.231-0.927j 0.332+0.871j 0.332-0.871j -0.777+0.334j -0.777-0.334j]
L0H3 Q/K/V/O norms [6.821 7.243 5.434 5.029] effective QK norm/top/sym/trace [0.3945 0.3588 0.4782 0.1968] query_bias_ratio 0.1771 key_bias_norm 0.0535 QKev [ 0.262+0.j 0.078+0.j 0.01 +0.031j 0.01 -0.031j -0.017+0.005j -0.017-0.005j -0.017+0.j -0.015+0.003j]
L0H4 Q/K/V/O norms [3.971 4.995 5.208 5.587] effective QK norm/top/sym/trace [ 0.1062 0.089 0.004 -0.0707] query_bias_ratio 0.1505 key_bias_norm 0.0112 QKev [-0.028+0.028j -0.028-0.028j -0.008+0.j -0.002+0.j -0.001+0.001j -0.001-0.001j -0.002+0.j 0.001+0.001j]
L0H5 Q/K/V/O norms [3.19 3.802 6.143 7.162] effective QK norm/top/sym/trace [ 0.0775 0.0759 0.1603 -0.0347] query_bias_ratio 0.1849 key_bias_norm 0.0162 QKev [-0.031+0.j 0.001+0.002j 0.001-0.002j -0.001+0.j -0.001-0.j -0.001+0.001j -0.001-0.001j 0.001+0.001j]
LAYER 1
L1H0 Q/K/V/O norms [10.716 9.827 11.156 9.984] effective QK norm/top/sym/trace [ 0.8864 0.5969 0.0086 -0.6436] query_bias_ratio 0.7821 key_bias_norm 1.3572 QKev [ 0.028+0.106j 0.028-0.106j -0.078+0.j -0.066+0.028j -0.066-0.028j -0.055+0.007j -0.055-0.007j -0.047+0.j ]
L1H1 Q/K/V/O norms [13.292 12.068 11.72 11.303] effective QK norm/top/sym/trace [ 1.3146 0.8811 0.0186 -1.1603] query_bias_ratio 0.8034 key_bias_norm 2.2848 QKev [ 0.07 +0.191j 0.07 -0.191j -0.153+0.j -0.107+0.024j -0.107-0.024j -0.108+0.006j -0.108-0.006j -0.086+0.j ]
L1H2 Q/K/V/O norms [14.476 13.092 10.668 10.272] effective QK norm/top/sym/trace [ 1.5491 0.9943 0.0179 -1.6165] query_bias_ratio 0.6959 key_bias_norm 2.5879 QKev [ 0.097+0.246j 0.097-0.246j -0.168+0.j -0.141+0.064j -0.141-0.064j -0.135+0.009j -0.135-0.009j -0.104+0.03j ]
L1H3 Q/K/V/O norms [13.358 12.153 10.904 9.928] effective QK norm/top/sym/trace [ 1.3742 0.9188 0.0172 -1.1551] query_bias_ratio 0.7616 key_bias_norm 2.3511 QKev [ 0.094+0.203j 0.094-0.203j -0.175+0.j -0.113+0.008j -0.113-0.008j -0.107+0.j -0.04 +0.091j -0.04 -0.091j]
L1H4 Q/K/V/O norms [13.518 12.129 11.148 10.794] effective QK norm/top/sym/trace [ 1.4497 0.9476 0.0108 -1.2432] query_bias_ratio 0.711 key_bias_norm 2.2902 QKev [ 0.062+0.214j 0.062-0.214j -0.135+0.01j -0.135-0.01j -0.127+0.j -0.099+0.j -0.081+0.053j -0.081-0.053j]
L1H5 Q/K/V/O norms [13.486 12.132 11.376 11.136] effective QK norm/top/sym/trace [ 1.3363 0.8925 -0.0027 -1.0127] query_bias_ratio 0.6733 key_bias_norm 1.7526 QKev [ 0.088+0.221j 0.088-0.221j -0.121+0.012j -0.121-0.012j -0.1 +0.004j -0.1 -0.004j -0.074+0.055j -0.074-0.055j]
LAYER 2
L2H0 Q/K/V/O norms [16.822 16.208 13.115 12.942] effective QK norm/top/sym/trace [ 2.4375 0.6943 -0.0335 -0.0216] query_bias_ratio 0.3088 key_bias_norm 1.6991 QKev [ 0.053+0.306j 0.053-0.306j 0.036+0.183j 0.036-0.183j 0.118+0.08j 0.118-0.08j -0.077+0.098j -0.077-0.098j]
L2H1 Q/K/V/O norms [15.963 15.287 13.768 13.525] effective QK norm/top/sym/trace [2.0677 0.5267 0.0101 0.53 ] query_bias_ratio 0.2076 key_bias_norm 0.7446 QKev [ 0.027+0.144j 0.027-0.144j 0.126+0.011j 0.126-0.011j 0.109+0.037j 0.109-0.037j -0.084+0.071j -0.084-0.071j]
L2H2 Q/K/V/O norms [13.97 13.239 14.567 14.24 ] effective QK norm/top/sym/trace [ 1.6127 0.6069 -0.0046 -1.1051] query_bias_ratio 0.2312 key_bias_norm 0.7554 QKev [ 0.037+0.175j 0.037-0.175j -0.106+0.j 0.067+0.055j 0.067-0.055j -0.084+0.015j -0.084-0.015j -0.076-0.014j]
L2H3 Q/K/V/O norms [16.121 15.554 13.498 13.391] effective QK norm/top/sym/trace [ 2.0862 0.5616 -0.0268 -0.7372] query_bias_ratio 0.2623 key_bias_norm 1.4625 QKev [ 0.072+0.307j 0.072-0.307j -0.076+0.084j -0.076-0.084j -0.102+0.042j -0.102-0.042j -0.104+0.005j -0.104-0.005j]
L2H4 Q/K/V/O norms [16.876 16.188 13.241 13.143] effective QK norm/top/sym/trace [ 2.3438 0.6278 -0.0248 -0.4747] query_bias_ratio 0.2685 key_bias_norm 1.5774 QKev [ 0.061+0.303j 0.061-0.303j 0.077+0.102j 0.077-0.102j -0.081+0.091j -0.081-0.091j -0.118+0.j -0.045+0.105j]
L2H5 Q/K/V/O norms [16.445 15.839 13.339 13.161] effective QK norm/top/sym/trace [ 2.2503 0.6375 -0.0283 -0.6957] query_bias_ratio 0.2779 key_bias_norm 1.5086 QKev [ 0.046+0.302j 0.046-0.302j -0.141+0.j -0.071-0.108j -0.071+0.108j -0.119+0.037j -0.119-0.037j -0.068+0.099j]
LAYER 3
L3H0 Q/K/V/O norms [15.747 15.258 14.874 14.87 ] effective QK norm/top/sym/trace [2.2322 0.4514 0.0503 3.4122] query_bias_ratio 0.1057 key_bias_norm 0.2581 QKev [0.182+0.j 0.141+0.078j 0.141-0.078j 0.158+0.019j 0.158-0.019j 0.143+0.056j 0.143-0.056j 0.141+0.036j]
L3H1 Q/K/V/O norms [14.22 14.197 17.885 18.331] effective QK norm/top/sym/trace [1.8295 0.3275 0.2114 6.3101] query_bias_ratio 0.1322 key_bias_norm 0.2408 QKev [0.174-0.06j 0.174+0.06j 0.171-0.067j 0.171+0.067j 0.149-0.082j 0.149+0.082j 0.164-0.031j 0.164+0.031j]
L3H2 Q/K/V/O norms [12.743 12.446 15.523 15.585] effective QK norm/top/sym/trace [ 1.505 0.4702 -0.0036 0.2216] query_bias_ratio 0.124 key_bias_norm 0.2895 QKev [ 0.03 +0.134j 0.03 -0.134j 0.081+0.038j 0.081-0.038j -0.035+0.082j -0.035-0.082j 0.082+0.j -0.077+0.j ]
L3H3 Q/K/V/O norms [13.283 13.057 15.732 16.118] effective QK norm/top/sym/trace [ 1.5985 0.4973 -0.0166 0.1116] query_bias_ratio 0.121 key_bias_norm 0.3603 QKev [ 0.023+0.159j 0.023-0.159j 0.099+0.j -0.023+0.095j -0.023-0.095j 0.076+0.036j 0.076-0.036j 0.078+0.j ]
L3H4 Q/K/V/O norms [14.356 13.874 15.175 15.3 ] effective QK norm/top/sym/trace [ 1.8555 0.5287 -0.0005 0.9035] query_bias_ratio 0.1217 key_bias_norm 0.3072 QKev [0.068+0.092j 0.068-0.092j 0.002+0.108j 0.002-0.108j 0.098+0.04j 0.098-0.04j 0.102+0.j 0.07 +0.066j]
L3H5 Q/K/V/O norms [ 9.514 9.231 17.993 18.512] effective QK norm/top/sym/trace [0.8339 0.2591 0.0242 0.1007] query_bias_ratio 0.1537 key_bias_norm 0.1645 QKev [0.075+0.035j 0.075-0.035j 0.078+0.j 0.02 +0.048j 0.02 -0.048j 0.023+0.036j 0.023-0.036j 0.038+0.011j]
SINGULAR VALUES: first 45 for selected matrices
W_q.0 [8.252 8.223 7.963 7.819 7.68 7.642 7.519 7.205 6.948 6.819 6.388 6.165 5.801 5.555 5.13 4.947 4.854 4.52 4.268 3.951 3.591 3.527 3.285 3.125 3.026 2.928 2.707 2.511 2.471
2.336 2.249 2.136 2.089 2.058 1.914 1.792 1.755 1.679 1.536 1.513 1.475 1.455 1.327 1.31 1.282]
W_k.0 [7.646 7.436 7.33 6.912 6.829 6.672 6.472 6.446 6.393 6.118 6.009 5.943 5.629 5.505 4.99 4.901 4.698 4.293 4.066 4.018 3.708 3.529 3.343 3.068 2.987 2.874 2.696 2.586 2.412
2.262 2.216 2.11 2.032 1.876 1.744 1.733 1.697 1.666 1.547 1.514 1.441 1.402 1.386 1.334 1.29 ]
W_v.0 [4.339 3.529 3.27 3.166 2.982 2.647 2.589 2.552 2.436 2.381 2.256 2.162 2.13 2.103 2.042 2.017 2.003 1.973 1.928 1.897 1.843 1.787 1.711 1.665 1.628 1.564 1.558 1.441 1.4
1.377 1.347 1.267 1.222 1.159 1.086 1.046 1.016 0.999 0.945 0.918 0.902 0.894 0.875 0.87 0.841]
W_o.0 [7.103 6.607 4.268 2.77 2.503 2.285 2.264 2.218 2.151 2.139 2.105 2.08 2.059 2.026 2.003 1.987 1.928 1.89 1.836 1.825 1.804 1.731 1.72 1.694 1.685 1.581 1.543 1.518 1.507
1.428 1.4 1.349 1.31 1.299 1.248 1.223 1.202 1.187 1.17 1.163 1.142 1.109 1.081 1.068 1.036]
W_fc.0 [18.176 11.018 8.737 7.452 7.212 7.062 7.011 6.62 6.57 6.42 6.343 6.29 6.137 6.043 5.974 5.918 5.807 5.779 5.583 5.513 5.5 5.36 5.16 5.12 5.086
5.032 4.899 4.879 4.819 4.732 4.601 4.533 4.479 4.441 4.268 4.198 4.031 3.974 3.869 3.824 3.72 3.683 3.646 3.574 3.516]
W_proj.0 [9.826 7.351 6.265 5.869 5.797 5.657 5.516 5.45 5.333 5.242 5.197 5.094 4.98 4.904 4.863 4.819 4.793 4.689 4.627 4.595 4.557 4.523 4.411 4.378 4.283 4.216 4.143 4.125 4.048
4.025 3.894 3.847 3.808 3.792 3.764 3.711 3.685 3.636 3.595 3.525 3.499 3.479 3.469 3.422 3.39 ]
W_proj.1 [8.193 7.894 7.199 6.997 6.671 6.433 6.274 6.133 6.099 6.021 5.942 5.865 5.829 5.789 5.698 5.675 5.646 5.629 5.581 5.564 5.553 5.538 5.528 5.477 5.456 5.447 5.438 5.403 5.399
5.38 5.367 5.351 5.332 5.327 5.314 5.296 5.252 5.235 5.219 5.208 5.197 5.183 5.174 5.17 5.146]
W_proj.2 [9.404 9.187 8.723 8.47 8.04 7.795 7.614 7.45 7.392 7.253 7.223 7.148 7.082 6.972 6.829 6.733 6.637 6.547 6.46 6.404 6.374 6.307 6.209 6.181 6.12 6.091 5.989 5.9 5.88
5.833 5.758 5.703 5.685 5.645 5.588 5.559 5.491 5.478 5.467 5.429 5.406 5.4 5.378 5.351 5.335]
W_proj.3 [26.309 20.685 19.699 19.261 18.594 18.197 17.862 17.667 17.044 16.749 16.039 15.151 14.756 14.326 14.05 13.5 13.293 12.531 12.401 11.958 11.74 11.594 10.367 8.351 8.064
7.914 7.461 7.166 5.833 4.072 3.221 3.139 2.842 2.8 2.687 2.571 2.403 2.328 2.244 2.185 2.181 2.116 2.081 2.053 1.996]
L0H2 left/right-space joint singular values [1.406 1.4046 1.4043 1.4001 1.3966 1.395 1.3932 1.3834 1.3806 1.373 1.3689 1.3616 1.3366 1.2648 1.2248 1.0401 0.9582 0.7071 0.6326 0.4621 0.382 0.3551 0.3389 0.3066 0.2937
0.2429 0.2324 0.2225 0.1996 0.1674 0.1643 0.1521]
Candidate position-space dimension 16
layer 0 Q/K/both energy: [[0.7 0.064 0.042]
[0.491 0.202 0.109]
[0.902 0.897 0.83 ]
[0.637 0.813 0.554]
[0.196 0.135 0.026]
[0.782 0.164 0.13 ]]
layer 1 Q/K/both energy: [[0.054 0.136 0.006]
[0.062 0.12 0.007]
[0.074 0.121 0.01 ]
[0.068 0.129 0.009]
[0.074 0.137 0.01 ]
[0.071 0.125 0.009]]
layer 2 Q/K/both energy: [[0.037 0.052 0.002]
[0.035 0.043 0.001]
[0.043 0.063 0.003]
[0.04 0.052 0.003]
[0.041 0.054 0.002]
[0.042 0.052 0.002]]
layer 3 Q/K/both energy: [[0.041 0.041 0.002]
[0.036 0.043 0.003]
[0.042 0.051 0.002]
[0.041 0.05 0.003]
[0.041 0.044 0.002]
[0.036 0.047 0.001]]
Candidate position-space dimension 24
layer 0 Q/K/both energy: [[0.745 0.159 0.118]
[0.56 0.326 0.202]
[0.969 0.97 0.947]
[0.805 0.837 0.706]
[0.487 0.334 0.172]
[0.84 0.473 0.398]]
layer 1 Q/K/both energy: [[0.076 0.229 0.018]
[0.082 0.205 0.019]
[0.097 0.214 0.024]
[0.098 0.22 0.025]
[0.099 0.242 0.027]
[0.095 0.216 0.022]]
layer 2 Q/K/both energy: [[0.059 0.082 0.007]
[0.052 0.065 0.004]
[0.065 0.106 0.008]
[0.061 0.082 0.007]
[0.066 0.084 0.007]
[0.064 0.081 0.007]]
layer 3 Q/K/both energy: [[0.059 0.06 0.004]
[0.053 0.062 0.005]
[0.064 0.072 0.005]
[0.064 0.073 0.006]
[0.06 0.062 0.004]
[0.06 0.07 0.004]]
Candidate position-space dimension 32
layer 0 Q/K/both energy: [[0.821 0.433 0.363]
[0.618 0.468 0.308]
[0.985 0.985 0.976]
[0.851 0.865 0.756]
[0.598 0.464 0.288]
[0.895 0.562 0.506]]
layer 1 Q/K/both energy: [[0.102 0.328 0.033]
[0.121 0.295 0.039]
[0.127 0.311 0.046]
[0.127 0.32 0.045]
[0.125 0.346 0.047]
[0.125 0.318 0.042]]
layer 2 Q/K/both energy: [[0.083 0.112 0.013]
[0.069 0.09 0.008]
[0.09 0.15 0.017]
[0.084 0.117 0.014]
[0.087 0.115 0.014]
[0.087 0.113 0.014]]
layer 3 Q/K/both energy: [[0.078 0.081 0.007]
[0.069 0.083 0.008]
[0.085 0.101 0.01 ]
[0.087 0.101 0.011]
[0.082 0.087 0.008]
[0.077 0.089 0.007]]
KEY-COMPOSITION CANDIDATES: QK_l @ OV_earlier
For each head: strongest positive trace/Frobenius scores; entries=(earlier head, trace/Fnorm, transpose symmetry, Fnorm).
L1H0 [('L0H0', np.float64(0.072), np.float64(0.007), np.float64(4.379)), ('L0H4', np.float64(0.062), np.float64(0.004), np.float64(1.866)), ('L0H1', np.float64(0.032), np.float64(0.006), np.float64(3.264)), ('L0H5', np.float64(0.007), np.float64(0.001), np.float64(4.613)), ('L0H2', np.float64(-0.113), np.float64(0.01), np.float64(0.108))]
L1H1 [('L0H4', np.float64(0.102), np.float64(0.007), np.float64(2.573)), ('L0H0', np.float64(0.092), np.float64(0.012), np.float64(5.94)), ('L0H1', np.float64(0.066), np.float64(0.009), np.float64(4.607)), ('L0H5', np.float64(0.033), np.float64(0.002), np.float64(6.86)), ('L0H2', np.float64(-0.003), np.float64(0.036), np.float64(0.213))]
L1H2 [('L0H2', np.float64(0.253), np.float64(0.012), np.float64(0.227)), ('L0H4', np.float64(0.112), np.float64(0.013), np.float64(3.017)), ('L0H0', np.float64(0.101), np.float64(0.017), np.float64(6.982)), ('L0H1', np.float64(0.084), np.float64(0.016), np.float64(5.425)), ('L0H5', np.float64(0.04), np.float64(0.007), np.float64(7.968))]
L1H3 [('L0H2', np.float64(0.233), np.float64(0.014), np.float64(0.22)), ('L0H4', np.float64(0.123), np.float64(0.013), np.float64(2.721)), ('L0H0', np.float64(0.116), np.float64(0.018), np.float64(6.348)), ('L0H1', np.float64(0.094), np.float64(0.013), np.float64(4.85)), ('L0H5', np.float64(0.05), np.float64(0.006), np.float64(7.229))]
L1H4 [('L0H2', np.float64(0.251), np.float64(0.007), np.float64(0.196)), ('L0H4', np.float64(0.163), np.float64(0.02), np.float64(3.015)), ('L0H0', np.float64(0.149), np.float64(0.023), np.float64(7.072)), ('L0H1', np.float64(0.134), np.float64(0.018), np.float64(5.351)), ('L0H5', np.float64(0.081), np.float64(0.006), np.float64(7.691))]
L1H5 [('L0H0', np.float64(0.078), np.float64(0.019), np.float64(6.069)), ('L0H2', np.float64(0.077), np.float64(0.004), np.float64(0.267)), ('L0H4', np.float64(0.076), np.float64(0.014), np.float64(2.605)), ('L0H1', np.float64(0.036), np.float64(0.019), np.float64(4.682)), ('L0H5', np.float64(-0.02), np.float64(0.012), np.float64(6.959))]
L2H0 [('L0H4', np.float64(0.096), np.float64(0.006), np.float64(1.879)), ('L0H0', np.float64(0.08), np.float64(0.007), np.float64(4.368)), ('L0H5', np.float64(0.071), np.float64(0.002), np.float64(5.026)), ('L0H1', np.float64(0.062), np.float64(0.003), np.float64(3.425)), ('L0H2', np.float64(-0.08), np.float64(-0.001), np.float64(0.388))]
L2H1 [('L0H4', np.float64(0.076), np.float64(0.003), np.float64(1.145)), ('L0H5', np.float64(0.071), np.float64(0.0), np.float64(3.117)), ('L0H0', np.float64(0.071), np.float64(0.001), np.float64(2.597)), ('L1H3', np.float64(0.063), np.float64(0.005), np.float64(1.019)), ('L0H1', np.float64(0.046), np.float64(0.002), np.float64(2.064))]
L2H2 [('L1H4', np.float64(0.174), np.float64(0.009), np.float64(0.748)), ('L0H4', np.float64(0.136), np.float64(0.013), np.float64(1.71)), ('L0H0', np.float64(0.11), np.float64(0.013), np.float64(3.936)), ('L1H3', np.float64(0.091), np.float64(0.001), np.float64(0.713)), ('L0H1', np.float64(0.087), np.float64(0.003), np.float64(3.13))]
L2H3 [('L0H4', np.float64(0.116), np.float64(0.008), np.float64(1.681)), ('L0H0', np.float64(0.111), np.float64(0.009), np.float64(3.904)), ('L0H5', np.float64(0.099), np.float64(0.003), np.float64(4.258)), ('L0H1', np.float64(0.08), np.float64(0.004), np.float64(3.015)), ('L0H2', np.float64(0.054), np.float64(-0.001), np.float64(0.286))]
L2H4 [('L0H4', np.float64(0.152), np.float64(0.016), np.float64(1.865)), ('L0H0', np.float64(0.144), np.float64(0.019), np.float64(4.344)), ('L0H1', np.float64(0.108), np.float64(0.006), np.float64(3.348)), ('L0H5', np.float64(0.1), np.float64(-0.001), np.float64(4.602)), ('L1H3', np.float64(-0.059), np.float64(-0.005), np.float64(1.194))]
L2H5 [('L0H0', np.float64(0.112), np.float64(0.008), np.float64(4.078)), ('L0H4', np.float64(0.11), np.float64(0.007), np.float64(1.759)), ('L0H5', np.float64(0.078), np.float64(-0.002), np.float64(4.482)), ('L0H1', np.float64(0.073), np.float64(0.001), np.float64(3.157)), ('L1H2', np.float64(0.007), np.float64(0.005), np.float64(1.03))]
L3H0 [('L0H4', np.float64(0.088), np.float64(0.006), np.float64(0.725)), ('L0H5', np.float64(0.078), np.float64(0.004), np.float64(1.848)), ('L0H1', np.float64(0.06), np.float64(0.003), np.float64(1.256)), ('L0H0', np.float64(0.057), np.float64(0.001), np.float64(1.653)), ('L0H2', np.float64(-0.042), np.float64(0.001), np.float64(0.309))]
L3H1 [('L2H1', np.float64(1.25), np.float64(0.041), np.float64(2.105)), ('L0H2', np.float64(1.173), np.float64(0.097), np.float64(0.376)), ('L1H2', np.float64(0.879), np.float64(0.033), np.float64(1.061)), ('L1H3', np.float64(0.768), np.float64(0.049), np.float64(1.36)), ('L1H1', np.float64(0.63), np.float64(0.012), np.float64(1.087))]
L3H2 [('L0H5', np.float64(0.101), np.float64(0.008), np.float64(3.049)), ('L0H0', np.float64(0.087), np.float64(0.006), np.float64(2.532)), ('L0H4', np.float64(0.082), np.float64(0.003), np.float64(1.052)), ('L0H2', np.float64(0.063), np.float64(0.0), np.float64(0.18)), ('L0H1', np.float64(0.059), np.float64(0.002), np.float64(1.86))]
L3H3 [('L0H5', np.float64(0.07), np.float64(0.002), np.float64(3.087)), ('L0H0', np.float64(0.06), np.float64(0.003), np.float64(2.578)), ('L0H4', np.float64(0.05), np.float64(0.002), np.float64(1.107)), ('L0H1', np.float64(0.03), np.float64(-0.001), np.float64(1.951)), ('L0H2', np.float64(0.024), np.float64(-0.004), np.float64(0.195))]
L3H4 [('L0H5', np.float64(0.164), np.float64(0.024), np.float64(2.969)), ('L0H4', np.float64(0.16), np.float64(0.019), np.float64(1.065)), ('L0H1', np.float64(0.142), np.float64(0.016), np.float64(1.879)), ('L0H0', np.float64(0.134), np.float64(0.015), np.float64(2.471)), ('L2H0', np.float64(0.015), np.float64(0.002), np.float64(1.46))]
L3H5 [('L2H3', np.float64(0.318), np.float64(0.009), np.float64(0.773)), ('L2H4', np.float64(0.228), np.float64(0.007), np.float64(0.678)), ('L2H1', np.float64(0.217), np.float64(0.005), np.float64(0.786)), ('L2H5', np.float64(0.198), np.float64(0.005), np.float64(0.689)), ('L1H0', np.float64(0.186), np.float64(0.006), np.float64(0.506))]
Pairwise cosine similarities of effective QK products, within each layer
Layer 0
[[ 1. 0.209 -0.016 -0.014 0.115 0.018]
[ 0.209 1. -0.004 -0.036 0.21 -0.008]
[-0.016 -0.004 1. 0.24 -0.006 -0.026]
[-0.014 -0.036 0.24 1. -0.054 -0.058]
[ 0.115 0.21 -0.006 -0.054 1. 0.172]
[ 0.018 -0.008 -0.026 -0.058 0.172 1. ]]
Layer 1
[[1. 0.578 0.416 0.448 0.455 0.464]
[0.578 1. 0.55 0.488 0.556 0.528]
[0.416 0.55 1. 0.608 0.667 0.564]
[0.448 0.488 0.608 1. 0.644 0.563]
[0.455 0.556 0.667 0.644 1. 0.577]
[0.464 0.528 0.564 0.563 0.577 1. ]]
Layer 2
[[1. 0.078 0.133 0.103 0.112 0.106]
[0.078 1. 0.093 0.086 0.077 0.07 ]
[0.133 0.093 1. 0.118 0.126 0.125]
[0.103 0.086 0.118 1. 0.12 0.113]
[0.112 0.077 0.126 0.12 1. 0.107]
[0.106 0.07 0.125 0.113 0.107 1. ]]
Layer 3
[[1. 0.071 0.044 0.036 0.075 0.03 ]
[0.071 1. 0.021 0.024 0.03 0.045]
[0.044 0.021 1. 0.094 0.103 0.054]
[0.036 0.024 0.094 1. 0.084 0.065]
[0.075 0.03 0.103 0.084 1. 0.049]
[0.03 0.045 0.054 0.065 0.049 1. ]]
```Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(2)
np.set_printoptions(precision=4, suppress=True, linewidth=180)
d=384
P=np.eye(d)-np.ones((d,d))/d
A={}; B={}; QB={}; KB={}
for l in range(4):
g=W[f"ln1_w.{l}"].astype(float)
b=W[f"ln1_b.{l}"].astype(float)
for h in range(6):
s=slice(64*h,64*(h+1))
q,k,v=[W[f"W_{t}.{l}"][s].astype(float) for t in ["q","k","v"]]
o=P@W[f"W_o.{l}"][:,s].astype(float)
qg,kg,vg=[(m*g)@P for m in [q,k,v]]
A[l,h]=qg.T@kg/8
B[l,h]=o@vg
KB[l,h]=kg.T@(q@b)/8
QB[l,h]=qg.T@(k@b)/8
print("L0 OV spectra")
for h in range(6):
s=np.linalg.svd(B[0,h],compute_uv=False)
print(h,np.round(s[:32],4))
print("\nL0 OV pairwise cosines")
z=np.array([B[0,h].ravel()/np.linalg.norm(B[0,h]) for h in range(6)])
print(np.round(z@z.T,3))
broad=[0,1,4,5]
ub,sb,_=np.linalg.svd(np.concatenate([B[0,h] for h in broad],axis=1),full_matrices=False)
print("\nJoint broad-head output spectrum",np.round(sb[:35],4))
for r in [2,4,8,16]:
T=ub[:,:r]
print("Broad-output subspace dimension",r)
for l in range(1,4):
print("layer",l,"Q/K/both energy",np.round([
[np.linalg.norm(T.T@A[l,h])**2/np.linalg.norm(A[l,h])**2,
np.linalg.norm(A[l,h]@T)**2/np.linalg.norm(A[l,h])**2,
np.linalg.norm(T.T@A[l,h]@T)**2/np.linalg.norm(A[l,h])**2]
for h in range(6)],3))
print("\nKey-bias vector cosines within layers")
for l in range(4):
z=np.array([KB[l,h]/np.linalg.norm(KB[l,h]) for h in range(6)])
print(l,np.round(z@z.T,3))
print("\nL0H2 nonzero effective QK eigenvalues, sorted by phase (positive imaginary part)")
ev=np.linalg.eigvals(A[0,2])
ev=ev[(ev.imag>1e-5)&(abs(ev)>.03)]
ev=ev[np.argsort(np.angle(ev))]
print("magnitude, phase",np.round(np.array([abs(ev),np.angle(ev)]).T,4))
# Input directions read by L0H2's value matrix; no activations.
g=W["ln1_w.0"].astype(float)
vg=(W["W_v.0"][128:192].astype(float)*g)@P
_,sv,tv=np.linalg.svd(vg,full_matrices=False)
print("\nL0H2 effective V singular values",np.round(sv,4))
for r in [16,24,28,40]:
T=tv[:r].T
print("V-input subspace dimension",r)
for l in range(1,4):
out=[]
for h in range(6):
c=T.T@A[l,h]@B[0,2]@T
n=np.linalg.norm(c)
ev=np.linalg.eigvalsh((c+c.T)/2)
out.append([np.trace(c)/max(n,1e-12),
np.sum(c*c.T)/max(n*n,1e-12),
np.sum(ev[ev>0])/max(np.sum(abs(ev)),1e-12)])
print("layer",l,"composition trace/F, symmetry, positive-symmetric-mass",np.round(out,3))
print("\nMLP matrix-product spectra (not applying ReLU)")
for l in range(4):
f=(W[f"W_fc.{l}"].astype(float)*W[f"ln2_w.{l}"])@P
o=P@W[f"W_proj.{l}"].astype(float)
m=o@f
s=np.linalg.svd(m,compute_uv=False)
print(l,"norm",round(np.linalg.norm(m),3),"trace",round(np.trace(m),3),
"topSV",np.round(s[:35],3))
```{
"run": 3,
"turn": "t003",
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(2)\nnp.set_printoptions(precision=4, suppress=True, linewidth=180)\n\nd=384\nP=np.eye(d)-np.ones((d,d))/d\nA={}; B={}; QB={}; KB={}\nfor l in range(4):\n g=W[f\"ln1_w.{l}\"].astype(float)\n b=W[f\"ln1_b.{l}\"].astype(float)\n for h in range(6):\n s=slice(64*h,64*(h+1))\n q,k,v=[W[f\"W_{t}.{l}\"][s].astype(float) for t in [\"q\",\"k\",\"v\"]]\n o=P@W[f\"W_o.{l}\"][:,s].astype(float)\n qg,kg,vg=[(m*g)@P for m in [q,k,v]]\n A[l,h]=qg.T@kg/8\n B[l,h]=o@vg\n KB[l,h]=kg.T@(q@b)/8\n QB[l,h]=qg.T@(k@b)/8\n\nprint(\"L0 OV spectra\")\nfor h in range(6):\n s=np.linalg.svd(B[0,h],compute_uv=False)\n print(h,np.round(s[:32],4))\n\nprint(\"\\nL0 OV pairwise cosines\")\nz=np.array([B[0,h].ravel()/np.linalg.norm(B[0,h]) for h in range(6)])\nprint(np.round(z@z.T,3))\n\nbroad=[0,1,4,5]\nub,sb,_=np.linalg.svd(np.concatenate([B[0,h] for h in broad],axis=1),full_matrices=False)\nprint(\"\\nJoint broad-head output spectrum\",np.round(sb[:35],4))\nfor r in [2,4,8,16]:\n T=ub[:,:r]\n print(\"Broad-output subspace dimension\",r)\n for l in range(1,4):\n print(\"layer\",l,\"Q/K/both energy\",np.round([\n [np.linalg.norm(T.T@A[l,h])**2/np.linalg.norm(A[l,h])**2,\n np.linalg.norm(A[l,h]@T)**2/np.linalg.norm(A[l,h])**2,\n np.linalg.norm(T.T@A[l,h]@T)**2/np.linalg.norm(A[l,h])**2]\n for h in range(6)],3))\n\nprint(\"\\nKey-bias vector cosines within layers\")\nfor l in range(4):\n z=np.array([KB[l,h]/np.linalg.norm(KB[l,h]) for h in range(6)])\n print(l,np.round(z@z.T,3))\n\nprint(\"\\nL0H2 nonzero effective QK eigenvalues, sorted by phase (positive imaginary part)\")\nev=np.linalg.eigvals(A[0,2])\nev=ev[(ev.imag>1e-5)&(abs(ev)>.03)]\nev=ev[np.argsort(np.angle(ev))]\nprint(\"magnitude, phase\",np.round(np.array([abs(ev),np.angle(ev)]).T,4))\n\n# Input directions read by L0H2's value matrix; no activations.\ng=W[\"ln1_w.0\"].astype(float)\nvg=(W[\"W_v.0\"][128:192].astype(float)*g)@P\n_,sv,tv=np.linalg.svd(vg,full_matrices=False)\nprint(\"\\nL0H2 effective V singular values\",np.round(sv,4))\nfor r in [16,24,28,40]:\n T=tv[:r].T\n print(\"V-input subspace dimension\",r)\n for l in range(1,4):\n out=[]\n for h in range(6):\n c=T.T@A[l,h]@B[0,2]@T\n n=np.linalg.norm(c)\n ev=np.linalg.eigvalsh((c+c.T)/2)\n out.append([np.trace(c)/max(n,1e-12),\n np.sum(c*c.T)/max(n*n,1e-12),\n np.sum(ev[ev>0])/max(np.sum(abs(ev)),1e-12)])\n print(\"layer\",l,\"composition trace/F, symmetry, positive-symmetric-mass\",np.round(out,3))\n\nprint(\"\\nMLP matrix-product spectra (not applying ReLU)\")\nfor l in range(4):\n f=(W[f\"W_fc.{l}\"].astype(float)*W[f\"ln2_w.{l}\"])@P\n o=P@W[f\"W_proj.{l}\"].astype(float)\n m=o@f\n s=np.linalg.svd(m,compute_uv=False)\n print(l,\"norm\",round(np.linalg.norm(m),3),\"trace\",round(np.trace(m),3),\n \"topSV\",np.round(s[:35],3))",
"out": "L0 OV spectra\n0 [8.0675 3.3266 2.0965 0.6725 0.5605 0.3138 0.2573 0.1923 0.1454 0.1409 0.1155 0.1128 0.1006 0.0849 0.0811 0.0724 0.0712 0.0656 0.0612 0.0571 0.0528 0.0489 0.047 0.0456 0.043\n 0.0395 0.0389 0.0361 0.0352 0.0335 0.0316 0.0304]\n1 [5.6578 3.8262 1.0639 0.5023 0.4283 0.2699 0.2168 0.1662 0.1512 0.112 0.0944 0.0923 0.0829 0.0769 0.0712 0.0676 0.0621 0.0566 0.0552 0.0525 0.0499 0.0447 0.0435 0.0419 0.0384\n 0.0362 0.0357 0.0323 0.0321 0.031 0.0302 0.0285]\n2 [0.7872 0.718 0.682 0.6712 0.6362 0.6009 0.5687 0.5579 0.55 0.5221 0.5148 0.4858 0.4799 0.4694 0.4386 0.4206 0.4004 0.3928 0.3466 0.3042 0.2771 0.2754 0.2554 0.2285 0.2179\n 0.2048 0.1813 0.1699 0.1626 0.1591 0.1313 0.1181]\n3 [1.0317 0.964 0.6167 0.5045 0.4475 0.2267 0.2014 0.1834 0.1719 0.1475 0.1364 0.117 0.1057 0.093 0.0874 0.0833 0.0765 0.065 0.0612 0.0591 0.0545 0.0495 0.0465 0.0458 0.0394\n 0.0382 0.0359 0.0322 0.0315 0.0298 0.0284 0.0259]\n4 [3.3943 1.6416 1.0818 0.6072 0.3784 0.3273 0.2313 0.2045 0.1705 0.1412 0.1263 0.1156 0.1029 0.1003 0.0812 0.0771 0.0741 0.0648 0.0593 0.0572 0.0538 0.0494 0.0464 0.0439 0.0401\n 0.0383 0.0367 0.0346 0.0334 0.0312 0.0306 0.0295]\n5 [8.3281 3.6039 1.6786 0.7676 0.5104 0.3903 0.3443 0.3045 0.2473 0.1927 0.1667 0.1265 0.1142 0.1088 0.0982 0.0936 0.0851 0.0714 0.0675 0.0611 0.06 0.0574 0.0562 0.0517 0.0493\n 0.0461 0.0436 0.0403 0.0381 0.0364 0.0343 0.032 ]\n\nL0 OV pairwise cosines\n[[1. 0.885 0.008 0.164 0.903 0.724]\n [0.885 1. 0.006 0.195 0.889 0.802]\n [0.008 0.006 1. 0.012 0.006 0.017]\n [0.164 0.195 0.012 1. 0.195 0.156]\n [0.903 0.889 0.006 0.195 1. 0.728]\n [0.724 0.802 0.017 0.156 0.728 1. ]]\n\nJoint broad-head output spectrum [11.9885 8.4303 3.3621 1.3104 0.9989 0.8608 0.73 0.6052 0.566 0.54 0.4823 0.4356 0.3927 0.3667 0.3463 0.3207 0.3058 0.2952 0.262 0.246 0.2384 0.215\n 0.2062 0.1875 0.1871 0.1741 0.171 0.1613 0.1555 0.153 0.1481 0.1405 0.1331 0.1328 0.125 ]\nBroad-output subspace dimension 2\nlayer 1 Q/K/both energy [[0.061 0.575 0.047]\n [0.065 0.537 0.052]\n [0.06 0.549 0.049]\n [0.059 0.555 0.046]\n [0.057 0.603 0.046]\n [0.068 0.549 0.053]]\nlayer 2 Q/K/both energy [[0.028 0.092 0.022]\n [0.008 0.044 0.007]\n [0.02 0.177 0.016]\n [0.038 0.092 0.031]\n [0.03 0.092 0.025]\n [0.031 0.088 0.025]]\nlayer 3 Q/K/both energy [[0.001 0.013 0.001]\n [0.001 0.002 0. ]\n [0.012 0.069 0.008]\n [0.016 0.065 0.011]\n [0.005 0.044 0.003]\n [0.003 0.008 0. ]]\nBroad-output subspace dimension 4\nlayer 1 Q/K/both energy [[0.099 0.65 0.08 ]\n [0.092 0.607 0.075]\n [0.086 0.642 0.076]\n [0.09 0.644 0.076]\n [0.091 0.701 0.078]\n [0.142 0.636 0.117]]\nlayer 2 Q/K/both energy [[0.037 0.101 0.026]\n [0.015 0.055 0.009]\n [0.042 0.2 0.031]\n [0.048 0.1 0.036]\n [0.039 0.102 0.029]\n [0.04 0.097 0.028]]\nlayer 3 Q/K/both energy [[0.005 0.017 0.001]\n [0.006 0.007 0.001]\n [0.02 0.076 0.009]\n [0.021 0.071 0.012]\n [0.012 0.048 0.004]\n [0.016 0.014 0.001]]\nBroad-output subspace dimension 8\nlayer 1 Q/K/both energy [[0.214 0.658 0.176]\n [0.216 0.614 0.183]\n [0.235 0.649 0.211]\n [0.229 0.653 0.199]\n [0.244 0.711 0.215]\n [0.27 0.644 0.228]]\nlayer 2 Q/K/both energy [[0.052 0.111 0.031]\n [0.029 0.067 0.01 ]\n [0.069 0.215 0.044]\n [0.064 0.11 0.042]\n [0.057 0.113 0.037]\n [0.057 0.107 0.035]]\nlayer 3 Q/K/both energy [[0.015 0.027 0.002]\n [0.019 0.021 0.002]\n [0.04 0.091 0.014]\n [0.039 0.082 0.015]\n [0.026 0.062 0.006]\n [0.048 0.031 0.004]]\nBroad-output subspace dimension 16\nlayer 1 Q/K/both energy [[0.258 0.672 0.204]\n [0.263 0.626 0.209]\n [0.278 0.662 0.241]\n [0.28 0.666 0.23 ]\n [0.303 0.726 0.259]\n [0.324 0.666 0.262]]\nlayer 2 Q/K/both energy [[0.081 0.137 0.041]\n [0.056 0.089 0.014]\n [0.098 0.244 0.058]\n [0.095 0.133 0.051]\n [0.087 0.134 0.047]\n [0.088 0.132 0.043]]\nlayer 3 Q/K/both energy [[0.037 0.051 0.004]\n [0.047 0.052 0.007]\n [0.067 0.111 0.019]\n [0.065 0.104 0.021]\n [0.054 0.083 0.01 ]\n [0.081 0.061 0.008]]\n\nKey-bias vector cosines within layers\n0 [[ 1. 0.259 0.055 0.18 -0.119 -0.169]\n [ 0.259 1. -0.169 0.089 -0.058 -0.425]\n [ 0.055 -0.169 1. 0.451 -0.225 0.019]\n [ 0.18 0.089 0.451 1. -0.446 -0.3 ]\n [-0.119 -0.058 -0.225 -0.446 1. 0.374]\n [-0.169 -0.425 0.019 -0.3 0.374 1. ]]\n1 [[1. 0.967 0.961 0.961 0.945 0.959]\n [0.967 1. 0.976 0.963 0.967 0.962]\n [0.961 0.976 1. 0.969 0.971 0.966]\n [0.961 0.963 0.969 1. 0.964 0.967]\n [0.945 0.967 0.971 0.964 1. 0.972]\n [0.959 0.962 0.966 0.967 0.972 1. ]]\n2 [[1. 0.839 0.875 0.909 0.926 0.907]\n [0.839 1. 0.858 0.845 0.849 0.832]\n [0.875 0.858 1. 0.863 0.88 0.851]\n [0.909 0.845 0.863 1. 0.925 0.914]\n [0.926 0.849 0.88 0.925 1. 0.904]\n [0.907 0.832 0.851 0.914 0.904 1. ]]\n3 [[ 1. 0.08 0.403 0.336 0.361 0.192]\n [ 0.08 1. -0.049 -0.173 -0.013 -0.028]\n [ 0.403 -0.049 1. 0.7 0.649 0.201]\n [ 0.336 -0.173 0.7 1. 0.628 0.258]\n [ 0.361 -0.013 0.649 0.628 1. 0.228]\n [ 0.192 -0.028 0.201 0.258 0.228 1. ]]\n\nL0H2 nonzero effective QK eigenvalues, sorted by phase (positive imaginary part)\nmagnitude, phase [[0.5123 0.0533]\n [0.7691 0.5616]\n [0.0324 0.655 ]\n [1.0295 0.9098]\n [0.6117 0.9859]\n [0.9325 1.2064]\n [0.1356 1.4186]\n [0.9555 1.8152]\n [0.0438 2.2779]\n [0.3409 2.3246]\n [0.3192 2.7337]\n [0.846 2.7354]]\n\nL0H2 effective V singular values [0.4305 0.4155 0.3844 0.3695 0.3615 0.3482 0.3276 0.3153 0.3091 0.3019 0.3002 0.2964 0.2916 0.2741 0.2653 0.2643 0.2597 0.2535 0.2471 0.2447 0.235 0.2317 0.23 0.2235 0.2159\n 0.2131 0.2056 0.1984 0.1913 0.1877 0.1848 0.181 0.1727 0.1611 0.1575 0.1521 0.1505 0.1366 0.1268 0.123 0.1143 0.1138 0.1063 0.1011 0.0968 0.0929 0.0898 0.0875 0.0852 0.0845\n 0.0816 0.0744 0.0733 0.0701 0.0672 0.0649 0.0628 0.0596 0.0582 0.0561 0.0545 0.0502 0.0469 0.0451]\nV-input subspace dimension 16\nlayer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.014 -0.007 0.496]\n [-0.11 0.257 0.474]\n [ 0.73 0.145 0.661]\n [ 0.589 0.225 0.624]\n [ 0.708 0.067 0.658]\n [-0.055 -0. 0.486]]\nlayer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.435 -0.039 0.403]\n [-0.443 0.248 0.411]\n [-0.015 0.053 0.497]\n [ 0.394 -0.01 0.585]\n [-0.855 0.031 0.319]\n [-0.087 0.053 0.482]]\nlayer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.269 0.032 0.445]\n [ 2.745 0.775 0.976]\n [ 0.39 0.054 0.578]\n [ 0.096 -0.064 0.521]\n [-0.372 0.205 0.426]\n [ 0.279 0.107 0.558]]\nV-input subspace dimension 24\nlayer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.266 0.022 0.435]\n [-0.025 0.224 0.495]\n [ 0.636 0.078 0.629]\n [ 0.674 0.128 0.629]\n [ 0.688 0.089 0.637]\n [ 0.201 0.022 0.545]]\nlayer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.451 -0.014 0.417]\n [-0.5 0.118 0.412]\n [ 0.121 0.031 0.522]\n [ 0.266 -0.009 0.548]\n [-0.578 -0.019 0.395]\n [-0.019 0.013 0.497]]\nlayer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.128 0.014 0.477]\n [ 2.907 0.649 0.95 ]\n [ 0.263 0.069 0.545]\n [-0.016 -0.083 0.497]\n [-0.296 0.166 0.45 ]\n [ 0.278 0.061 0.55 ]]\nV-input subspace dimension 28\nlayer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.308 0.048 0.431]\n [-0.022 0.209 0.496]\n [ 0.576 0.066 0.612]\n [ 0.711 0.118 0.633]\n [ 0.719 0.065 0.638]\n [ 0.189 0.013 0.54 ]]\nlayer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.296 -0.013 0.447]\n [-0.419 0.11 0.431]\n [ 0.111 0.042 0.519]\n [ 0.207 -0.02 0.536]\n [-0.515 -0.006 0.414]\n [-0.018 0.009 0.497]]\nlayer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.159 -0.003 0.473]\n [ 2.954 0.629 0.931]\n [ 0.238 0.012 0.539]\n [ 0.012 -0.08 0.502]\n [-0.328 0.124 0.448]\n [ 0.185 0.054 0.532]]\nV-input subspace dimension 40\nlayer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.263 0.057 0.444]\n [-0.009 0.185 0.498]\n [ 0.575 0.061 0.605]\n [ 0.582 0.089 0.605]\n [ 0.6 0.042 0.611]\n [ 0.18 0.022 0.537]]\nlayer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.233 -0.009 0.462]\n [-0.306 0.063 0.455]\n [ 0.07 0.031 0.51 ]\n [ 0.18 -0.008 0.527]\n [-0.434 -0.006 0.436]\n [ 0.024 0.006 0.504]]\nlayer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.127 0.008 0.482]\n [ 2.748 0.536 0.879]\n [ 0.182 0.001 0.526]\n [ 0.083 -0.044 0.512]\n [-0.302 0.074 0.457]\n [ 0.147 0.042 0.522]]\n\nMLP matrix-product spectra (not applying ReLU)\n0 norm 182.075 trace 108.759 topSV [125.589 39.838 32.492 30.802 28.038 27.085 26.097 25.622 23.929 23.842 23.274 22.856 22.364 22.005 21.639 20.807 20.641 20.022 19.713 18.934 18.543 17.739\n 17.313 17.036 16.473 15.955 15.529 15.188 14.793 14.06 13.656 13.021 12.343 12.15 11.156]\n1 norm 173.737 trace -379.418 topSV [76.145 32.918 27.909 26.868 25.244 24.417 24.076 23.193 23.072 22.428 22.228 22.016 21.36 21.149 21.061 20.413 20.317 19.89 19.459 19.177 18.917 18.702 18.598 18.15 18.072\n 17.965 17.612 17.442 17.182 16.847 16.777 16.276 16.127 16.003 15.684]\n2 norm 207.809 trace -889.497 topSV [67.851 29.898 27.249 26.671 26.128 25.792 25.323 25.096 24.492 24.232 23.887 23.663 22.747 22.503 22.125 21.628 21.282 21.039 20.607 20.483 20.344 20.152 20.062 19.816 19.573\n 19.249 18.907 18.719 18.626 18.472 18.392 18.304 17.976 17.857 17.803]\n3 norm 232.49 trace -926.657 topSV [133.224 55.432 48.819 45.113 38.683 38.229 37.439 36.181 35.707 35.113 33.956 33.208 31.855 30.777 29.986 29.814 29.384 28.138 26.731 25.424 25.262 24.435\n 24.105 23.314 22.456 21.569 20.308 19.311 18.704 18.259 17.235 16.782 16.385 15.897 15.172]\n",
"seconds": 0.598,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "76d5de71a89e12331a5aa8d34db19f0a9b66cf0dadc08d97e24f54a8c62e9d44",
"weights_sha256": "130135d2b27829df2734d78fa3af27b00b1cc1bf34c41b01a09b350e6c09a7de"
}L0 OV spectra 0 [8.0675 3.3266 2.0965 0.6725 0.5605 0.3138 0.2573 0.1923 0.1454 0.1409 0.1155 0.1128 0.1006 0.0849 0.0811 0.0724 0.0712 0.0656 0.0612 0.0571 0.0528 0.0489 0.047 0.0456 0.043 0.0395 0.0389 0.0361 0.0352 0.0335 0.0316 0.0304] 1 [5.6578 3.8262 1.0639 0.5023 0.4283 0.2699 0.2168 0.1662 0.1512 0.112 0.0944 0.0923 0.0829 0.0769 0.0712 0.0676 0.0621 0.0566 0.0552 0.0525 0.0499 0.0447 0.0435 0.0419 0.0384 0.0362 0.0357 0.0323 0.0321 0.031 0.0302 0.0285] 2 [0.7872 0.718 0.682 0.6712 0.6362 0.6009 0.5687 0.5579 0.55 0.5221 0.5148 0.4858 0.4799 0.4694 0.4386 0.4206 0.4004 0.3928 0.3466 0.3042 0.2771 0.2754 0.2554 0.2285 0.2179 0.2048 0.1813 0.1699 0.1626 0.1591 0.1313 0.1181] 3 [1.0317 0.964 0.6167 0.5045 0.4475 0.2267 0.2014 0.1834 0.1719 0.1475 0.1364 0.117 0.1057 0.093 0.0874 0.0833 0.0765 0.065 0.0612 0.0591 0.0545 0.0495 0.0465 0.0458 0.0394 0.0382 0.0359 0.0322 0.0315 0.0298 0.0284 0.0259] 4 [3.3943 1.6416 1.0818 0.6072 0.3784 0.3273 0.2313 0.2045 0.1705 0.1412 0.1263 0.1156 0.1029 0.1003 0.0812 0.0771 0.0741 0.0648 0.0593 0.0572 0.0538 0.0494 0.0464 0.0439 0.0401 0.0383 0.0367 0.0346 0.0334 0.0312 0.0306 0.0295] 5 [8.3281 3.6039 1.6786 0.7676 0.5104 0.3903 0.3443 0.3045 0.2473 0.1927 0.1667 0.1265 0.1142 0.1088 0.0982 0.0936 0.0851 0.0714 0.0675 0.0611 0.06 0.0574 0.0562 0.0517 0.0493 0.0461 0.0436 0.0403 0.0381 0.0364 0.0343 0.032 ] L0 OV pairwise cosines [[1. 0.885 0.008 0.164 0.903 0.724] [0.885 1. 0.006 0.195 0.889 0.802] [0.008 0.006 1. 0.012 0.006 0.017] [0.164 0.195 0.012 1. 0.195 0.156] [0.903 0.889 0.006 0.195 1. 0.728] [0.724 0.802 0.017 0.156 0.728 1. ]] Joint broad-head output spectrum [11.9885 8.4303 3.3621 1.3104 0.9989 0.8608 0.73 0.6052 0.566 0.54 0.4823 0.4356 0.3927 0.3667 0.3463 0.3207 0.3058 0.2952 0.262 0.246 0.2384 0.215 0.2062 0.1875 0.1871 0.1741 0.171 0.1613 0.1555 0.153 0.1481 0.1405 0.1331 0.1328 0.125 ] Broad-output subspace dimension 2 layer 1 Q/K/both energy [[0.061 0.575 0.047] [0.065 0.537 0.052] [0.06 0.549 0.049] [0.059 0.555 0.046] [0.057 0.603 0.046] [0.068 0.549 0.053]] layer 2 Q/K/both energy [[0.028 0.092 0.022] [0.008 0.044 0.007] [0.02 0.177 0.016] [0.038 0.092 0.031] [0.03 0.092 0.025] [0.031 0.088 0.025]] layer 3 Q/K/both energy [[0.001 0.013 0.001] [0.001 0.002 0. ] [0.012 0.069 0.008] [0.016 0.065 0.011] [0.005 0.044 0.003] [0.003 0.008 0. ]] Broad-output subspace dimension 4 layer 1 Q/K/both energy [[0.099 0.65 0.08 ] [0.092 0.607 0.075] [0.086 0.642 0.076] [0.09 0.644 0.076] [0.091 0.701 0.078] [0.142 0.636 0.117]] layer 2 Q/K/both energy [[0.037 0.101 0.026] [0.015 0.055 0.009] [0.042 0.2 0.031] [0.048 0.1 0.036] [0.039 0.102 0.029] [0.04 0.097 0.028]] layer 3 Q/K/both energy [[0.005 0.017 0.001] [0.006 0.007 0.001] [0.02 0.076 0.009] [0.021 0.071 0.012] [0.012 0.048 0.004] [0.016 0.014 0.001]] Broad-output subspace dimension 8 layer 1 Q/K/both energy [[0.214 0.658 0.176] [0.216 0.614 0.183] [0.235 0.649 0.211] [0.229 0.653 0.199] [0.244 0.711 0.215] [0.27 0.644 0.228]] layer 2 Q/K/both energy [[0.052 0.111 0.031] [0.029 0.067 0.01 ] [0.069 0.215 0.044] [0.064 0.11 0.042] [0.057 0.113 0.037] [0.057 0.107 0.035]] layer 3 Q/K/both energy [[0.015 0.027 0.002] [0.019 0.021 0.002] [0.04 0.091 0.014] [0.039 0.082 0.015] [0.026 0.062 0.006] [0.048 0.031 0.004]] Broad-output subspace dimension 16 layer 1 Q/K/both energy [[0.258 0.672 0.204] [0.263 0.626 0.209] [0.278 0.662 0.241] [0.28 0.666 0.23 ] [0.303 0.726 0.259] [0.324 0.666 0.262]] layer 2 Q/K/both energy [[0.081 0.137 0.041] [0.056 0.089 0.014] [0.098 0.244 0.058] [0.095 0.133 0.051] [0.087 0.134 0.047] [0.088 0.132 0.043]] layer 3 Q/K/both energy [[0.037 0.051 0.004] [0.047 0.052 0.007] [0.067 0.111 0.019] [0.065 0.104 0.021] [0.054 0.083 0.01 ] [0.081 0.061 0.008]] Key-bias vector cosines within layers 0 [[ 1. 0.259 0.055 0.18 -0.119 -0.169] [ 0.259 1. -0.169 0.089 -0.058 -0.425] [ 0.055 -0.169 1. 0.451 -0.225 0.019] [ 0.18 0.089 0.451 1. -0.446 -0.3 ] [-0.119 -0.058 -0.225 -0.446 1. 0.374] [-0.169 -0.425 0.019 -0.3 0.374 1. ]] 1 [[1. 0.967 0.961 0.961 0.945 0.959] [0.967 1. 0.976 0.963 0.967 0.962] [0.961 0.976 1. 0.969 0.971 0.966] [0.961 0.963 0.969 1. 0.964 0.967] [0.945 0.967 0.971 0.964 1. 0.972] [0.959 0.962 0.966 0.967 0.972 1. ]] 2 [[1. 0.839 0.875 0.909 0.926 0.907] [0.839 1. 0.858 0.845 0.849 0.832] [0.875 0.858 1. 0.863 0.88 0.851] [0.909 0.845 0.863 1. 0.925 0.914] [0.926 0.849 0.88 0.925 1. 0.904] [0.907 0.832 0.851 0.914 0.904 1. ]] 3 [[ 1. 0.08 0.403 0.336 0.361 0.192] [ 0.08 1. -0.049 -0.173 -0.013 -0.028] [ 0.403 -0.049 1. 0.7 0.649 0.201] [ 0.336 -0.173 0.7 1. 0.628 0.258] [ 0.361 -0.013 0.649 0.628 1. 0.228] [ 0.192 -0.028 0.201 0.258 0.228 1. ]] L0H2 nonzero effective QK eigenvalues, sorted by phase (positive imaginary part) magnitude, phase [[0.5123 0.0533] [0.7691 0.5616] [0.0324 0.655 ] [1.0295 0.9098] [0.6117 0.9859] [0.9325 1.2064] [0.1356 1.4186] [0.9555 1.8152] [0.0438 2.2779] [0.3409 2.3246] [0.3192 2.7337] [0.846 2.7354]] L0H2 effective V singular values [0.4305 0.4155 0.3844 0.3695 0.3615 0.3482 0.3276 0.3153 0.3091 0.3019 0.3002 0.2964 0.2916 0.2741 0.2653 0.2643 0.2597 0.2535 0.2471 0.2447 0.235 0.2317 0.23 0.2235 0.2159 0.2131 0.2056 0.1984 0.1913 0.1877 0.1848 0.181 0.1727 0.1611 0.1575 0.1521 0.1505 0.1366 0.1268 0.123 0.1143 0.1138 0.1063 0.1011 0.0968 0.0929 0.0898 0.0875 0.0852 0.0845 0.0816 0.0744 0.0733 0.0701 0.0672 0.0649 0.0628 0.0596 0.0582 0.0561 0.0545 0.0502 0.0469 0.0451] V-input subspace dimension 16 layer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.014 -0.007 0.496] [-0.11 0.257 0.474] [ 0.73 0.145 0.661] [ 0.589 0.225 0.624] [ 0.708 0.067 0.658] [-0.055 -0. 0.486]] layer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.435 -0.039 0.403] [-0.443 0.248 0.411] [-0.015 0.053 0.497] [ 0.394 -0.01 0.585] [-0.855 0.031 0.319] [-0.087 0.053 0.482]] layer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.269 0.032 0.445] [ 2.745 0.775 0.976] [ 0.39 0.054 0.578] [ 0.096 -0.064 0.521] [-0.372 0.205 0.426] [ 0.279 0.107 0.558]] V-input subspace dimension 24 layer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.266 0.022 0.435] [-0.025 0.224 0.495] [ 0.636 0.078 0.629] [ 0.674 0.128 0.629] [ 0.688 0.089 0.637] [ 0.201 0.022 0.545]] layer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.451 -0.014 0.417] [-0.5 0.118 0.412] [ 0.121 0.031 0.522] [ 0.266 -0.009 0.548] [-0.578 -0.019 0.395] [-0.019 0.013 0.497]] layer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.128 0.014 0.477] [ 2.907 0.649 0.95 ] [ 0.263 0.069 0.545] [-0.016 -0.083 0.497] [-0.296 0.166 0.45 ] [ 0.278 0.061 0.55 ]] V-input subspace dimension 28 layer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.308 0.048 0.431] [-0.022 0.209 0.496] [ 0.576 0.066 0.612] [ 0.711 0.118 0.633] [ 0.719 0.065 0.638] [ 0.189 0.013 0.54 ]] layer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.296 -0.013 0.447] [-0.419 0.11 0.431] [ 0.111 0.042 0.519] [ 0.207 -0.02 0.536] [-0.515 -0.006 0.414] [-0.018 0.009 0.497]] layer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.159 -0.003 0.473] [ 2.954 0.629 0.931] [ 0.238 0.012 0.539] [ 0.012 -0.08 0.502] [-0.328 0.124 0.448] [ 0.185 0.054 0.532]] V-input subspace dimension 40 layer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.263 0.057 0.444] [-0.009 0.185 0.498] [ 0.575 0.061 0.605] [ 0.582 0.089 0.605] [ 0.6 0.042 0.611] [ 0.18 0.022 0.537]] layer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.233 -0.009 0.462] [-0.306 0.063 0.455] [ 0.07 0.031 0.51 ] [ 0.18 -0.008 0.527] [-0.434 -0.006 0.436] [ 0.024 0.006 0.504]] layer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.127 0.008 0.482] [ 2.748 0.536 0.879] [ 0.182 0.001 0.526] [ 0.083 -0.044 0.512] [-0.302 0.074 0.457] [ 0.147 0.042 0.522]] MLP matrix-product spectra (not applying ReLU) 0 norm 182.075 trace 108.759 topSV [125.589 39.838 32.492 30.802 28.038 27.085 26.097 25.622 23.929 23.842 23.274 22.856 22.364 22.005 21.639 20.807 20.641 20.022 19.713 18.934 18.543 17.739 17.313 17.036 16.473 15.955 15.529 15.188 14.793 14.06 13.656 13.021 12.343 12.15 11.156] 1 norm 173.737 trace -379.418 topSV [76.145 32.918 27.909 26.868 25.244 24.417 24.076 23.193 23.072 22.428 22.228 22.016 21.36 21.149 21.061 20.413 20.317 19.89 19.459 19.177 18.917 18.702 18.598 18.15 18.072 17.965 17.612 17.442 17.182 16.847 16.777 16.276 16.127 16.003 15.684] 2 norm 207.809 trace -889.497 topSV [67.851 29.898 27.249 26.671 26.128 25.792 25.323 25.096 24.492 24.232 23.887 23.663 22.747 22.503 22.125 21.628 21.282 21.039 20.607 20.483 20.344 20.152 20.062 19.816 19.573 19.249 18.907 18.719 18.626 18.472 18.392 18.304 17.976 17.857 17.803] 3 norm 232.49 trace -926.657 topSV [133.224 55.432 48.819 45.113 38.683 38.229 37.439 36.181 35.707 35.113 33.956 33.208 31.855 30.777 29.986 29.814 29.384 28.138 26.731 25.424 25.262 24.435 24.105 23.314 22.456 21.569 20.308 19.311 18.704 18.259 17.235 16.782 16.385 15.897 15.172]
Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 3 of at most 100: ``` L0 OV spectra 0 [8.0675 3.3266 2.0965 0.6725 0.5605 0.3138 0.2573 0.1923 0.1454 0.1409 0.1155 0.1128 0.1006 0.0849 0.0811 0.0724 0.0712 0.0656 0.0612 0.0571 0.0528 0.0489 0.047 0.0456 0.043 0.0395 0.0389 0.0361 0.0352 0.0335 0.0316 0.0304] 1 [5.6578 3.8262 1.0639 0.5023 0.4283 0.2699 0.2168 0.1662 0.1512 0.112 0.0944 0.0923 0.0829 0.0769 0.0712 0.0676 0.0621 0.0566 0.0552 0.0525 0.0499 0.0447 0.0435 0.0419 0.0384 0.0362 0.0357 0.0323 0.0321 0.031 0.0302 0.0285] 2 [0.7872 0.718 0.682 0.6712 0.6362 0.6009 0.5687 0.5579 0.55 0.5221 0.5148 0.4858 0.4799 0.4694 0.4386 0.4206 0.4004 0.3928 0.3466 0.3042 0.2771 0.2754 0.2554 0.2285 0.2179 0.2048 0.1813 0.1699 0.1626 0.1591 0.1313 0.1181] 3 [1.0317 0.964 0.6167 0.5045 0.4475 0.2267 0.2014 0.1834 0.1719 0.1475 0.1364 0.117 0.1057 0.093 0.0874 0.0833 0.0765 0.065 0.0612 0.0591 0.0545 0.0495 0.0465 0.0458 0.0394 0.0382 0.0359 0.0322 0.0315 0.0298 0.0284 0.0259] 4 [3.3943 1.6416 1.0818 0.6072 0.3784 0.3273 0.2313 0.2045 0.1705 0.1412 0.1263 0.1156 0.1029 0.1003 0.0812 0.0771 0.0741 0.0648 0.0593 0.0572 0.0538 0.0494 0.0464 0.0439 0.0401 0.0383 0.0367 0.0346 0.0334 0.0312 0.0306 0.0295] 5 [8.3281 3.6039 1.6786 0.7676 0.5104 0.3903 0.3443 0.3045 0.2473 0.1927 0.1667 0.1265 0.1142 0.1088 0.0982 0.0936 0.0851 0.0714 0.0675 0.0611 0.06 0.0574 0.0562 0.0517 0.0493 0.0461 0.0436 0.0403 0.0381 0.0364 0.0343 0.032 ] L0 OV pairwise cosines [[1. 0.885 0.008 0.164 0.903 0.724] [0.885 1. 0.006 0.195 0.889 0.802] [0.008 0.006 1. 0.012 0.006 0.017] [0.164 0.195 0.012 1. 0.195 0.156] [0.903 0.889 0.006 0.195 1. 0.728] [0.724 0.802 0.017 0.156 0.728 1. ]] Joint broad-head output spectrum [11.9885 8.4303 3.3621 1.3104 0.9989 0.8608 0.73 0.6052 0.566 0.54 0.4823 0.4356 0.3927 0.3667 0.3463 0.3207 0.3058 0.2952 0.262 0.246 0.2384 0.215 0.2062 0.1875 0.1871 0.1741 0.171 0.1613 0.1555 0.153 0.1481 0.1405 0.1331 0.1328 0.125 ] Broad-output subspace dimension 2 layer 1 Q/K/both energy [[0.061 0.575 0.047] [0.065 0.537 0.052] [0.06 0.549 0.049] [0.059 0.555 0.046] [0.057 0.603 0.046] [0.068 0.549 0.053]] layer 2 Q/K/both energy [[0.028 0.092 0.022] [0.008 0.044 0.007] [0.02 0.177 0.016] [0.038 0.092 0.031] [0.03 0.092 0.025] [0.031 0.088 0.025]] layer 3 Q/K/both energy [[0.001 0.013 0.001] [0.001 0.002 0. ] [0.012 0.069 0.008] [0.016 0.065 0.011] [0.005 0.044 0.003] [0.003 0.008 0. ]] Broad-output subspace dimension 4 layer 1 Q/K/both energy [[0.099 0.65 0.08 ] [0.092 0.607 0.075] [0.086 0.642 0.076] [0.09 0.644 0.076] [0.091 0.701 0.078] [0.142 0.636 0.117]] layer 2 Q/K/both energy [[0.037 0.101 0.026] [0.015 0.055 0.009] [0.042 0.2 0.031] [0.048 0.1 0.036] [0.039 0.102 0.029] [0.04 0.097 0.028]] layer 3 Q/K/both energy [[0.005 0.017 0.001] [0.006 0.007 0.001] [0.02 0.076 0.009] [0.021 0.071 0.012] [0.012 0.048 0.004] [0.016 0.014 0.001]] Broad-output subspace dimension 8 layer 1 Q/K/both energy [[0.214 0.658 0.176] [0.216 0.614 0.183] [0.235 0.649 0.211] [0.229 0.653 0.199] [0.244 0.711 0.215] [0.27 0.644 0.228]] layer 2 Q/K/both energy [[0.052 0.111 0.031] [0.029 0.067 0.01 ] [0.069 0.215 0.044] [0.064 0.11 0.042] [0.057 0.113 0.037] [0.057 0.107 0.035]] layer 3 Q/K/both energy [[0.015 0.027 0.002] [0.019 0.021 0.002] [0.04 0.091 0.014] [0.039 0.082 0.015] [0.026 0.062 0.006] [0.048 0.031 0.004]] Broad-output subspace dimension 16 layer 1 Q/K/both energy [[0.258 0.672 0.204] [0.263 0.626 0.209] [0.278 0.662 0.241] [0.28 0.666 0.23 ] [0.303 0.726 0.259] [0.324 0.666 0.262]] layer 2 Q/K/both energy [[0.081 0.137 0.041] [0.056 0.089 0.014] [0.098 0.244 0.058] [0.095 0.133 0.051] [0.087 0.134 0.047] [0.088 0.132 0.043]] layer 3 Q/K/both energy [[0.037 0.051 0.004] [0.047 0.052 0.007] [0.067 0.111 0.019] [0.065 0.104 0.021] [0.054 0.083 0.01 ] [0.081 0.061 0.008]] Key-bias vector cosines within layers 0 [[ 1. 0.259 0.055 0.18 -0.119 -0.169] [ 0.259 1. -0.169 0.089 -0.058 -0.425] [ 0.055 -0.169 1. 0.451 -0.225 0.019] [ 0.18 0.089 0.451 1. -0.446 -0.3 ] [-0.119 -0.058 -0.225 -0.446 1. 0.374] [-0.169 -0.425 0.019 -0.3 0.374 1. ]] 1 [[1. 0.967 0.961 0.961 0.945 0.959] [0.967 1. 0.976 0.963 0.967 0.962] [0.961 0.976 1. 0.969 0.971 0.966] [0.961 0.963 0.969 1. 0.964 0.967] [0.945 0.967 0.971 0.964 1. 0.972] [0.959 0.962 0.966 0.967 0.972 1. ]] 2 [[1. 0.839 0.875 0.909 0.926 0.907] [0.839 1. 0.858 0.845 0.849 0.832] [0.875 0.858 1. 0.863 0.88 0.851] [0.909 0.845 0.863 1. 0.925 0.914] [0.926 0.849 0.88 0.925 1. 0.904] [0.907 0.832 0.851 0.914 0.904 1. ]] 3 [[ 1. 0.08 0.403 0.336 0.361 0.192] [ 0.08 1. -0.049 -0.173 -0.013 -0.028] [ 0.403 -0.049 1. 0.7 0.649 0.201] [ 0.336 -0.173 0.7 1. 0.628 0.258] [ 0.361 -0.013 0.649 0.628 1. 0.228] [ 0.192 -0.028 0.201 0.258 0.228 1. ]] L0H2 nonzero effective QK eigenvalues, sorted by phase (positive imaginary part) magnitude, phase [[0.5123 0.0533] [0.7691 0.5616] [0.0324 0.655 ] [1.0295 0.9098] [0.6117 0.9859] [0.9325 1.2064] [0.1356 1.4186] [0.9555 1.8152] [0.0438 2.2779] [0.3409 2.3246] [0.3192 2.7337] [0.846 2.7354]] L0H2 effective V singular values [0.4305 0.4155 0.3844 0.3695 0.3615 0.3482 0.3276 0.3153 0.3091 0.3019 0.3002 0.2964 0.2916 0.2741 0.2653 0.2643 0.2597 0.2535 0.2471 0.2447 0.235 0.2317 0.23 0.2235 0.2159 0.2131 0.2056 0.1984 0.1913 0.1877 0.1848 0.181 0.1727 0.1611 0.1575 0.1521 0.1505 0.1366 0.1268 0.123 0.1143 0.1138 0.1063 0.1011 0.0968 0.0929 0.0898 0.0875 0.0852 0.0845 0.0816 0.0744 0.0733 0.0701 0.0672 0.0649 0.0628 0.0596 0.0582 0.0561 0.0545 0.0502 0.0469 0.0451] V-input subspace dimension 16 layer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.014 -0.007 0.496] [-0.11 0.257 0.474] [ 0.73 0.145 0.661] [ 0.589 0.225 0.624] [ 0.708 0.067 0.658] [-0.055 -0. 0.486]] layer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.435 -0.039 0.403] [-0.443 0.248 0.411] [-0.015 0.053 0.497] [ 0.394 -0.01 0.585] [-0.855 0.031 0.319] [-0.087 0.053 0.482]] layer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.269 0.032 0.445] [ 2.745 0.775 0.976] [ 0.39 0.054 0.578] [ 0.096 -0.064 0.521] [-0.372 0.205 0.426] [ 0.279 0.107 0.558]] V-input subspace dimension 24 layer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.266 0.022 0.435] [-0.025 0.224 0.495] [ 0.636 0.078 0.629] [ 0.674 0.128 0.629] [ 0.688 0.089 0.637] [ 0.201 0.022 0.545]] layer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.451 -0.014 0.417] [-0.5 0.118 0.412] [ 0.121 0.031 0.522] [ 0.266 -0.009 0.548] [-0.578 -0.019 0.395] [-0.019 0.013 0.497]] layer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.128 0.014 0.477] [ 2.907 0.649 0.95 ] [ 0.263 0.069 0.545] [-0.016 -0.083 0.497] [-0.296 0.166 0.45 ] [ 0.278 0.061 0.55 ]] V-input subspace dimension 28 layer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.308 0.048 0.431] [-0.022 0.209 0.496] [ 0.576 0.066 0.612] [ 0.711 0.118 0.633] [ 0.719 0.065 0.638] [ 0.189 0.013 0.54 ]] layer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.296 -0.013 0.447] [-0.419 0.11 0.431] [ 0.111 0.042 0.519] [ 0.207 -0.02 0.536] [-0.515 -0.006 0.414] [-0.018 0.009 0.497]] layer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.159 -0.003 0.473] [ 2.954 0.629 0.931] [ 0.238 0.012 0.539] [ 0.012 -0.08 0.502] [-0.328 0.124 0.448] [ 0.185 0.054 0.532]] V-input subspace dimension 40 layer 1 composition trace/F, symmetry, positive-symmetric-mass [[-0.263 0.057 0.444] [-0.009 0.185 0.498] [ 0.575 0.061 0.605] [ 0.582 0.089 0.605] [ 0.6 0.042 0.611] [ 0.18 0.022 0.537]] layer 2 composition trace/F, symmetry, positive-symmetric-mass [[-0.233 -0.009 0.462] [-0.306 0.063 0.455] [ 0.07 0.031 0.51 ] [ 0.18 -0.008 0.527] [-0.434 -0.006 0.436] [ 0.024 0.006 0.504]] layer 3 composition trace/F, symmetry, positive-symmetric-mass [[-0.127 0.008 0.482] [ 2.748 0.536 0.879] [ 0.182 0.001 0.526] [ 0.083 -0.044 0.512] [-0.302 0.074 0.457] [ 0.147 0.042 0.522]] MLP matrix-product spectra (not applying ReLU) 0 norm 182.075 trace 108.759 topSV [125.589 39.838 32.492 30.802 28.038 27.085 26.097 25.622 23.929 23.842 23.274 22.856 22.364 22.005 21.639 20.807 20.641 20.022 19.713 18.934 18.543 17.739 17.313 17.036 16.473 15.955 15.529 15.188 14.793 14.06 13.656 13.021 12.343 12.15 11.156] 1 norm 173.737 trace -379.418 topSV [76.145 32.918 27.909 26.868 25.244 24.417 24.076 23.193 23.072 22.428 22.228 22.016 21.36 21.149 21.061 20.413 20.317 19.89 19.459 19.177 18.917 18.702 18.598 18.15 18.072 17.965 17.612 17.442 17.182 16.847 16.777 16.276 16.127 16.003 15.684] 2 norm 207.809 trace -889.497 topSV [67.851 29.898 27.249 26.671 26.128 25.792 25.323 25.096 24.492 24.232 23.887 23.663 22.747 22.503 22.125 21.628 21.282 21.039 20.607 20.483 20.344 20.152 20.062 19.816 19.573 19.249 18.907 18.719 18.626 18.472 18.392 18.304 17.976 17.857 17.803] 3 norm 232.49 trace -926.657 topSV [133.224 55.432 48.819 45.113 38.683 38.229 37.439 36.181 35.707 35.113 33.956 33.208 31.855 30.777 29.986 29.814 29.384 28.138 26.731 25.424 25.262 24.435 24.105 23.314 22.456 21.569 20.308 19.311 18.704 18.259 17.235 16.782 16.385 15.897 15.172] ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(2)
np.set_printoptions(precision=4,suppress=True,linewidth=180)
d=384
P=np.eye(d)-np.ones((d,d))/d
A={}; B={}
for l in range(4):
g=W[f"ln1_w.{l}"].astype(float)
for h in range(6):
sl=slice(64*h,64*(h+1))
q,k,v=[(W[f"W_{t}.{l}"][sl].astype(float)*g)@P for t in ["q","k","v"]]
A[l,h]=q.T@k/8
B[l,h]=P@W[f"W_o.{l}"][:,sl].astype(float)@v
u,s,vt=np.linalg.svd(A[0,2])
up,sp,_=np.linalg.svd(np.concatenate([u[:,:16],vt[:16].T],axis=1),full_matrices=False)
T=up[:,:16]
print("L0 OV input/output energy in candidate position space, dimension 16")
for h in range(6):
b=B[0,h]
uu,ss,vv=np.linalg.svd(b)
print(h,"total input/output",np.round([
np.linalg.norm(b@T)**2/np.linalg.norm(b)**2,
np.linalg.norm(T.T@b)**2/np.linalg.norm(b)**2],4),
"top 5 input-vector overlaps",np.round(np.sum((vv[:5]@T)**2,axis=1),4),
"top 5 output-vector overlaps",np.round(np.sum((uu[:,:5].T@T)**2,axis=1),4))
print("\nL0 head top QK and OV singular-direction pairwise alignments")
# Directions, not input examples.
Qdirs=[]; Kdirs=[]; Vdirs=[]
for h in range(6):
u,s,vt=np.linalg.svd(A[0,h])
Qdirs.append(u[:,0]); Kdirs.append(vt[0])
u,s,vt=np.linalg.svd(B[0,h])
Vdirs.append(vt[0])
for label,rows,cols in [("Q vs Q",Qdirs,Qdirs),("K vs K",Kdirs,Kdirs),
("V vs V",Vdirs,Vdirs),("K vs V",Kdirs,Vdirs)]:
print(label,"\n",np.round(np.array(rows)@np.array(cols).T,3))
print("\nInduction-candidate symmetric composition eigenvalues")
c=A[3,1]@B[0,2]
ev,u=np.linalg.eigh((c+c.T)/2)
idx=np.argsort(ev)[::-1]
print("Largest 45",np.round(ev[idx[:45]],5))
print("Smallest 15",np.round(ev[:15],5))
Tc=u[:,idx[:24]]
print("Positive composition-space overlap with position",np.round(np.sum((Tc.T@T)**2,axis=1),3))
# Structural analysis of first-MLP read directions.
f=(W["W_fc.0"].astype(float)*W["ln2_w.0"])@P
o=P@W["W_proj.0"].astype(float)
for label,m in [("current read",f@Tc),("previous-head read",f@B[0,2]@Tc)]:
sv=np.linalg.svd(m,compute_uv=False)
print(label,"SV",np.round(sv,3))
norms=np.linalg.norm(m,axis=1)
print(label,"row-norm quantiles",np.round(np.quantile(norms,[0,.1,.25,.5,.75,.9,.99,1]),3))
# The size of nonlinear-layer weight products through the putative
# previous-character channel, compared with the other five channels.
print("\nFirst-MLP read composition strengths")
for h in range(6):
m=f@B[0,h]
print(h,"F norm",round(np.linalg.norm(m),3),
"top SV",np.round(np.linalg.svd(m,compute_uv=False)[:12],3))
print("\nFinal-MLP projection spectrum tail and consecutive ratios")
sv=np.linalg.svd(W["W_proj.3"].astype(float),compute_uv=False)
for j in range(15,40):
print(j+1,round(sv[j],4),round(sv[j]/sv[j+1],4))
```{
"run": 4,
"turn": "t004",
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(2)\nnp.set_printoptions(precision=4,suppress=True,linewidth=180)\n\nd=384\nP=np.eye(d)-np.ones((d,d))/d\nA={}; B={}\nfor l in range(4):\n g=W[f\"ln1_w.{l}\"].astype(float)\n for h in range(6):\n sl=slice(64*h,64*(h+1))\n q,k,v=[(W[f\"W_{t}.{l}\"][sl].astype(float)*g)@P for t in [\"q\",\"k\",\"v\"]]\n A[l,h]=q.T@k/8\n B[l,h]=P@W[f\"W_o.{l}\"][:,sl].astype(float)@v\n\nu,s,vt=np.linalg.svd(A[0,2])\nup,sp,_=np.linalg.svd(np.concatenate([u[:,:16],vt[:16].T],axis=1),full_matrices=False)\nT=up[:,:16]\nprint(\"L0 OV input/output energy in candidate position space, dimension 16\")\nfor h in range(6):\n b=B[0,h]\n uu,ss,vv=np.linalg.svd(b)\n print(h,\"total input/output\",np.round([\n np.linalg.norm(b@T)**2/np.linalg.norm(b)**2,\n np.linalg.norm(T.T@b)**2/np.linalg.norm(b)**2],4),\n \"top 5 input-vector overlaps\",np.round(np.sum((vv[:5]@T)**2,axis=1),4),\n \"top 5 output-vector overlaps\",np.round(np.sum((uu[:,:5].T@T)**2,axis=1),4))\n\nprint(\"\\nL0 head top QK and OV singular-direction pairwise alignments\")\n# Directions, not input examples.\nQdirs=[]; Kdirs=[]; Vdirs=[]\nfor h in range(6):\n u,s,vt=np.linalg.svd(A[0,h])\n Qdirs.append(u[:,0]); Kdirs.append(vt[0])\n u,s,vt=np.linalg.svd(B[0,h])\n Vdirs.append(vt[0])\nfor label,rows,cols in [(\"Q vs Q\",Qdirs,Qdirs),(\"K vs K\",Kdirs,Kdirs),\n (\"V vs V\",Vdirs,Vdirs),(\"K vs V\",Kdirs,Vdirs)]:\n print(label,\"\\n\",np.round(np.array(rows)@np.array(cols).T,3))\n\nprint(\"\\nInduction-candidate symmetric composition eigenvalues\")\nc=A[3,1]@B[0,2]\nev,u=np.linalg.eigh((c+c.T)/2)\nidx=np.argsort(ev)[::-1]\nprint(\"Largest 45\",np.round(ev[idx[:45]],5))\nprint(\"Smallest 15\",np.round(ev[:15],5))\nTc=u[:,idx[:24]]\nprint(\"Positive composition-space overlap with position\",np.round(np.sum((Tc.T@T)**2,axis=1),3))\n\n# Structural analysis of first-MLP read directions.\nf=(W[\"W_fc.0\"].astype(float)*W[\"ln2_w.0\"])@P\no=P@W[\"W_proj.0\"].astype(float)\nfor label,m in [(\"current read\",f@Tc),(\"previous-head read\",f@B[0,2]@Tc)]:\n sv=np.linalg.svd(m,compute_uv=False)\n print(label,\"SV\",np.round(sv,3))\n norms=np.linalg.norm(m,axis=1)\n print(label,\"row-norm quantiles\",np.round(np.quantile(norms,[0,.1,.25,.5,.75,.9,.99,1]),3))\n\n# The size of nonlinear-layer weight products through the putative\n# previous-character channel, compared with the other five channels.\nprint(\"\\nFirst-MLP read composition strengths\")\nfor h in range(6):\n m=f@B[0,h]\n print(h,\"F norm\",round(np.linalg.norm(m),3),\n \"top SV\",np.round(np.linalg.svd(m,compute_uv=False)[:12],3))\n\nprint(\"\\nFinal-MLP projection spectrum tail and consecutive ratios\")\nsv=np.linalg.svd(W[\"W_proj.3\"].astype(float),compute_uv=False)\nfor j in range(15,40):\n print(j+1,round(sv[j],4),round(sv[j]/sv[j+1],4))",
"out": "L0 OV input/output energy in candidate position space, dimension 16\n0 total input/output [0.8579 0.216 ] top 5 input-vector overlaps [0.9539 0.5383 0.4343 0.0771 0.0913] top 5 output-vector overlaps [0.2457 0.0895 0.1317 0.0972 0.0261]\n1 total input/output [0.8805 0.1878] top 5 input-vector overlaps [0.925 0.878 0.1423 0.0393 0.0678] top 5 output-vector overlaps [0.2379 0.0876 0.1431 0.0817 0.042 ]\n2 total input/output [0.085 0.1071] top 5 input-vector overlaps [0.0367 0.0384 0.0507 0.0456 0.0534] top 5 output-vector overlaps [0.0753 0.0774 0.1159 0.0668 0.047 ]\n3 total input/output [0.1107 0.0716] top 5 input-vector overlaps [0.1557 0.0964 0.0366 0.094 0.0347] top 5 output-vector overlaps [0.1296 0.0318 0.0227 0.0476 0.066 ]\n4 total input/output [0.7985 0.1946] top 5 input-vector overlaps [0.9263 0.7577 0.1711 0.048 0.0472] top 5 output-vector overlaps [0.2369 0.0879 0.13 0.0363 0.059 ]\n5 total input/output [0.8505 0.1501] top 5 input-vector overlaps [0.8973 0.8437 0.1056 0.152 0.069 ] top 5 output-vector overlaps [0.145 0.1923 0.1161 0.1198 0.0688]\n\nL0 head top QK and OV singular-direction pairwise alignments\nQ vs Q \n [[ 1. -0.77 0.434 0.797 -0.129 -0.24 ]\n [-0.77 1. -0.528 -0.738 0.229 0.048]\n [ 0.434 -0.528 1. 0.492 -0.113 0.07 ]\n [ 0.797 -0.738 0.492 1. 0.064 0.033]\n [-0.129 0.229 -0.113 0.064 1. 0.326]\n [-0.24 0.048 0.07 0.033 0.326 1. ]]\nK vs K \n [[ 1. -0.264 -0.057 -0.003 -0.38 -0.098]\n [-0.264 1. 0.02 0.124 0.258 -0.143]\n [-0.057 0.02 1. 0.529 0.019 0.13 ]\n [-0.003 0.124 0.529 1. -0.101 0.012]\n [-0.38 0.258 0.019 -0.101 1. 0.643]\n [-0.098 -0.143 0.13 0.012 0.643 1. ]]\nV vs V \n [[ 1. 0.945 0.066 -0.044 0.962 -0.692]\n [ 0.945 1. 0.076 0.181 0.957 -0.639]\n [ 0.066 0.076 1. -0.005 0.037 -0.077]\n [-0.044 0.181 -0.005 1. 0.134 0.259]\n [ 0.962 0.957 0.037 0.134 1. -0.71 ]\n [-0.692 -0.639 -0.077 0.259 -0.71 1. ]]\nK vs V \n [[ 0.093 0.189 -0.417 0.262 0.156 0.075]\n [-0.011 -0.108 0.137 -0.175 -0.076 0.012]\n [-0.55 -0.535 -0.006 0.06 -0.531 0.509]\n [-0.81 -0.763 0.019 0.17 -0.826 0.926]\n [ 0.03 -0.198 -0.061 -0.471 -0.032 -0.166]\n [-0.211 -0.399 -0.244 -0.447 -0.243 -0.035]]\n\nInduction-candidate symmetric composition eigenvalues\nLargest 45 [0.1007 0.0826 0.0765 0.0738 0.0698 0.0647 0.0602 0.0561 0.0543 0.0525 0.0467 0.0417 0.039 0.035 0.0274 0.0252 0.0242 0.0186 0.0168 0.0127 0.0119 0.0111 0.0106 0.0089 0.0075\n 0.0063 0.006 0.0052 0.0046 0.0041 0.0035 0.0032 0.0031 0.0028 0.0024 0.0022 0.002 0.0018 0.0016 0.0014 0.0012 0.0011 0.001 0.0007 0.0007]\nSmallest 15 [-0.049 -0.0465 -0.0392 -0.039 -0.0354 -0.0339 -0.0308 -0.0297 -0.0291 -0.0285 -0.0249 -0.0234 -0.0228 -0.0192 -0.019 ]\nPositive composition-space overlap with position [0.028 0.027 0.027 0.027 0.026 0.05 0.034 0.029 0.028 0.017 0.022 0.029 0.02 0.024 0.059 0.07 0.049 0.095 0.063 0.18 0.157 0.235 0.132 0.274]\ncurrent read SV [5.217 4.652 4.602 4.101 4.026 3.99 3.691 3.508 3.387 3.352 3.266 3.163 3.043 2.978 2.898 2.744 2.717 2.514 2.338 2.144 1.946 1.868 1.618 1.451]\ncurrent read row-norm quantiles [0.04 0.074 0.085 0.104 0.308 0.752 1.332 2.341]\nprevious-head read SV [3.087 2.192 2.091 1.913 1.84 1.665 1.595 1.526 1.441 1.306 1.235 1.122 1.046 0.97 0.901 0.861 0.767 0.661 0.503 0.333 0.266 0.23 0.189 0.165]\nprevious-head read row-norm quantiles [0.014 0.024 0.028 0.034 0.085 0.339 0.579 0.937]\n\nFirst-MLP read composition strengths\n0 F norm 15.046 top SV [13.942 4.323 2.45 1.777 1.434 0.975 0.637 0.548 0.342 0.273 0.22 0.186]\n1 F norm 12.339 top SV [11.585 3.113 1.773 1.447 1.274 0.75 0.612 0.44 0.273 0.251 0.209 0.18 ]\n2 F norm 8.418 top SV [3.742 2.715 2.602 2.407 2.33 1.984 1.953 1.904 1.774 1.624 1.61 1.404]\n3 F norm 5.595 top SV [3.768 3.265 1.68 1.388 0.948 0.48 0.336 0.294 0.254 0.25 0.228 0.204]\n4 F norm 7.0 top SV [6.275 2.011 1.433 1.046 0.894 0.781 0.52 0.497 0.357 0.326 0.272 0.188]\n5 F norm 10.572 top SV [8.934 3.367 2.835 2.263 1.652 1.242 1.077 0.839 0.744 0.577 0.296 0.261]\n\nFinal-MLP projection spectrum tail and consecutive ratios\n16 13.4999 1.0156\n17 13.2929 1.0608\n18 12.5312 1.0105\n19 12.4015 1.0371\n20 11.9578 1.0185\n21 11.7405 1.0126\n22 11.5943 1.1184\n23 10.3667 1.2414\n24 8.3505 1.0356\n25 8.0636 1.0189\n26 7.9141 1.0607\n27 7.461 1.0412\n28 7.1656 1.2284\n29 5.8331 1.4326\n30 4.0718 1.2643\n31 3.2206 1.0261\n32 3.1387 1.1046\n33 2.8415 1.0148\n34 2.8001 1.0419\n35 2.6874 1.0454\n36 2.5707 1.0698\n37 2.4029 1.0321\n38 2.3282 1.0376\n39 2.2438 1.0269\n40 2.185 1.0019\n",
"seconds": 0.898,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "f0ba733eb49742a82933445e8a247da16265721a811d102b8beccbe11bfea86e",
"weights_sha256": "130135d2b27829df2734d78fa3af27b00b1cc1bf34c41b01a09b350e6c09a7de"
}L0 OV input/output energy in candidate position space, dimension 16 0 total input/output [0.8579 0.216 ] top 5 input-vector overlaps [0.9539 0.5383 0.4343 0.0771 0.0913] top 5 output-vector overlaps [0.2457 0.0895 0.1317 0.0972 0.0261] 1 total input/output [0.8805 0.1878] top 5 input-vector overlaps [0.925 0.878 0.1423 0.0393 0.0678] top 5 output-vector overlaps [0.2379 0.0876 0.1431 0.0817 0.042 ] 2 total input/output [0.085 0.1071] top 5 input-vector overlaps [0.0367 0.0384 0.0507 0.0456 0.0534] top 5 output-vector overlaps [0.0753 0.0774 0.1159 0.0668 0.047 ] 3 total input/output [0.1107 0.0716] top 5 input-vector overlaps [0.1557 0.0964 0.0366 0.094 0.0347] top 5 output-vector overlaps [0.1296 0.0318 0.0227 0.0476 0.066 ] 4 total input/output [0.7985 0.1946] top 5 input-vector overlaps [0.9263 0.7577 0.1711 0.048 0.0472] top 5 output-vector overlaps [0.2369 0.0879 0.13 0.0363 0.059 ] 5 total input/output [0.8505 0.1501] top 5 input-vector overlaps [0.8973 0.8437 0.1056 0.152 0.069 ] top 5 output-vector overlaps [0.145 0.1923 0.1161 0.1198 0.0688] L0 head top QK and OV singular-direction pairwise alignments Q vs Q [[ 1. -0.77 0.434 0.797 -0.129 -0.24 ] [-0.77 1. -0.528 -0.738 0.229 0.048] [ 0.434 -0.528 1. 0.492 -0.113 0.07 ] [ 0.797 -0.738 0.492 1. 0.064 0.033] [-0.129 0.229 -0.113 0.064 1. 0.326] [-0.24 0.048 0.07 0.033 0.326 1. ]] K vs K [[ 1. -0.264 -0.057 -0.003 -0.38 -0.098] [-0.264 1. 0.02 0.124 0.258 -0.143] [-0.057 0.02 1. 0.529 0.019 0.13 ] [-0.003 0.124 0.529 1. -0.101 0.012] [-0.38 0.258 0.019 -0.101 1. 0.643] [-0.098 -0.143 0.13 0.012 0.643 1. ]] V vs V [[ 1. 0.945 0.066 -0.044 0.962 -0.692] [ 0.945 1. 0.076 0.181 0.957 -0.639] [ 0.066 0.076 1. -0.005 0.037 -0.077] [-0.044 0.181 -0.005 1. 0.134 0.259] [ 0.962 0.957 0.037 0.134 1. -0.71 ] [-0.692 -0.639 -0.077 0.259 -0.71 1. ]] K vs V [[ 0.093 0.189 -0.417 0.262 0.156 0.075] [-0.011 -0.108 0.137 -0.175 -0.076 0.012] [-0.55 -0.535 -0.006 0.06 -0.531 0.509] [-0.81 -0.763 0.019 0.17 -0.826 0.926] [ 0.03 -0.198 -0.061 -0.471 -0.032 -0.166] [-0.211 -0.399 -0.244 -0.447 -0.243 -0.035]] Induction-candidate symmetric composition eigenvalues Largest 45 [0.1007 0.0826 0.0765 0.0738 0.0698 0.0647 0.0602 0.0561 0.0543 0.0525 0.0467 0.0417 0.039 0.035 0.0274 0.0252 0.0242 0.0186 0.0168 0.0127 0.0119 0.0111 0.0106 0.0089 0.0075 0.0063 0.006 0.0052 0.0046 0.0041 0.0035 0.0032 0.0031 0.0028 0.0024 0.0022 0.002 0.0018 0.0016 0.0014 0.0012 0.0011 0.001 0.0007 0.0007] Smallest 15 [-0.049 -0.0465 -0.0392 -0.039 -0.0354 -0.0339 -0.0308 -0.0297 -0.0291 -0.0285 -0.0249 -0.0234 -0.0228 -0.0192 -0.019 ] Positive composition-space overlap with position [0.028 0.027 0.027 0.027 0.026 0.05 0.034 0.029 0.028 0.017 0.022 0.029 0.02 0.024 0.059 0.07 0.049 0.095 0.063 0.18 0.157 0.235 0.132 0.274] current read SV [5.217 4.652 4.602 4.101 4.026 3.99 3.691 3.508 3.387 3.352 3.266 3.163 3.043 2.978 2.898 2.744 2.717 2.514 2.338 2.144 1.946 1.868 1.618 1.451] current read row-norm quantiles [0.04 0.074 0.085 0.104 0.308 0.752 1.332 2.341] previous-head read SV [3.087 2.192 2.091 1.913 1.84 1.665 1.595 1.526 1.441 1.306 1.235 1.122 1.046 0.97 0.901 0.861 0.767 0.661 0.503 0.333 0.266 0.23 0.189 0.165] previous-head read row-norm quantiles [0.014 0.024 0.028 0.034 0.085 0.339 0.579 0.937] First-MLP read composition strengths 0 F norm 15.046 top SV [13.942 4.323 2.45 1.777 1.434 0.975 0.637 0.548 0.342 0.273 0.22 0.186] 1 F norm 12.339 top SV [11.585 3.113 1.773 1.447 1.274 0.75 0.612 0.44 0.273 0.251 0.209 0.18 ] 2 F norm 8.418 top SV [3.742 2.715 2.602 2.407 2.33 1.984 1.953 1.904 1.774 1.624 1.61 1.404] 3 F norm 5.595 top SV [3.768 3.265 1.68 1.388 0.948 0.48 0.336 0.294 0.254 0.25 0.228 0.204] 4 F norm 7.0 top SV [6.275 2.011 1.433 1.046 0.894 0.781 0.52 0.497 0.357 0.326 0.272 0.188] 5 F norm 10.572 top SV [8.934 3.367 2.835 2.263 1.652 1.242 1.077 0.839 0.744 0.577 0.296 0.261] Final-MLP projection spectrum tail and consecutive ratios 16 13.4999 1.0156 17 13.2929 1.0608 18 12.5312 1.0105 19 12.4015 1.0371 20 11.9578 1.0185 21 11.7405 1.0126 22 11.5943 1.1184 23 10.3667 1.2414 24 8.3505 1.0356 25 8.0636 1.0189 26 7.9141 1.0607 27 7.461 1.0412 28 7.1656 1.2284 29 5.8331 1.4326 30 4.0718 1.2643 31 3.2206 1.0261 32 3.1387 1.1046 33 2.8415 1.0148 34 2.8001 1.0419 35 2.6874 1.0454 36 2.5707 1.0698 37 2.4029 1.0321 38 2.3282 1.0376 39 2.2438 1.0269 40 2.185 1.0019
Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 4 of at most 100: ``` L0 OV input/output energy in candidate position space, dimension 16 0 total input/output [0.8579 0.216 ] top 5 input-vector overlaps [0.9539 0.5383 0.4343 0.0771 0.0913] top 5 output-vector overlaps [0.2457 0.0895 0.1317 0.0972 0.0261] 1 total input/output [0.8805 0.1878] top 5 input-vector overlaps [0.925 0.878 0.1423 0.0393 0.0678] top 5 output-vector overlaps [0.2379 0.0876 0.1431 0.0817 0.042 ] 2 total input/output [0.085 0.1071] top 5 input-vector overlaps [0.0367 0.0384 0.0507 0.0456 0.0534] top 5 output-vector overlaps [0.0753 0.0774 0.1159 0.0668 0.047 ] 3 total input/output [0.1107 0.0716] top 5 input-vector overlaps [0.1557 0.0964 0.0366 0.094 0.0347] top 5 output-vector overlaps [0.1296 0.0318 0.0227 0.0476 0.066 ] 4 total input/output [0.7985 0.1946] top 5 input-vector overlaps [0.9263 0.7577 0.1711 0.048 0.0472] top 5 output-vector overlaps [0.2369 0.0879 0.13 0.0363 0.059 ] 5 total input/output [0.8505 0.1501] top 5 input-vector overlaps [0.8973 0.8437 0.1056 0.152 0.069 ] top 5 output-vector overlaps [0.145 0.1923 0.1161 0.1198 0.0688] L0 head top QK and OV singular-direction pairwise alignments Q vs Q [[ 1. -0.77 0.434 0.797 -0.129 -0.24 ] [-0.77 1. -0.528 -0.738 0.229 0.048] [ 0.434 -0.528 1. 0.492 -0.113 0.07 ] [ 0.797 -0.738 0.492 1. 0.064 0.033] [-0.129 0.229 -0.113 0.064 1. 0.326] [-0.24 0.048 0.07 0.033 0.326 1. ]] K vs K [[ 1. -0.264 -0.057 -0.003 -0.38 -0.098] [-0.264 1. 0.02 0.124 0.258 -0.143] [-0.057 0.02 1. 0.529 0.019 0.13 ] [-0.003 0.124 0.529 1. -0.101 0.012] [-0.38 0.258 0.019 -0.101 1. 0.643] [-0.098 -0.143 0.13 0.012 0.643 1. ]] V vs V [[ 1. 0.945 0.066 -0.044 0.962 -0.692] [ 0.945 1. 0.076 0.181 0.957 -0.639] [ 0.066 0.076 1. -0.005 0.037 -0.077] [-0.044 0.181 -0.005 1. 0.134 0.259] [ 0.962 0.957 0.037 0.134 1. -0.71 ] [-0.692 -0.639 -0.077 0.259 -0.71 1. ]] K vs V [[ 0.093 0.189 -0.417 0.262 0.156 0.075] [-0.011 -0.108 0.137 -0.175 -0.076 0.012] [-0.55 -0.535 -0.006 0.06 -0.531 0.509] [-0.81 -0.763 0.019 0.17 -0.826 0.926] [ 0.03 -0.198 -0.061 -0.471 -0.032 -0.166] [-0.211 -0.399 -0.244 -0.447 -0.243 -0.035]] Induction-candidate symmetric composition eigenvalues Largest 45 [0.1007 0.0826 0.0765 0.0738 0.0698 0.0647 0.0602 0.0561 0.0543 0.0525 0.0467 0.0417 0.039 0.035 0.0274 0.0252 0.0242 0.0186 0.0168 0.0127 0.0119 0.0111 0.0106 0.0089 0.0075 0.0063 0.006 0.0052 0.0046 0.0041 0.0035 0.0032 0.0031 0.0028 0.0024 0.0022 0.002 0.0018 0.0016 0.0014 0.0012 0.0011 0.001 0.0007 0.0007] Smallest 15 [-0.049 -0.0465 -0.0392 -0.039 -0.0354 -0.0339 -0.0308 -0.0297 -0.0291 -0.0285 -0.0249 -0.0234 -0.0228 -0.0192 -0.019 ] Positive composition-space overlap with position [0.028 0.027 0.027 0.027 0.026 0.05 0.034 0.029 0.028 0.017 0.022 0.029 0.02 0.024 0.059 0.07 0.049 0.095 0.063 0.18 0.157 0.235 0.132 0.274] current read SV [5.217 4.652 4.602 4.101 4.026 3.99 3.691 3.508 3.387 3.352 3.266 3.163 3.043 2.978 2.898 2.744 2.717 2.514 2.338 2.144 1.946 1.868 1.618 1.451] current read row-norm quantiles [0.04 0.074 0.085 0.104 0.308 0.752 1.332 2.341] previous-head read SV [3.087 2.192 2.091 1.913 1.84 1.665 1.595 1.526 1.441 1.306 1.235 1.122 1.046 0.97 0.901 0.861 0.767 0.661 0.503 0.333 0.266 0.23 0.189 0.165] previous-head read row-norm quantiles [0.014 0.024 0.028 0.034 0.085 0.339 0.579 0.937] First-MLP read composition strengths 0 F norm 15.046 top SV [13.942 4.323 2.45 1.777 1.434 0.975 0.637 0.548 0.342 0.273 0.22 0.186] 1 F norm 12.339 top SV [11.585 3.113 1.773 1.447 1.274 0.75 0.612 0.44 0.273 0.251 0.209 0.18 ] 2 F norm 8.418 top SV [3.742 2.715 2.602 2.407 2.33 1.984 1.953 1.904 1.774 1.624 1.61 1.404] 3 F norm 5.595 top SV [3.768 3.265 1.68 1.388 0.948 0.48 0.336 0.294 0.254 0.25 0.228 0.204] 4 F norm 7.0 top SV [6.275 2.011 1.433 1.046 0.894 0.781 0.52 0.497 0.357 0.326 0.272 0.188] 5 F norm 10.572 top SV [8.934 3.367 2.835 2.263 1.652 1.242 1.077 0.839 0.744 0.577 0.296 0.261] Final-MLP projection spectrum tail and consecutive ratios 16 13.4999 1.0156 17 13.2929 1.0608 18 12.5312 1.0105 19 12.4015 1.0371 20 11.9578 1.0185 21 11.7405 1.0126 22 11.5943 1.1184 23 10.3667 1.2414 24 8.3505 1.0356 25 8.0636 1.0189 26 7.9141 1.0607 27 7.461 1.0412 28 7.1656 1.2284 29 5.8331 1.4326 30 4.0718 1.2643 31 3.2206 1.0261 32 3.1387 1.1046 33 2.8415 1.0148 34 2.8001 1.0419 35 2.6874 1.0454 36 2.5707 1.0698 37 2.4029 1.0321 38 2.3282 1.0376 39 2.2438 1.0269 40 2.185 1.0019 ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from threadpoolctl import threadpool_limits
threadpool_limits(2)
np.set_printoptions(precision=3,suppress=True,linewidth=180)
d=384
P=np.eye(d)-np.ones((d,d))/d
def products(l,h):
s=slice(64*h,64*(h+1))
g=W[f"ln1_w.{l}"].astype(float)
q,k,v=[(W[f"W_{t}.{l}"][s].astype(float)*g)@P for t in ["q","k","v"]]
return q.T@k/8, P@W[f"W_o.{l}"][:,s].astype(float)@v
a02,b02=products(0,2)
a31,b31=products(3,1)
c=a31@b02
ev,u=np.linalg.eigh((c+c.T)/2)
T=u[:,np.argsort(ev)[::-1][:24]]
f=(W["W_fc.0"].astype(float)*W["ln2_w.0"])@P
o=P@W["W_proj.0"].astype(float)
X=f@T
Y=f@b02@T
nx=np.linalg.norm(X,axis=1)
ny=np.linalg.norm(Y,axis=1)
on=np.linalg.norm(o,axis=0)
print("First-MLP output-column norm quantiles",
np.quantile(on,[0,.1,.25,.5,.75,.9,.99,1]))
print("First-MLP current/previous/output norm correlations",
np.corrcoef(np.stack([nx,ny,on])))
print("Number of neurons above current-read thresholds",
[(t,int(np.sum(nx>t))) for t in [.1,.2,.3,.5,1]])
print("Number above previous-read thresholds",
[(t,int(np.sum(ny>t))) for t in [.05,.1,.2,.3,.5]])
# Cluster weight directions only. No nonlinear layer is evaluated.
for whiten in [False,True]:
C=np.concatenate([X[nx>.3],Y[ny>.15]],axis=0)
if whiten:
cov=C.T@C/len(C)
e,v=np.linalg.eigh(cov)
D=(v*(1/np.sqrt(np.maximum(e,.01*e.max()))))@v.T
else:
D=np.eye(24)
xx=X@D; yy=Y@D
xx=xx/np.maximum(np.linalg.norm(xx,axis=1,keepdims=True),1e-12)
yy=yy/np.maximum(np.linalg.norm(yy,axis=1,keepdims=True),1e-12)
Z=np.concatenate([xx[nx>.3],yy[ny>.15]],axis=0)
gram=Z@Z.T
np.fill_diagonal(gram,-1)
print("\nWhiten",whiten,"samples",len(Z),
"nearest-other cosine quantiles",np.quantile(gram.max(axis=1),[0,.1,.5,.9,1]))
for k in [16,20,24,28,32,40]:
km=KMeans(n_clusters=k,n_init=10,random_state=0).fit(Z)
centers=km.cluster_centers_
centers/=np.maximum(np.linalg.norm(centers,axis=1,keepdims=True),1e-12)
sim=Z@centers.T
print("K",k,"mean best cosine",round(sim.max(axis=1).mean(),3),
"silhouette",round(silhouette_score(Z,km.labels_,metric="cosine"),3),
"sorted counts",np.sort(np.bincount(km.labels_))[::-1])
if k==24:
sx=xx@centers.T; sy=yy@centers.T
ix=sx.argmax(axis=1); iy=sy.argmax(axis=1)
good=(nx>.3)&(ny>.15)&(sx.max(axis=1)>.65)&(sy.max(axis=1)>.65)
counts=np.zeros((k,k))
for i in np.where(good)[0]:
counts[iy[i],ix[i]]+=1
print("paired clear weight directions:",int(good.sum()))
print("nonzero previous/current cluster pairs:",
[(int(i),int(j),int(counts[i,j])) for i,j in zip(*np.where(counts>=2))])
if counts.sum()>0:
adj=counts+counts.T
deg=adj.sum(axis=1)
norm=adj/np.sqrt(np.maximum(deg[:,None]*deg[None,:],1))
print("symmetric normalized graph eigenvalues",
np.linalg.eigvalsh(norm))
```{
"run": 5,
"turn": "t005",
"code": "import numpy as np\nfrom sklearn.cluster import KMeans\nfrom sklearn.metrics import silhouette_score\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(2)\nnp.set_printoptions(precision=3,suppress=True,linewidth=180)\n\nd=384\nP=np.eye(d)-np.ones((d,d))/d\ndef products(l,h):\n s=slice(64*h,64*(h+1))\n g=W[f\"ln1_w.{l}\"].astype(float)\n q,k,v=[(W[f\"W_{t}.{l}\"][s].astype(float)*g)@P for t in [\"q\",\"k\",\"v\"]]\n return q.T@k/8, P@W[f\"W_o.{l}\"][:,s].astype(float)@v\na02,b02=products(0,2)\na31,b31=products(3,1)\nc=a31@b02\nev,u=np.linalg.eigh((c+c.T)/2)\nT=u[:,np.argsort(ev)[::-1][:24]]\n\nf=(W[\"W_fc.0\"].astype(float)*W[\"ln2_w.0\"])@P\no=P@W[\"W_proj.0\"].astype(float)\nX=f@T\nY=f@b02@T\nnx=np.linalg.norm(X,axis=1)\nny=np.linalg.norm(Y,axis=1)\non=np.linalg.norm(o,axis=0)\nprint(\"First-MLP output-column norm quantiles\",\n np.quantile(on,[0,.1,.25,.5,.75,.9,.99,1]))\nprint(\"First-MLP current/previous/output norm correlations\",\n np.corrcoef(np.stack([nx,ny,on])))\nprint(\"Number of neurons above current-read thresholds\",\n [(t,int(np.sum(nx>t))) for t in [.1,.2,.3,.5,1]])\nprint(\"Number above previous-read thresholds\",\n [(t,int(np.sum(ny>t))) for t in [.05,.1,.2,.3,.5]])\n\n# Cluster weight directions only. No nonlinear layer is evaluated.\nfor whiten in [False,True]:\n C=np.concatenate([X[nx>.3],Y[ny>.15]],axis=0)\n if whiten:\n cov=C.T@C/len(C)\n e,v=np.linalg.eigh(cov)\n D=(v*(1/np.sqrt(np.maximum(e,.01*e.max()))))@v.T\n else:\n D=np.eye(24)\n xx=X@D; yy=Y@D\n xx=xx/np.maximum(np.linalg.norm(xx,axis=1,keepdims=True),1e-12)\n yy=yy/np.maximum(np.linalg.norm(yy,axis=1,keepdims=True),1e-12)\n Z=np.concatenate([xx[nx>.3],yy[ny>.15]],axis=0)\n gram=Z@Z.T\n np.fill_diagonal(gram,-1)\n print(\"\\nWhiten\",whiten,\"samples\",len(Z),\n \"nearest-other cosine quantiles\",np.quantile(gram.max(axis=1),[0,.1,.5,.9,1]))\n for k in [16,20,24,28,32,40]:\n km=KMeans(n_clusters=k,n_init=10,random_state=0).fit(Z)\n centers=km.cluster_centers_\n centers/=np.maximum(np.linalg.norm(centers,axis=1,keepdims=True),1e-12)\n sim=Z@centers.T\n print(\"K\",k,\"mean best cosine\",round(sim.max(axis=1).mean(),3),\n \"silhouette\",round(silhouette_score(Z,km.labels_,metric=\"cosine\"),3),\n \"sorted counts\",np.sort(np.bincount(km.labels_))[::-1])\n if k==24:\n sx=xx@centers.T; sy=yy@centers.T\n ix=sx.argmax(axis=1); iy=sy.argmax(axis=1)\n good=(nx>.3)&(ny>.15)&(sx.max(axis=1)>.65)&(sy.max(axis=1)>.65)\n counts=np.zeros((k,k))\n for i in np.where(good)[0]:\n counts[iy[i],ix[i]]+=1\n print(\"paired clear weight directions:\",int(good.sum()))\n print(\"nonzero previous/current cluster pairs:\",\n [(int(i),int(j),int(counts[i,j])) for i,j in zip(*np.where(counts>=2))])\n if counts.sum()>0:\n adj=counts+counts.T\n deg=adj.sum(axis=1)\n norm=adj/np.sqrt(np.maximum(deg[:,None]*deg[None,:],1))\n print(\"symmetric normalized graph eigenvalues\",\n np.linalg.eigvalsh(norm))",
"out": "First-MLP output-column norm quantiles [0.132 0.172 0.194 0.237 1.272 1.99 2.465 4.795]\nFirst-MLP current/previous/output norm correlations [[1. 0.835 0.924]\n [0.835 1. 0.88 ]\n [0.924 0.88 1. ]]\nNumber of neurons above current-read thresholds [(0.1, 827), (0.2, 426), (0.3, 386), (0.5, 326), (1, 50)]\nNumber above previous-read thresholds [(0.05, 464), (0.1, 355), (0.2, 276), (0.3, 198), (0.5, 32)]\n\nWhiten False samples 686 nearest-other cosine quantiles [0.438 0.654 0.849 0.988 0.997]\nK 16 mean best cosine 0.615 silhouette 0.192 sorted counts [77 71 69 68 55 46 41 38 36 35 29 26 26 25 23 21]\nK 20 mean best cosine 0.66 silhouette 0.225 sorted counts [58 57 53 50 47 40 37 34 34 33 31 31 28 28 24 23 23 21 18 16]\nK 24 mean best cosine 0.686 silhouette 0.245 sorted counts [49 41 40 38 38 34 34 33 31 28 28 28 28 26 24 24 23 22 22 21 20 20 18 16]\npaired clear weight directions: 97\nnonzero previous/current cluster pairs: [(0, 0, 3), (0, 8, 2), (1, 4, 2), (3, 3, 5), (5, 5, 6), (6, 6, 6), (7, 7, 2), (11, 11, 4), (12, 9, 2), (12, 10, 2), (12, 20, 2), (13, 4, 2), (13, 13, 3), (15, 15, 3), (16, 16, 2), (17, 4, 2), (17, 8, 2), (18, 18, 4), (19, 19, 4), (22, 22, 3), (23, 14, 3)]\nsymmetric normalized graph eigenvalues [-0.587 -0.511 -0.407 -0.219 -0.066 0.135 0.237 0.337 0.383 0.428 0.506 0.538 0.601 0.634 0.675 0.78 0.796 0.824 0.852 0.908 0.965 1. 1. 1. ]\nK 28 mean best cosine 0.713 silhouette 0.265 sorted counts [47 37 37 36 31 31 31 29 26 26 26 24 24 24 24 23 23 20 20 20 19 18 18 17 17 16 13 9]\nK 32 mean best cosine 0.726 silhouette 0.262 sorted counts [36 35 29 28 28 28 27 25 25 25 25 24 23 23 23 20 20 20 19 19 19 18 17 17 16 16 16 15 14 12 12 12]\nK 40 mean best cosine 0.755 silhouette 0.277 sorted counts [29 28 25 23 22 22 21 21 21 21 20 20 20 19 19 18 18 18 18 18 17 17 17 17 16 15 14 14 14 14 14 13 13 12 11 11 11 11 8 6]\n\nWhiten True samples 686 nearest-other cosine quantiles [0.474 0.615 0.809 0.985 0.997]\nK 16 mean best cosine 0.571 silhouette 0.178 sorted counts [80 65 62 59 51 49 46 39 37 33 32 29 29 27 26 22]\nK 20 mean best cosine 0.617 silhouette 0.212 sorted counts [55 51 51 46 42 42 38 36 34 33 32 32 31 28 27 27 26 21 17 17]\nK 24 mean best cosine 0.65 silhouette 0.232 sorted counts [45 41 39 39 39 37 34 32 32 32 29 29 27 26 23 23 23 23 22 21 20 19 16 15]\npaired clear weight directions: 72\nnonzero previous/current cluster pairs: [(0, 0, 4), (1, 1, 5), (3, 3, 2), (4, 4, 4), (7, 7, 4), (9, 9, 4), (10, 10, 3), (11, 11, 3), (12, 0, 2), (13, 13, 2), (14, 14, 4), (17, 17, 4), (18, 18, 4), (21, 21, 2), (22, 22, 2), (23, 23, 6)]\nsymmetric normalized graph eigenvalues [-0.738 -0.736 -0.306 -0.281 -0. 0.061 0.435 0.454 0.522 0.759 0.788 0.889 0.897 0.906 0.908 0.947 1. 1. 1. 1. 1. 1. 1. 1. ]\nK 28 mean best cosine 0.678 silhouette 0.248 sorted counts [37 34 34 34 33 33 33 29 27 26 26 26 25 24 24 23 23 22 22 22 18 17 17 16 16 16 15 14]\nK 32 mean best cosine 0.699 silhouette 0.263 sorted counts [32 31 30 28 28 27 25 24 24 23 23 23 23 22 22 22 21 21 20 20 20 19 19 19 19 18 17 15 14 14 13 10]\nK 40 mean best cosine 0.722 silhouette 0.26 sorted counts [34 34 28 25 25 22 22 22 21 21 20 20 19 19 19 18 18 18 18 18 18 17 17 15 15 14 14 13 13 13 12 11 11 10 10 9 9 9 8 7]\n",
"seconds": 1.389,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "9b0b95b8426d0dfcd294da8b9d18789b210f26324f1838faf4dc0f73f6f17412",
"weights_sha256": "130135d2b27829df2734d78fa3af27b00b1cc1bf34c41b01a09b350e6c09a7de"
}First-MLP output-column norm quantiles [0.132 0.172 0.194 0.237 1.272 1.99 2.465 4.795] First-MLP current/previous/output norm correlations [[1. 0.835 0.924] [0.835 1. 0.88 ] [0.924 0.88 1. ]] Number of neurons above current-read thresholds [(0.1, 827), (0.2, 426), (0.3, 386), (0.5, 326), (1, 50)] Number above previous-read thresholds [(0.05, 464), (0.1, 355), (0.2, 276), (0.3, 198), (0.5, 32)] Whiten False samples 686 nearest-other cosine quantiles [0.438 0.654 0.849 0.988 0.997] K 16 mean best cosine 0.615 silhouette 0.192 sorted counts [77 71 69 68 55 46 41 38 36 35 29 26 26 25 23 21] K 20 mean best cosine 0.66 silhouette 0.225 sorted counts [58 57 53 50 47 40 37 34 34 33 31 31 28 28 24 23 23 21 18 16] K 24 mean best cosine 0.686 silhouette 0.245 sorted counts [49 41 40 38 38 34 34 33 31 28 28 28 28 26 24 24 23 22 22 21 20 20 18 16] paired clear weight directions: 97 nonzero previous/current cluster pairs: [(0, 0, 3), (0, 8, 2), (1, 4, 2), (3, 3, 5), (5, 5, 6), (6, 6, 6), (7, 7, 2), (11, 11, 4), (12, 9, 2), (12, 10, 2), (12, 20, 2), (13, 4, 2), (13, 13, 3), (15, 15, 3), (16, 16, 2), (17, 4, 2), (17, 8, 2), (18, 18, 4), (19, 19, 4), (22, 22, 3), (23, 14, 3)] symmetric normalized graph eigenvalues [-0.587 -0.511 -0.407 -0.219 -0.066 0.135 0.237 0.337 0.383 0.428 0.506 0.538 0.601 0.634 0.675 0.78 0.796 0.824 0.852 0.908 0.965 1. 1. 1. ] K 28 mean best cosine 0.713 silhouette 0.265 sorted counts [47 37 37 36 31 31 31 29 26 26 26 24 24 24 24 23 23 20 20 20 19 18 18 17 17 16 13 9] K 32 mean best cosine 0.726 silhouette 0.262 sorted counts [36 35 29 28 28 28 27 25 25 25 25 24 23 23 23 20 20 20 19 19 19 18 17 17 16 16 16 15 14 12 12 12] K 40 mean best cosine 0.755 silhouette 0.277 sorted counts [29 28 25 23 22 22 21 21 21 21 20 20 20 19 19 18 18 18 18 18 17 17 17 17 16 15 14 14 14 14 14 13 13 12 11 11 11 11 8 6] Whiten True samples 686 nearest-other cosine quantiles [0.474 0.615 0.809 0.985 0.997] K 16 mean best cosine 0.571 silhouette 0.178 sorted counts [80 65 62 59 51 49 46 39 37 33 32 29 29 27 26 22] K 20 mean best cosine 0.617 silhouette 0.212 sorted counts [55 51 51 46 42 42 38 36 34 33 32 32 31 28 27 27 26 21 17 17] K 24 mean best cosine 0.65 silhouette 0.232 sorted counts [45 41 39 39 39 37 34 32 32 32 29 29 27 26 23 23 23 23 22 21 20 19 16 15] paired clear weight directions: 72 nonzero previous/current cluster pairs: [(0, 0, 4), (1, 1, 5), (3, 3, 2), (4, 4, 4), (7, 7, 4), (9, 9, 4), (10, 10, 3), (11, 11, 3), (12, 0, 2), (13, 13, 2), (14, 14, 4), (17, 17, 4), (18, 18, 4), (21, 21, 2), (22, 22, 2), (23, 23, 6)] symmetric normalized graph eigenvalues [-0.738 -0.736 -0.306 -0.281 -0. 0.061 0.435 0.454 0.522 0.759 0.788 0.889 0.897 0.906 0.908 0.947 1. 1. 1. 1. 1. 1. 1. 1. ] K 28 mean best cosine 0.678 silhouette 0.248 sorted counts [37 34 34 34 33 33 33 29 27 26 26 26 25 24 24 23 23 22 22 22 18 17 17 16 16 16 15 14] K 32 mean best cosine 0.699 silhouette 0.263 sorted counts [32 31 30 28 28 27 25 24 24 23 23 23 23 22 22 22 21 21 20 20 20 19 19 19 19 18 17 15 14 14 13 10] K 40 mean best cosine 0.722 silhouette 0.26 sorted counts [34 34 28 25 25 22 22 22 21 21 20 20 19 19 19 18 18 18 18 18 18 17 17 15 15 14 14 13 13 13 12 11 11 10 10 9 9 9 8 7]
Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 5 of at most 100: ``` First-MLP output-column norm quantiles [0.132 0.172 0.194 0.237 1.272 1.99 2.465 4.795] First-MLP current/previous/output norm correlations [[1. 0.835 0.924] [0.835 1. 0.88 ] [0.924 0.88 1. ]] Number of neurons above current-read thresholds [(0.1, 827), (0.2, 426), (0.3, 386), (0.5, 326), (1, 50)] Number above previous-read thresholds [(0.05, 464), (0.1, 355), (0.2, 276), (0.3, 198), (0.5, 32)] Whiten False samples 686 nearest-other cosine quantiles [0.438 0.654 0.849 0.988 0.997] K 16 mean best cosine 0.615 silhouette 0.192 sorted counts [77 71 69 68 55 46 41 38 36 35 29 26 26 25 23 21] K 20 mean best cosine 0.66 silhouette 0.225 sorted counts [58 57 53 50 47 40 37 34 34 33 31 31 28 28 24 23 23 21 18 16] K 24 mean best cosine 0.686 silhouette 0.245 sorted counts [49 41 40 38 38 34 34 33 31 28 28 28 28 26 24 24 23 22 22 21 20 20 18 16] paired clear weight directions: 97 nonzero previous/current cluster pairs: [(0, 0, 3), (0, 8, 2), (1, 4, 2), (3, 3, 5), (5, 5, 6), (6, 6, 6), (7, 7, 2), (11, 11, 4), (12, 9, 2), (12, 10, 2), (12, 20, 2), (13, 4, 2), (13, 13, 3), (15, 15, 3), (16, 16, 2), (17, 4, 2), (17, 8, 2), (18, 18, 4), (19, 19, 4), (22, 22, 3), (23, 14, 3)] symmetric normalized graph eigenvalues [-0.587 -0.511 -0.407 -0.219 -0.066 0.135 0.237 0.337 0.383 0.428 0.506 0.538 0.601 0.634 0.675 0.78 0.796 0.824 0.852 0.908 0.965 1. 1. 1. ] K 28 mean best cosine 0.713 silhouette 0.265 sorted counts [47 37 37 36 31 31 31 29 26 26 26 24 24 24 24 23 23 20 20 20 19 18 18 17 17 16 13 9] K 32 mean best cosine 0.726 silhouette 0.262 sorted counts [36 35 29 28 28 28 27 25 25 25 25 24 23 23 23 20 20 20 19 19 19 18 17 17 16 16 16 15 14 12 12 12] K 40 mean best cosine 0.755 silhouette 0.277 sorted counts [29 28 25 23 22 22 21 21 21 21 20 20 20 19 19 18 18 18 18 18 17 17 17 17 16 15 14 14 14 14 14 13 13 12 11 11 11 11 8 6] Whiten True samples 686 nearest-other cosine quantiles [0.474 0.615 0.809 0.985 0.997] K 16 mean best cosine 0.571 silhouette 0.178 sorted counts [80 65 62 59 51 49 46 39 37 33 32 29 29 27 26 22] K 20 mean best cosine 0.617 silhouette 0.212 sorted counts [55 51 51 46 42 42 38 36 34 33 32 32 31 28 27 27 26 21 17 17] K 24 mean best cosine 0.65 silhouette 0.232 sorted counts [45 41 39 39 39 37 34 32 32 32 29 29 27 26 23 23 23 23 22 21 20 19 16 15] paired clear weight directions: 72 nonzero previous/current cluster pairs: [(0, 0, 4), (1, 1, 5), (3, 3, 2), (4, 4, 4), (7, 7, 4), (9, 9, 4), (10, 10, 3), (11, 11, 3), (12, 0, 2), (13, 13, 2), (14, 14, 4), (17, 17, 4), (18, 18, 4), (21, 21, 2), (22, 22, 2), (23, 23, 6)] symmetric normalized graph eigenvalues [-0.738 -0.736 -0.306 -0.281 -0. 0.061 0.435 0.454 0.522 0.759 0.788 0.889 0.897 0.906 0.908 0.947 1. 1. 1. 1. 1. 1. 1. 1. ] K 28 mean best cosine 0.678 silhouette 0.248 sorted counts [37 34 34 34 33 33 33 29 27 26 26 26 25 24 24 23 23 22 22 22 18 17 17 16 16 16 15 14] K 32 mean best cosine 0.699 silhouette 0.263 sorted counts [32 31 30 28 28 27 25 24 24 23 23 23 23 22 22 22 21 21 20 20 20 19 19 19 19 18 17 15 14 14 13 10] K 40 mean best cosine 0.722 silhouette 0.26 sorted counts [34 34 28 25 25 22 22 22 21 21 20 20 19 19 19 18 18 18 18 18 18 17 17 15 15 14 14 13 13 13 12 11 11 10 10 9 9 9 8 7] ```
Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
from threadpoolctl import threadpool_limits
threadpool_limits(2)
np.set_printoptions(precision=3,suppress=True,linewidth=180)
d=384
P=np.eye(d)-np.ones((d,d))/d
A={}; B={}; kb={}
for l in range(4):
g=W[f"ln1_w.{l}"].astype(float)
b=W[f"ln1_b.{l}"].astype(float)
for h in range(6):
s=slice(64*h,64*(h+1))
q,k,v=[W[f"W_{t}.{l}"][s].astype(float) for t in ["q","k","v"]]
qg,kg,vg=[(m*g)@P for m in [q,k,v]]
A[l,h]=qg.T@kg/8
B[l,h]=P@W[f"W_o.{l}"][:,s].astype(float)@vg
kb[l,h]=kg.T@(q@b)/8
r=sum(kb[1,h]/np.linalg.norm(kb[1,h]) for h in range(6))
r/=np.linalg.norm(r)
print("Common L1 key-selector direction: each head's cosine and bias/QK norm ratio")
for l in range(4):
print(l,np.round([[kb[l,h]@r/np.linalg.norm(kb[l,h]),
np.linalg.norm(kb[l,h])/np.linalg.norm(A[l,h])]
for h in range(6)],3))
ub,sb,_=np.linalg.svd(np.concatenate([B[0,h] for h in [0,1,4,5]],axis=1),full_matrices=False)
Tb=ub[:,:4]
print("Common selector fraction in top 2/4/8 broad-output dimensions",
[round(np.linalg.norm(ub[:,:k].T@r)**2,4) for k in [2,4,8]])
print("Common selector composed with L0 OV heads: norms",
[round(np.linalg.norm(r@B[0,h]),4) for h in range(6)])
u0,s0,_=np.linalg.svd(P@W["W_proj.0"].astype(float),full_matrices=False)
u1,s1,_=np.linalg.svd(P@W["W_proj.1"].astype(float),full_matrices=False)
_,_,v0=np.linalg.svd((W["W_v.0"][128:192].astype(float)*W["ln1_w.0"])@P,full_matrices=False)
spaces={"V0H2-24":v0[:24].T,
"MLP0-32":u0[:,:32],
"MLP0-64":u0[:,:64],
"MLP1-64":u1[:,:64]}
print("\nProjected key-composition scores: strongest earlier heads")
print("tuple = (earlier head, trace/F, symmetry, norm)")
for name,T in spaces.items():
print("\nSPACE",name)
for l in [2,3]:
for h in range(6):
results=[]
for j in range(l):
for k in range(6):
c=T.T@A[l,h]@B[j,k]@T
fn=np.linalg.norm(c)
results.append((f"L{j}H{k}",np.trace(c)/fn,
np.sum(c*c.T)/(fn*fn),fn))
results.sort(key=lambda x:x[1],reverse=True)
print(f"L{l}H{h}",[(n,round(t,3),round(y,3),round(f,3))
for n,t,y,f in results[:2]])
print("\nDirect-match versus predecessor-match on V0H2's 24-dimensional read space")
T=spaces["V0H2-24"]
for l in [1,2,3]:
for h in range(6):
out=[]
for c in [T.T@A[l,h]@T,T.T@A[l,h]@B[0,2]@T,
T.T@B[0,2].T@A[l,h]@T]:
fn=np.linalg.norm(c)
out.append([np.trace(c)/fn,np.sum(c*c.T)/(fn*fn),fn])
print(f"L{l}H{h}",np.round(out,3))
print("\nMLP neuron weight-norm quantiles")
for l in range(4):
a=np.linalg.norm(W[f"W_fc.{l}"],axis=1)
b=np.linalg.norm(W[f"W_proj.{l}"],axis=0)
print(l,"read",np.round(np.quantile(a,[.1,.25,.5,.75,.9]),3),
"write",np.round(np.quantile(b,[.1,.25,.5,.75,.9]),3),
"small-write fraction",round(float(np.mean(b<.4)),3))
```{
"run": 6,
"turn": "t006",
"code": "import numpy as np\nfrom threadpoolctl import threadpool_limits\nthreadpool_limits(2)\nnp.set_printoptions(precision=3,suppress=True,linewidth=180)\n\nd=384\nP=np.eye(d)-np.ones((d,d))/d\nA={}; B={}; kb={}\nfor l in range(4):\n g=W[f\"ln1_w.{l}\"].astype(float)\n b=W[f\"ln1_b.{l}\"].astype(float)\n for h in range(6):\n s=slice(64*h,64*(h+1))\n q,k,v=[W[f\"W_{t}.{l}\"][s].astype(float) for t in [\"q\",\"k\",\"v\"]]\n qg,kg,vg=[(m*g)@P for m in [q,k,v]]\n A[l,h]=qg.T@kg/8\n B[l,h]=P@W[f\"W_o.{l}\"][:,s].astype(float)@vg\n kb[l,h]=kg.T@(q@b)/8\n\nr=sum(kb[1,h]/np.linalg.norm(kb[1,h]) for h in range(6))\nr/=np.linalg.norm(r)\nprint(\"Common L1 key-selector direction: each head's cosine and bias/QK norm ratio\")\nfor l in range(4):\n print(l,np.round([[kb[l,h]@r/np.linalg.norm(kb[l,h]),\n np.linalg.norm(kb[l,h])/np.linalg.norm(A[l,h])]\n for h in range(6)],3))\n\nub,sb,_=np.linalg.svd(np.concatenate([B[0,h] for h in [0,1,4,5]],axis=1),full_matrices=False)\nTb=ub[:,:4]\nprint(\"Common selector fraction in top 2/4/8 broad-output dimensions\",\n [round(np.linalg.norm(ub[:,:k].T@r)**2,4) for k in [2,4,8]])\nprint(\"Common selector composed with L0 OV heads: norms\",\n [round(np.linalg.norm(r@B[0,h]),4) for h in range(6)])\n\nu0,s0,_=np.linalg.svd(P@W[\"W_proj.0\"].astype(float),full_matrices=False)\nu1,s1,_=np.linalg.svd(P@W[\"W_proj.1\"].astype(float),full_matrices=False)\n_,_,v0=np.linalg.svd((W[\"W_v.0\"][128:192].astype(float)*W[\"ln1_w.0\"])@P,full_matrices=False)\nspaces={\"V0H2-24\":v0[:24].T,\n \"MLP0-32\":u0[:,:32],\n \"MLP0-64\":u0[:,:64],\n \"MLP1-64\":u1[:,:64]}\n\nprint(\"\\nProjected key-composition scores: strongest earlier heads\")\nprint(\"tuple = (earlier head, trace/F, symmetry, norm)\")\nfor name,T in spaces.items():\n print(\"\\nSPACE\",name)\n for l in [2,3]:\n for h in range(6):\n results=[]\n for j in range(l):\n for k in range(6):\n c=T.T@A[l,h]@B[j,k]@T\n fn=np.linalg.norm(c)\n results.append((f\"L{j}H{k}\",np.trace(c)/fn,\n np.sum(c*c.T)/(fn*fn),fn))\n results.sort(key=lambda x:x[1],reverse=True)\n print(f\"L{l}H{h}\",[(n,round(t,3),round(y,3),round(f,3))\n for n,t,y,f in results[:2]])\n\nprint(\"\\nDirect-match versus predecessor-match on V0H2's 24-dimensional read space\")\nT=spaces[\"V0H2-24\"]\nfor l in [1,2,3]:\n for h in range(6):\n out=[]\n for c in [T.T@A[l,h]@T,T.T@A[l,h]@B[0,2]@T,\n T.T@B[0,2].T@A[l,h]@T]:\n fn=np.linalg.norm(c)\n out.append([np.trace(c)/fn,np.sum(c*c.T)/(fn*fn),fn])\n print(f\"L{l}H{h}\",np.round(out,3))\n\nprint(\"\\nMLP neuron weight-norm quantiles\")\nfor l in range(4):\n a=np.linalg.norm(W[f\"W_fc.{l}\"],axis=1)\n b=np.linalg.norm(W[f\"W_proj.{l}\"],axis=0)\n print(l,\"read\",np.round(np.quantile(a,[.1,.25,.5,.75,.9]),3),\n \"write\",np.round(np.quantile(b,[.1,.25,.5,.75,.9]),3),\n \"small-write fraction\",round(float(np.mean(b<.4)),3))",
"out": "Common L1 key-selector direction: each head's cosine and bias/QK norm ratio\n0 [[-0.118 0.25 ]\n [-0.403 0.113]\n [ 0.149 0.035]\n [-0.039 0.136]\n [-0.141 0.106]\n [ 0.387 0.21 ]]\n1 [[0.98 1.531]\n [0.987 1.738]\n [0.989 1.671]\n [0.985 1.711]\n [0.984 1.58 ]\n [0.986 1.312]]\n2 [[0.806 0.697]\n [0.781 0.36 ]\n [0.786 0.468]\n [0.814 0.701]\n [0.804 0.673]\n [0.814 0.67 ]]\n3 [[ 0.318 0.116]\n [-0.129 0.132]\n [ 0.621 0.192]\n [ 0.652 0.225]\n [ 0.602 0.166]\n [ 0.08 0.197]]\nCommon selector fraction in top 2/4/8 broad-output dimensions [np.float64(0.824), np.float64(0.9207), np.float64(0.9221)]\nCommon selector composed with L0 OV heads: norms [np.float64(4.2877), np.float64(4.0564), np.float64(0.0362), np.float64(0.1418), np.float64(2.0539), np.float64(7.59)]\n\nProjected key-composition scores: strongest earlier heads\ntuple = (earlier head, trace/F, symmetry, norm)\n\nSPACE V0H2-24\nL2H0 [('L0H4', np.float64(0.257), np.float64(0.09), np.float64(0.17)), ('L0H0', np.float64(0.149), np.float64(0.05), np.float64(0.443))]\nL2H1 [('L1H3', np.float64(0.409), np.float64(-0.011), np.float64(0.096)), ('L0H4', np.float64(0.228), np.float64(0.071), np.float64(0.116))]\nL2H2 [('L1H2', np.float64(0.369), np.float64(0.016), np.float64(0.056)), ('L1H3', np.float64(0.366), np.float64(0.044), np.float64(0.065))]\nL2H3 [('L0H0', np.float64(0.335), np.float64(0.133), np.float64(0.451)), ('L0H4', np.float64(0.317), np.float64(0.129), np.float64(0.167))]\nL2H4 [('L0H4', np.float64(0.227), np.float64(0.09), np.float64(0.191)), ('L0H0', np.float64(0.214), np.float64(0.064), np.float64(0.511))]\nL2H5 [('L1H2', np.float64(0.243), np.float64(-0.066), np.float64(0.1)), ('L0H0', np.float64(0.231), np.float64(0.082), np.float64(0.485))]\nL3H0 [('L1H0', np.float64(0.089), np.float64(-0.036), np.float64(0.099)), ('L1H2', np.float64(0.066), np.float64(-0.026), np.float64(0.106))]\nL3H1 [('L0H2', np.float64(2.907), np.float64(0.649), np.float64(0.135)), ('L1H2', np.float64(1.933), np.float64(0.216), np.float64(0.118))]\nL3H2 [('L0H2', np.float64(0.263), np.float64(0.069), np.float64(0.041)), ('L2H0', np.float64(0.212), np.float64(0.09), np.float64(0.08))]\nL3H3 [('L1H3', np.float64(0.212), np.float64(0.049), np.float64(0.064)), ('L1H4', np.float64(0.186), np.float64(0.068), np.float64(0.076))]\nL3H4 [('L1H5', np.float64(0.316), np.float64(0.1), np.float64(0.111)), ('L0H4', np.float64(0.077), np.float64(-0.002), np.float64(0.101))]\nL3H5 [('L2H3', np.float64(0.705), np.float64(0.173), np.float64(0.045)), ('L1H0', np.float64(0.416), np.float64(0.118), np.float64(0.04))]\n\nSPACE MLP0-32\nL2H0 [('L0H4', np.float64(0.143), np.float64(0.025), np.float64(0.225)), ('L0H1', np.float64(0.09), np.float64(0.04), np.float64(0.464))]\nL2H1 [('L1H3', np.float64(0.304), np.float64(0.049), np.float64(0.168)), ('L0H4', np.float64(0.185), np.float64(0.059), np.float64(0.124))]\nL2H2 [('L1H2', np.float64(0.237), np.float64(0.0), np.float64(0.104)), ('L1H4', np.float64(0.134), np.float64(0.041), np.float64(0.119))]\nL2H3 [('L0H1', np.float64(0.14), np.float64(0.103), np.float64(0.382)), ('L0H4', np.float64(0.132), np.float64(0.066), np.float64(0.195))]\nL2H4 [('L0H4', np.float64(0.194), np.float64(0.031), np.float64(0.209)), ('L0H1', np.float64(0.186), np.float64(0.048), np.float64(0.419))]\nL2H5 [('L1H2', np.float64(0.203), np.float64(-0.012), np.float64(0.185)), ('L0H4', np.float64(0.187), np.float64(0.063), np.float64(0.195))]\nL3H0 [('L0H4', np.float64(0.182), np.float64(0.023), np.float64(0.083)), ('L0H0', np.float64(0.176), np.float64(0.03), np.float64(0.188))]\nL3H1 [('L2H1', np.float64(2.027), np.float64(0.274), np.float64(0.241)), ('L0H2', np.float64(1.772), np.float64(0.452), np.float64(0.062))]\nL3H2 [('L2H2', np.float64(0.216), np.float64(0.003), np.float64(0.142)), ('L2H4', np.float64(0.207), np.float64(0.021), np.float64(0.115))]\nL3H3 [('L0H2', np.float64(0.394), np.float64(-0.004), np.float64(0.021)), ('L1H3', np.float64(0.386), np.float64(-0.03), np.float64(0.107))]\nL3H4 [('L1H2', np.float64(0.18), np.float64(-0.005), np.float64(0.13)), ('L2H0', np.float64(0.169), np.float64(0.029), np.float64(0.132))]\nL3H5 [('L2H2', np.float64(0.638), np.float64(0.137), np.float64(0.091)), ('L2H3', np.float64(0.594), np.float64(0.158), np.float64(0.082))]\n\nSPACE MLP0-64\nL2H0 [('L0H4', np.float64(0.183), np.float64(0.022), np.float64(0.393)), ('L0H1', np.float64(0.106), np.float64(0.015), np.float64(0.761))]\nL2H1 [('L0H4', np.float64(0.193), np.float64(0.021), np.float64(0.225)), ('L1H3', np.float64(0.173), np.float64(0.012), np.float64(0.323))]\nL2H2 [('L0H4', np.float64(0.223), np.float64(0.028), np.float64(0.414)), ('L0H2', np.float64(0.173), np.float64(-0.009), np.float64(0.051))]\nL2H3 [('L0H4', np.float64(0.147), np.float64(0.026), np.float64(0.327)), ('L0H1', np.float64(0.125), np.float64(0.036), np.float64(0.602))]\nL2H4 [('L0H4', np.float64(0.266), np.float64(0.015), np.float64(0.376)), ('L0H0', np.float64(0.19), np.float64(-0.014), np.float64(0.86))]\nL2H5 [('L0H4', np.float64(0.188), np.float64(0.02), np.float64(0.359)), ('L0H0', np.float64(0.137), np.float64(-0.007), np.float64(0.818))]\nL3H0 [('L0H4', np.float64(0.113), np.float64(0.005), np.float64(0.149)), ('L0H0', np.float64(0.085), np.float64(0.009), np.float64(0.337))]\nL3H1 [('L0H2', np.float64(2.527), np.float64(0.42), np.float64(0.138)), ('L2H1', np.float64(2.124), np.float64(0.193), np.float64(0.452))]\nL3H2 [('L0H2', np.float64(0.221), np.float64(0.027), np.float64(0.042)), ('L1H0', np.float64(0.135), np.float64(-0.003), np.float64(0.26))]\nL3H3 [('L0H2', np.float64(0.308), np.float64(0.006), np.float64(0.044)), ('L1H3', np.float64(0.103), np.float64(0.002), np.float64(0.2))]\nL3H4 [('L2H5', np.float64(0.144), np.float64(0.009), np.float64(0.271)), ('L1H2', np.float64(0.117), np.float64(-0.008), np.float64(0.256))]\nL3H5 [('L2H3', np.float64(0.568), np.float64(0.094), np.float64(0.153)), ('L1H0', np.float64(0.364), np.float64(0.068), np.float64(0.168))]\n\nSPACE MLP1-64\nL2H0 [('L1H1', np.float64(0.019), np.float64(0.033), np.float64(0.191)), ('L1H0', np.float64(-0.044), np.float64(0.053), np.float64(0.263))]\nL2H1 [('L0H3', np.float64(0.134), np.float64(0.019), np.float64(0.044)), ('L0H4', np.float64(0.038), np.float64(0.02), np.float64(0.127))]\nL2H2 [('L1H4', np.float64(0.405), np.float64(0.086), np.float64(0.116)), ('L1H0', np.float64(0.266), np.float64(0.053), np.float64(0.212))]\nL2H3 [('L0H1', np.float64(0.129), np.float64(0.015), np.float64(0.224)), ('L1H0', np.float64(0.127), np.float64(-0.015), np.float64(0.163))]\nL2H4 [('L0H1', np.float64(0.221), np.float64(0.011), np.float64(0.262)), ('L0H0', np.float64(0.189), np.float64(0.012), np.float64(0.322))]\nL2H5 [('L1H4', np.float64(0.118), np.float64(0.049), np.float64(0.187)), ('L0H4', np.float64(0.06), np.float64(-0.0), np.float64(0.133))]\nL3H0 [('L0H4', np.float64(0.105), np.float64(0.002), np.float64(0.094)), ('L0H5', np.float64(0.052), np.float64(0.008), np.float64(0.246))]\nL3H1 [('L2H1', np.float64(2.796), np.float64(0.367), np.float64(0.452)), ('L2H3', np.float64(1.692), np.float64(0.103), np.float64(0.506))]\nL3H2 [('L2H0', np.float64(0.203), np.float64(-0.01), np.float64(0.218)), ('L1H0', np.float64(0.183), np.float64(0.023), np.float64(0.112))]\nL3H3 [('L1H0', np.float64(0.306), np.float64(-0.014), np.float64(0.103)), ('L1H1', np.float64(0.144), np.float64(0.003), np.float64(0.097))]\nL3H4 [('L0H4', np.float64(0.326), np.float64(0.081), np.float64(0.122)), ('L0H1', np.float64(0.306), np.float64(0.07), np.float64(0.216))]\nL3H5 [('L2H3', np.float64(0.697), np.float64(0.044), np.float64(0.217)), ('L2H4', np.float64(0.642), np.float64(0.047), np.float64(0.143))]\n\nDirect-match versus predecessor-match on V0H2's 24-dimensional read space\nL1H0 [[-1.058 -0.145 0.125]\n [-0.266 0.022 0.033]\n [-1.401 0.322 0.063]]\nL1H1 [[-1.489 0.037 0.215]\n [-0.025 0.224 0.074]\n [-0.682 0.264 0.096]]\nL1H2 [[-2.134 0.16 0.226]\n [ 0.636 0.078 0.08 ]\n [ 0.585 0.025 0.111]]\nL1H3 [[-1.368 -0.039 0.186]\n [ 0.674 0.128 0.065]\n [ 0.574 0.115 0.119]]\nL1H4 [[-1.568 0.097 0.221]\n [ 0.688 0.089 0.065]\n [-0.278 0.076 0.142]]\nL1H5 [[-1.199 -0.049 0.201]\n [ 0.201 0.022 0.086]\n [-0.681 0.047 0.12 ]]\nL2H0 [[-0.425 -0.178 0.207]\n [-0.451 -0.014 0.097]\n [ 0.019 0.11 0.089]]\nL2H1 [[-0.374 0.088 0.157]\n [-0.5 0.118 0.071]\n [ 1.192 0.104 0.095]]\nL2H2 [[-0.456 -0.18 0.149]\n [ 0.121 0.031 0.049]\n [-0.371 0.072 0.057]]\nL2H3 [[-0.789 -0.266 0.175]\n [ 0.266 -0.009 0.068]\n [ 0.417 0.13 0.073]]\nL2H4 [[-0.459 -0.172 0.209]\n [-0.578 -0.019 0.08 ]\n [-0.156 0.032 0.086]]\nL2H5 [[-0.497 -0.288 0.202]\n [-0.019 0.013 0.081]\n [ 0.287 -0.021 0.084]]\nL3H0 [[ 1.23 0.246 0.157]\n [-0.128 0.014 0.079]\n [ 1.039 0.06 0.081]]\nL3H1 [[ 1.236 0.271 0.139]\n [ 2.907 0.649 0.135]\n [-1.302 0.1 0.072]]\nL3H2 [[ 0.162 -0.208 0.116]\n [ 0.263 0.069 0.041]\n [ 0.104 0.099 0.043]]\nL3H3 [[ 0.015 -0.233 0.115]\n [-0.016 -0.083 0.043]\n [ 0.355 0.042 0.049]]\nL3H4 [[ 0.341 -0.057 0.12 ]\n [-0.296 0.166 0.056]\n [ 0.142 -0.052 0.052]]\nL3H5 [[0.32 0.055 0.047]\n [0.278 0.061 0.026]\n [0.123 0.06 0.02 ]]\n\nMLP neuron weight-norm quantiles\n0 read [0.37 0.389 0.423 0.967 2.255] write [0.172 0.194 0.237 1.272 1.99 ] small-write fraction 0.693\n1 read [1.829 1.906 2.004 2.105 2.213] write [1.78 1.847 1.922 2.007 2.085] small-write fraction 0.004\n2 read [1.928 1.971 2.01 2.055 2.095] write [1.82 1.888 1.957 2.025 2.091] small-write fraction 0.0\n3 read [1.907 1.938 1.976 2.014 2.053] write [1.809 1.921 2.062 2.222 2.396] small-write fraction 0.0\n",
"seconds": 0.718,
"rc": 0,
"timed_out": false,
"truncated": false,
"code_sha256": "cf4ddf7b461ba0132d9bf12bc2417d0f973de8e42164ce20562697034b98ce01",
"weights_sha256": "130135d2b27829df2734d78fa3af27b00b1cc1bf34c41b01a09b350e6c09a7de"
}Common L1 key-selector direction: each head's cosine and bias/QK norm ratio
0 [[-0.118 0.25 ]
[-0.403 0.113]
[ 0.149 0.035]
[-0.039 0.136]
[-0.141 0.106]
[ 0.387 0.21 ]]
1 [[0.98 1.531]
[0.987 1.738]
[0.989 1.671]
[0.985 1.711]
[0.984 1.58 ]
[0.986 1.312]]
2 [[0.806 0.697]
[0.781 0.36 ]
[0.786 0.468]
[0.814 0.701]
[0.804 0.673]
[0.814 0.67 ]]
3 [[ 0.318 0.116]
[-0.129 0.132]
[ 0.621 0.192]
[ 0.652 0.225]
[ 0.602 0.166]
[ 0.08 0.197]]
Common selector fraction in top 2/4/8 broad-output dimensions [np.float64(0.824), np.float64(0.9207), np.float64(0.9221)]
Common selector composed with L0 OV heads: norms [np.float64(4.2877), np.float64(4.0564), np.float64(0.0362), np.float64(0.1418), np.float64(2.0539), np.float64(7.59)]
Projected key-composition scores: strongest earlier heads
tuple = (earlier head, trace/F, symmetry, norm)
SPACE V0H2-24
L2H0 [('L0H4', np.float64(0.257), np.float64(0.09), np.float64(0.17)), ('L0H0', np.float64(0.149), np.float64(0.05), np.float64(0.443))]
L2H1 [('L1H3', np.float64(0.409), np.float64(-0.011), np.float64(0.096)), ('L0H4', np.float64(0.228), np.float64(0.071), np.float64(0.116))]
L2H2 [('L1H2', np.float64(0.369), np.float64(0.016), np.float64(0.056)), ('L1H3', np.float64(0.366), np.float64(0.044), np.float64(0.065))]
L2H3 [('L0H0', np.float64(0.335), np.float64(0.133), np.float64(0.451)), ('L0H4', np.float64(0.317), np.float64(0.129), np.float64(0.167))]
L2H4 [('L0H4', np.float64(0.227), np.float64(0.09), np.float64(0.191)), ('L0H0', np.float64(0.214), np.float64(0.064), np.float64(0.511))]
L2H5 [('L1H2', np.float64(0.243), np.float64(-0.066), np.float64(0.1)), ('L0H0', np.float64(0.231), np.float64(0.082), np.float64(0.485))]
L3H0 [('L1H0', np.float64(0.089), np.float64(-0.036), np.float64(0.099)), ('L1H2', np.float64(0.066), np.float64(-0.026), np.float64(0.106))]
L3H1 [('L0H2', np.float64(2.907), np.float64(0.649), np.float64(0.135)), ('L1H2', np.float64(1.933), np.float64(0.216), np.float64(0.118))]
L3H2 [('L0H2', np.float64(0.263), np.float64(0.069), np.float64(0.041)), ('L2H0', np.float64(0.212), np.float64(0.09), np.float64(0.08))]
L3H3 [('L1H3', np.float64(0.212), np.float64(0.049), np.float64(0.064)), ('L1H4', np.float64(0.186), np.float64(0.068), np.float64(0.076))]
L3H4 [('L1H5', np.float64(0.316), np.float64(0.1), np.float64(0.111)), ('L0H4', np.float64(0.077), np.float64(-0.002), np.float64(0.101))]
L3H5 [('L2H3', np.float64(0.705), np.float64(0.173), np.float64(0.045)), ('L1H0', np.float64(0.416), np.float64(0.118), np.float64(0.04))]
SPACE MLP0-32
L2H0 [('L0H4', np.float64(0.143), np.float64(0.025), np.float64(0.225)), ('L0H1', np.float64(0.09), np.float64(0.04), np.float64(0.464))]
L2H1 [('L1H3', np.float64(0.304), np.float64(0.049), np.float64(0.168)), ('L0H4', np.float64(0.185), np.float64(0.059), np.float64(0.124))]
L2H2 [('L1H2', np.float64(0.237), np.float64(0.0), np.float64(0.104)), ('L1H4', np.float64(0.134), np.float64(0.041), np.float64(0.119))]
L2H3 [('L0H1', np.float64(0.14), np.float64(0.103), np.float64(0.382)), ('L0H4', np.float64(0.132), np.float64(0.066), np.float64(0.195))]
L2H4 [('L0H4', np.float64(0.194), np.float64(0.031), np.float64(0.209)), ('L0H1', np.float64(0.186), np.float64(0.048), np.float64(0.419))]
L2H5 [('L1H2', np.float64(0.203), np.float64(-0.012), np.float64(0.185)), ('L0H4', np.float64(0.187), np.float64(0.063), np.float64(0.195))]
L3H0 [('L0H4', np.float64(0.182), np.float64(0.023), np.float64(0.083)), ('L0H0', np.float64(0.176), np.float64(0.03), np.float64(0.188))]
L3H1 [('L2H1', np.float64(2.027), np.float64(0.274), np.float64(0.241)), ('L0H2', np.float64(1.772), np.float64(0.452), np.float64(0.062))]
L3H2 [('L2H2', np.float64(0.216), np.float64(0.003), np.float64(0.142)), ('L2H4', np.float64(0.207), np.float64(0.021), np.float64(0.115))]
L3H3 [('L0H2', np.float64(0.394), np.float64(-0.004), np.float64(0.021)), ('L1H3', np.float64(0.386), np.float64(-0.03), np.float64(0.107))]
L3H4 [('L1H2', np.float64(0.18), np.float64(-0.005), np.float64(0.13)), ('L2H0', np.float64(0.169), np.float64(0.029), np.float64(0.132))]
L3H5 [('L2H2', np.float64(0.638), np.float64(0.137), np.float64(0.091)), ('L2H3', np.float64(0.594), np.float64(0.158), np.float64(0.082))]
SPACE MLP0-64
L2H0 [('L0H4', np.float64(0.183), np.float64(0.022), np.float64(0.393)), ('L0H1', np.float64(0.106), np.float64(0.015), np.float64(0.761))]
L2H1 [('L0H4', np.float64(0.193), np.float64(0.021), np.float64(0.225)), ('L1H3', np.float64(0.173), np.float64(0.012), np.float64(0.323))]
L2H2 [('L0H4', np.float64(0.223), np.float64(0.028), np.float64(0.414)), ('L0H2', np.float64(0.173), np.float64(-0.009), np.float64(0.051))]
L2H3 [('L0H4', np.float64(0.147), np.float64(0.026), np.float64(0.327)), ('L0H1', np.float64(0.125), np.float64(0.036), np.float64(0.602))]
L2H4 [('L0H4', np.float64(0.266), np.float64(0.015), np.float64(0.376)), ('L0H0', np.float64(0.19), np.float64(-0.014), np.float64(0.86))]
L2H5 [('L0H4', np.float64(0.188), np.float64(0.02), np.float64(0.359)), ('L0H0', np.float64(0.137), np.float64(-0.007), np.float64(0.818))]
L3H0 [('L0H4', np.float64(0.113), np.float64(0.005), np.float64(0.149)), ('L0H0', np.float64(0.085), np.float64(0.009), np.float64(0.337))]
L3H1 [('L0H2', np.float64(2.527), np.float64(0.42), np.float64(0.138)), ('L2H1', np.float64(2.124), np.float64(0.193), np.float64(0.452))]
L3H2 [('L0H2', np.float64(0.221), np.float64(0.027), np.float64(0.042)), ('L1H0', np.float64(0.135), np.float64(-0.003), np.float64(0.26))]
L3H3 [('L0H2', np.float64(0.308), np.float64(0.006), np.float64(0.044)), ('L1H3', np.float64(0.103), np.float64(0.002), np.float64(0.2))]
L3H4 [('L2H5', np.float64(0.144), np.float64(0.009), np.float64(0.271)), ('L1H2', np.float64(0.117), np.float64(-0.008), np.float64(0.256))]
L3H5 [('L2H3', np.float64(0.568), np.float64(0.094), np.float64(0.153)), ('L1H0', np.float64(0.364), np.float64(0.068), np.float64(0.168))]
SPACE MLP1-64
L2H0 [('L1H1', np.float64(0.019), np.float64(0.033), np.float64(0.191)), ('L1H0', np.float64(-0.044), np.float64(0.053), np.float64(0.263))]
L2H1 [('L0H3', np.float64(0.134), np.float64(0.019), np.float64(0.044)), ('L0H4', np.float64(0.038), np.float64(0.02), np.float64(0.127))]
L2H2 [('L1H4', np.float64(0.405), np.float64(0.086), np.float64(0.116)), ('L1H0', np.float64(0.266), np.float64(0.053), np.float64(0.212))]
L2H3 [('L0H1', np.float64(0.129), np.float64(0.015), np.float64(0.224)), ('L1H0', np.float64(0.127), np.float64(-0.015), np.float64(0.163))]
L2H4 [('L0H1', np.float64(0.221), np.float64(0.011), np.float64(0.262)), ('L0H0', np.float64(0.189), np.float64(0.012), np.float64(0.322))]
L2H5 [('L1H4', np.float64(0.118), np.float64(0.049), np.float64(0.187)), ('L0H4', np.float64(0.06), np.float64(-0.0), np.float64(0.133))]
L3H0 [('L0H4', np.float64(0.105), np.float64(0.002), np.float64(0.094)), ('L0H5', np.float64(0.052), np.float64(0.008), np.float64(0.246))]
L3H1 [('L2H1', np.float64(2.796), np.float64(0.367), np.float64(0.452)), ('L2H3', np.float64(1.692), np.float64(0.103), np.float64(0.506))]
L3H2 [('L2H0', np.float64(0.203), np.float64(-0.01), np.float64(0.218)), ('L1H0', np.float64(0.183), np.float64(0.023), np.float64(0.112))]
L3H3 [('L1H0', np.float64(0.306), np.float64(-0.014), np.float64(0.103)), ('L1H1', np.float64(0.144), np.float64(0.003), np.float64(0.097))]
L3H4 [('L0H4', np.float64(0.326), np.float64(0.081), np.float64(0.122)), ('L0H1', np.float64(0.306), np.float64(0.07), np.float64(0.216))]
L3H5 [('L2H3', np.float64(0.697), np.float64(0.044), np.float64(0.217)), ('L2H4', np.float64(0.642), np.float64(0.047), np.float64(0.143))]
Direct-match versus predecessor-match on V0H2's 24-dimensional read space
L1H0 [[-1.058 -0.145 0.125]
[-0.266 0.022 0.033]
[-1.401 0.322 0.063]]
L1H1 [[-1.489 0.037 0.215]
[-0.025 0.224 0.074]
[-0.682 0.264 0.096]]
L1H2 [[-2.134 0.16 0.226]
[ 0.636 0.078 0.08 ]
[ 0.585 0.025 0.111]]
L1H3 [[-1.368 -0.039 0.186]
[ 0.674 0.128 0.065]
[ 0.574 0.115 0.119]]
L1H4 [[-1.568 0.097 0.221]
[ 0.688 0.089 0.065]
[-0.278 0.076 0.142]]
L1H5 [[-1.199 -0.049 0.201]
[ 0.201 0.022 0.086]
[-0.681 0.047 0.12 ]]
L2H0 [[-0.425 -0.178 0.207]
[-0.451 -0.014 0.097]
[ 0.019 0.11 0.089]]
L2H1 [[-0.374 0.088 0.157]
[-0.5 0.118 0.071]
[ 1.192 0.104 0.095]]
L2H2 [[-0.456 -0.18 0.149]
[ 0.121 0.031 0.049]
[-0.371 0.072 0.057]]
L2H3 [[-0.789 -0.266 0.175]
[ 0.266 -0.009 0.068]
[ 0.417 0.13 0.073]]
L2H4 [[-0.459 -0.172 0.209]
[-0.578 -0.019 0.08 ]
[-0.156 0.032 0.086]]
L2H5 [[-0.497 -0.288 0.202]
[-0.019 0.013 0.081]
[ 0.287 -0.021 0.084]]
L3H0 [[ 1.23 0.246 0.157]
[-0.128 0.014 0.079]
[ 1.039 0.06 0.081]]
L3H1 [[ 1.236 0.271 0.139]
[ 2.907 0.649 0.135]
[-1.302 0.1 0.072]]
L3H2 [[ 0.162 -0.208 0.116]
[ 0.263 0.069 0.041]
[ 0.104 0.099 0.043]]
L3H3 [[ 0.015 -0.233 0.115]
[-0.016 -0.083 0.043]
[ 0.355 0.042 0.049]]
L3H4 [[ 0.341 -0.057 0.12 ]
[-0.296 0.166 0.056]
[ 0.142 -0.052 0.052]]
L3H5 [[0.32 0.055 0.047]
[0.278 0.061 0.026]
[0.123 0.06 0.02 ]]
MLP neuron weight-norm quantiles
0 read [0.37 0.389 0.423 0.967 2.255] write [0.172 0.194 0.237 1.272 1.99 ] small-write fraction 0.693
1 read [1.829 1.906 2.004 2.105 2.213] write [1.78 1.847 1.922 2.007 2.085] small-write fraction 0.004
2 read [1.928 1.971 2.01 2.055 2.095] write [1.82 1.888 1.957 2.025 2.091] small-write fraction 0.0
3 read [1.907 1.938 1.976 2.014 2.053] write [1.809 1.921 2.062 2.222 2.396] small-write fraction 0.0
Exact prompt · Raw CLI events · Settings and audit
OUTPUT of run 6 of at most 100:
```
Common L1 key-selector direction: each head's cosine and bias/QK norm ratio
0 [[-0.118 0.25 ]
[-0.403 0.113]
[ 0.149 0.035]
[-0.039 0.136]
[-0.141 0.106]
[ 0.387 0.21 ]]
1 [[0.98 1.531]
[0.987 1.738]
[0.989 1.671]
[0.985 1.711]
[0.984 1.58 ]
[0.986 1.312]]
2 [[0.806 0.697]
[0.781 0.36 ]
[0.786 0.468]
[0.814 0.701]
[0.804 0.673]
[0.814 0.67 ]]
3 [[ 0.318 0.116]
[-0.129 0.132]
[ 0.621 0.192]
[ 0.652 0.225]
[ 0.602 0.166]
[ 0.08 0.197]]
Common selector fraction in top 2/4/8 broad-output dimensions [np.float64(0.824), np.float64(0.9207), np.float64(0.9221)]
Common selector composed with L0 OV heads: norms [np.float64(4.2877), np.float64(4.0564), np.float64(0.0362), np.float64(0.1418), np.float64(2.0539), np.float64(7.59)]
Projected key-composition scores: strongest earlier heads
tuple = (earlier head, trace/F, symmetry, norm)
SPACE V0H2-24
L2H0 [('L0H4', np.float64(0.257), np.float64(0.09), np.float64(0.17)), ('L0H0', np.float64(0.149), np.float64(0.05), np.float64(0.443))]
L2H1 [('L1H3', np.float64(0.409), np.float64(-0.011), np.float64(0.096)), ('L0H4', np.float64(0.228), np.float64(0.071), np.float64(0.116))]
L2H2 [('L1H2', np.float64(0.369), np.float64(0.016), np.float64(0.056)), ('L1H3', np.float64(0.366), np.float64(0.044), np.float64(0.065))]
L2H3 [('L0H0', np.float64(0.335), np.float64(0.133), np.float64(0.451)), ('L0H4', np.float64(0.317), np.float64(0.129), np.float64(0.167))]
L2H4 [('L0H4', np.float64(0.227), np.float64(0.09), np.float64(0.191)), ('L0H0', np.float64(0.214), np.float64(0.064), np.float64(0.511))]
L2H5 [('L1H2', np.float64(0.243), np.float64(-0.066), np.float64(0.1)), ('L0H0', np.float64(0.231), np.float64(0.082), np.float64(0.485))]
L3H0 [('L1H0', np.float64(0.089), np.float64(-0.036), np.float64(0.099)), ('L1H2', np.float64(0.066), np.float64(-0.026), np.float64(0.106))]
L3H1 [('L0H2', np.float64(2.907), np.float64(0.649), np.float64(0.135)), ('L1H2', np.float64(1.933), np.float64(0.216), np.float64(0.118))]
L3H2 [('L0H2', np.float64(0.263), np.float64(0.069), np.float64(0.041)), ('L2H0', np.float64(0.212), np.float64(0.09), np.float64(0.08))]
L3H3 [('L1H3', np.float64(0.212), np.float64(0.049), np.float64(0.064)), ('L1H4', np.float64(0.186), np.float64(0.068), np.float64(0.076))]
L3H4 [('L1H5', np.float64(0.316), np.float64(0.1), np.float64(0.111)), ('L0H4', np.float64(0.077), np.float64(-0.002), np.float64(0.101))]
L3H5 [('L2H3', np.float64(0.705), np.float64(0.173), np.float64(0.045)), ('L1H0', np.float64(0.416), np.float64(0.118), np.float64(0.04))]
SPACE MLP0-32
L2H0 [('L0H4', np.float64(0.143), np.float64(0.025), np.float64(0.225)), ('L0H1', np.float64(0.09), np.float64(0.04), np.float64(0.464))]
L2H1 [('L1H3', np.float64(0.304), np.float64(0.049), np.float64(0.168)), ('L0H4', np.float64(0.185), np.float64(0.059), np.float64(0.124))]
L2H2 [('L1H2', np.float64(0.237), np.float64(0.0), np.float64(0.104)), ('L1H4', np.float64(0.134), np.float64(0.041), np.float64(0.119))]
L2H3 [('L0H1', np.float64(0.14), np.float64(0.103), np.float64(0.382)), ('L0H4', np.float64(0.132), np.float64(0.066), np.float64(0.195))]
L2H4 [('L0H4', np.float64(0.194), np.float64(0.031), np.float64(0.209)), ('L0H1', np.float64(0.186), np.float64(0.048), np.float64(0.419))]
L2H5 [('L1H2', np.float64(0.203), np.float64(-0.012), np.float64(0.185)), ('L0H4', np.float64(0.187), np.float64(0.063), np.float64(0.195))]
L3H0 [('L0H4', np.float64(0.182), np.float64(0.023), np.float64(0.083)), ('L0H0', np.float64(0.176), np.float64(0.03), np.float64(0.188))]
L3H1 [('L2H1', np.float64(2.027), np.float64(0.274), np.float64(0.241)), ('L0H2', np.float64(1.772), np.float64(0.452), np.float64(0.062))]
L3H2 [('L2H2', np.float64(0.216), np.float64(0.003), np.float64(0.142)), ('L2H4', np.float64(0.207), np.float64(0.021), np.float64(0.115))]
L3H3 [('L0H2', np.float64(0.394), np.float64(-0.004), np.float64(0.021)), ('L1H3', np.float64(0.386), np.float64(-0.03), np.float64(0.107))]
L3H4 [('L1H2', np.float64(0.18), np.float64(-0.005), np.float64(0.13)), ('L2H0', np.float64(0.169), np.float64(0.029), np.float64(0.132))]
L3H5 [('L2H2', np.float64(0.638), np.float64(0.137), np.float64(0.091)), ('L2H3', np.float64(0.594), np.float64(0.158), np.float64(0.082))]
SPACE MLP0-64
L2H0 [('L0H4', np.float64(0.183), np.float64(0.022), np.float64(0.393)), ('L0H1', np.float64(0.106), np.float64(0.015), np.float64(0.761))]
L2H1 [('L0H4', np.float64(0.193), np.float64(0.021), np.float64(0.225)), ('L1H3', np.float64(0.173), np.float64(0.012), np.float64(0.323))]
L2H2 [('L0H4', np.float64(0.223), np.float64(0.028), np.float64(0.414)), ('L0H2', np.float64(0.173), np.float64(-0.009), np.float64(0.051))]
L2H3 [('L0H4', np.float64(0.147), np.float64(0.026), np.float64(0.327)), ('L0H1', np.float64(0.125), np.float64(0.036), np.float64(0.602))]
L2H4 [('L0H4', np.float64(0.266), np.float64(0.015), np.float64(0.376)), ('L0H0', np.float64(0.19), np.float64(-0.014), np.float64(0.86))]
L2H5 [('L0H4', np.float64(0.188), np.float64(0.02), np.float64(0.359)), ('L0H0', np.float64(0.137), np.float64(-0.007), np.float64(0.818))]
L3H0 [('L0H4', np.float64(0.113), np.float64(0.005), np.float64(0.149)), ('L0H0', np.float64(0.085), np.float64(0.009), np.float64(0.337))]
L3H1 [('L0H2', np.float64(2.527), np.float64(0.42), np.float64(0.138)), ('L2H1', np.float64(2.124), np.float64(0.193), np.float64(0.452))]
L3H2 [('L0H2', np.float64(0.221), np.float64(0.027), np.float64(0.042)), ('L1H0', np.float64(0.135), np.float64(-0.003), np.float64(0.26))]
L3H3 [('L0H2', np.float64(0.308), np.float64(0.006), np.float64(0.044)), ('L1H3', np.float64(0.103), np.float64(0.002), np.float64(0.2))]
L3H4 [('L2H5', np.float64(0.144), np.float64(0.009), np.float64(0.271)), ('L1H2', np.float64(0.117), np.float64(-0.008), np.float64(0.256))]
L3H5 [('L2H3', np.float64(0.568), np.float64(0.094), np.float64(0.153)), ('L1H0', np.float64(0.364), np.float64(0.068), np.float64(0.168))]
SPACE MLP1-64
L2H0 [('L1H1', np.float64(0.019), np.float64(0.033), np.float64(0.191)), ('L1H0', np.float64(-0.044), np.float64(0.053), np.float64(0.263))]
L2H1 [('L0H3', np.float64(0.134), np.float64(0.019), np.float64(0.044)), ('L0H4', np.float64(0.038), np.float64(0.02), np.float64(0.127))]
L2H2 [('L1H4', np.float64(0.405), np.float64(0.086), np.float64(0.116)), ('L1H0', np.float64(0.266), np.float64(0.053), np.float64(0.212))]
L2H3 [('L0H1', np.float64(0.129), np.float64(0.015), np.float64(0.224)), ('L1H0', np.float64(0.127), np.float64(-0.015), np.float64(0.163))]
L2H4 [('L0H1', np.float64(0.221), np.float64(0.011), np.float64(0.262)), ('L0H0', np.float64(0.189), np.float64(0.012), np.float64(0.322))]
L2H5 [('L1H4', np.float64(0.118), np.float64(0.049), np.float64(0.187)), ('L0H4', np.float64(0.06), np.float64(-0.0), np.float64(0.133))]
L3H0 [('L0H4', np.float64(0.105), np.float64(0.002), np.float64(0.094)), ('L0H5', np.float64(0.052), np.float64(0.008), np.float64(0.246))]
L3H1 [('L2H1', np.float64(2.796), np.float64(0.367), np.float64(0.452)), ('L2H3', np.float64(1.692), np.float64(0.103), np.float64(0.506))]
L3H2 [('L2H0', np.float64(0.203), np.float64(-0.01), np.float64(0.218)), ('L1H0', np.float64(0.183), np.float64(0.023), np.float64(0.112))]
L3H3 [('L1H0', np.float64(0.306), np.float64(-0.014), np.float64(0.103)), ('L1H1', np.float64(0.144), np.float64(0.003), np.float64(0.097))]
L3H4 [('L0H4', np.float64(0.326), np.float64(0.081), np.float64(0.122)), ('L0H1', np.float64(0.306), np.float64(0.07), np.float64(0.216))]
L3H5 [('L2H3', np.float64(0.697), np.float64(0.044), np.float64(0.217)), ('L2H4', np.float64(0.642), np.float64(0.047), np.float64(0.143))]
Direct-match versus predecessor-match on V0H2's 24-dimensional read space
L1H0 [[-1.058 -0.145 0.125]
[-0.266 0.022 0.033]
[-1.401 0.322 0.063]]
L1H1 [[-1.489 0.037 0.215]
[-0.025 0.224 0.074]
[-0.682 0.264 0.096]]
L1H2 [[-2.134 0.16 0.226]
[ 0.636 0.078 0.08 ]
[ 0.585 0.025 0.111]]
L1H3 [[-1.368 -0.039 0.186]
[ 0.674 0.128 0.065]
[ 0.574 0.115 0.119]]
L1H4 [[-1.568 0.097 0.221]
[ 0.688 0.089 0.065]
[-0.278 0.076 0.142]]
L1H5 [[-1.199 -0.049 0.201]
[ 0.201 0.022 0.086]
[-0.681 0.047 0.12 ]]
L2H0 [[-0.425 -0.178 0.207]
[-0.451 -0.014 0.097]
[ 0.019 0.11 0.089]]
L2H1 [[-0.374 0.088 0.157]
[-0.5 0.118 0.071]
[ 1.192 0.104 0.095]]
L2H2 [[-0.456 -0.18 0.149]
[ 0.121 0.031 0.049]
[-0.371 0.072 0.057]]
L2H3 [[-0.789 -0.266 0.175]
[ 0.266 -0.009 0.068]
[ 0.417 0.13 0.073]]
L2H4 [[-0.459 -0.172 0.209]
[-0.578 -0.019 0.08 ]
[-0.156 0.032 0.086]]
L2H5 [[-0.497 -0.288 0.202]
[-0.019 0.013 0.081]
[ 0.287 -0.021 0.084]]
L3H0 [[ 1.23 0.246 0.157]
[-0.128 0.014 0.079]
[ 1.039 0.06 0.081]]
L3H1 [[ 1.236 0.271 0.139]
[ 2.907 0.649 0.135]
[-1.302 0.1 0.072]]
L3H2 [[ 0.162 -0.208 0.116]
[ 0.263 0.069 0.041]
[ 0.104 0.099 0.043]]
L3H3 [[ 0.015 -0.233 0.115]
[-0.016 -0.083 0.043]
[ 0.355 0.042 0.049]]
L3H4 [[ 0.341 -0.057 0.12 ]
[-0.296 0.166 0.056]
[ 0.142 -0.052 0.052]]
L3H5 [[0.32 0.055 0.047]
[0.278 0.061 0.026]
[0.123 0.06 0.02 ]]
MLP neuron weight-norm quantiles
0 read [0.37 0.389 0.423 0.967 2.255] write [0.172 0.194 0.237 1.272 1.99 ] small-write fraction 0.693
1 read [1.829 1.906 2.004 2.105 2.213] write [1.78 1.847 1.922 2.007 2.085] small-write fraction 0.004
2 read [1.928 1.971 2.01 2.055 2.095] write [1.82 1.888 1.957 2.025 2.091] small-write fraction 0.0
3 read [1.907 1.938 1.976 2.014 2.053] write [1.809 1.921 2.062 2.222 2.396] small-write fraction 0.0
```Failed to read global AGENTS.md instructions from `<local-home>/.codex/AGENTS.md`: Operation not permitted (os error 1)
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
## Observations
**The strongest evidence concerns ordered context use, not corpus identity.** These are weight-based hypotheses; no attention patterns or predictions were evaluated.
Let \(A_{lh}\) denote the head’s QK matrix product, including the LayerNorm gain and centering matrices, and \(B_{lh}\) its corresponding OV product.
- **L0H2 is exceptional.** Its slices of `W_q.0` and `W_k.0` have Frobenius norms **30.818 and 28.300**; every other L0 head is below **7.25**. Its effective QK product has paired complex eigenvalues, including **0.632 ± 0.813i** and **−0.231 ± 0.927i**. This suggests an oscillatory positional mechanism, plausibly a one-character shift.
- **L0H0/H1/H4/H5 share a positional-summary pathway.** Their OV-product cosine similarities are **0.724–0.903**, and **80–88%** of their OV input energy lies in the candidate positional subspace extracted from L0H2.
- **L1 has a common selection mechanism.** Its six bias-derived key selectors have pairwise cosines **0.945–0.976**. Their common direction has **92.1%** of its energy in the leading four output directions of those L0 summary heads. This suggests shared positional selection, plausibly recency.
- **L3H1 has a distinctive induction signature.** On L0H2’s leading 24 value-read directions, \(A_{31}B_{02}\) has normalized trace **2.907**, transpose symmetry **0.649**, and **95.0% positive symmetric eigenvalue mass**. The other final-layer heads lack a comparable predecessor-matching signature.
- **Corpus evidence is weak.** `W_proj.3` retains substantial singular values through dimensions 23–28: **10.367, 8.351, 8.064, 7.914, 7.461, 7.166**. There is no compelling small-alphabet cutoff, but these dimensions cannot simply be identified with characters.
## Heads
“Local” assignments below are tentative: the shared positional selector does not establish an exact attention distance.
L0H0: **broad** — Low-rank positional pooling candidate; effective QK norm 0.0546, OV positional-input fraction 0.858.
L0H1: **broad** — Shares the pooling pathway; QK norm 0.0295, OV positional-input fraction 0.881.
L0H2: **previous** — Strong, oscillatory, approximately shared query/key subspace; clearest shift candidate.
L0H3: **local** — Tentative positional-locality head; QK transpose symmetry 0.478 and substantial overlap with L0H2’s positional subspace.
L0H4: **broad** — Low-rank pooling pathway; OV cosine 0.903 with L0H0 and positional-input fraction 0.799.
L0H5: **broad** — Same positional-summary family; positional-input fraction 0.851 and dominant OV singular value 8.328.
L1H0: **local** — Common positional selector; bias/QK norm ratio 1.531.
L1H1: **local** — Same mechanism, with ratio 1.738; 53.7% of QK key-side energy reads the leading two summary-output directions.
L1H2: **local** — Common selector, ratio 1.671; no comparably strong induction signature.
L1H3: **local** — Common selector, ratio 1.711; strong dependence on the positional-summary pathway.
L1H4: **local** — Strongest two-dimensional summary-key overlap in L1, 60.3%; selector ratio 1.580.
L1H5: **local** — Common selector, ratio 1.312; summary-key overlap 54.9%.
L2H0: **local** — Selector cosine 0.806 with L1’s common direction; substantial positional-selection component.
L2H1: **unclear** — Weaker positional bias; strong downstream composition with L3H1 suggests a supporting context-retrieval role, but not a specific distance.
L2H2: **local** — Positional selector persists; leading two summary directions account for 17.7% of key-side energy.
L2H3: **local** — Selector cosine 0.814 and bias/QK ratio 0.701.
L2H4: **local** — Selector cosine 0.804 and ratio 0.673; no strong predecessor-matching signature.
L2H5: **local** — Selector cosine 0.814 and ratio 0.670.
L3H0: **unclear** — Positive direct matching, but not predecessor matching; this does not establish induction or attention distance.
L3H1: **copy** — Clear positive, relatively symmetric composition with L0H2’s OV pathway; strongest induction candidate.
L3H2: **local** — Tentative: retains the common positional selector, cosine 0.621, without strong induction composition.
L3H3: **local** — Tentative: selector cosine 0.652; predecessor matching is negligible.
L3H4: **local** — Tentative: selector cosine 0.602; no convincing induction signature.
L3H5: **unclear** — Strong OV and relatively weak QK could support broad aggregation, but do not establish its attention span.
## Reasoning
A likely predecessor channel feeding a later induction-like matcher favors **as-is** training: ordered repetitions provide a reason to match an earlier predecessor and retrieve its successor. Line shuffling would substantially weaken that benefit, though it would not eliminate all sequential structure.
**English is my narrow corpus preference**, followed by Python. The architecture suggests ordinary local character prediction plus reuse of earlier text, without a clearly identifiable indentation-specific mechanism or restricted-alphabet signature. Neither absence is decisive: the missing character interfaces prevent assigning these features to letters, spaces, or newlines. Corpus probabilities therefore remain deliberately modest.
corpus: **english**
order: **as is**
probabilities: python=0.36 romaji=0.20 english=0.44; as is=0.94 shuffled=0.06## Observations
**The strongest evidence concerns ordered context use, not corpus identity.** These are weight-based hypotheses; no attention patterns or predictions were evaluated.
Let \(A_{lh}\) denote the head’s QK matrix product, including the LayerNorm gain and centering matrices, and \(B_{lh}\) its corresponding OV product.
- **L0H2 is exceptional.** Its slices of `W_q.0` and `W_k.0` have Frobenius norms **30.818 and 28.300**; every other L0 head is below **7.25**. Its effective QK product has paired complex eigenvalues, including **0.632 ± 0.813i** and **−0.231 ± 0.927i**. This suggests an oscillatory positional mechanism, plausibly a one-character shift.
- **L0H0/H1/H4/H5 share a positional-summary pathway.** Their OV-product cosine similarities are **0.724–0.903**, and **80–88%** of their OV input energy lies in the candidate positional subspace extracted from L0H2.
- **L1 has a common selection mechanism.** Its six bias-derived key selectors have pairwise cosines **0.945–0.976**. Their common direction has **92.1%** of its energy in the leading four output directions of those L0 summary heads. This suggests shared positional selection, plausibly recency.
- **L3H1 has a distinctive induction signature.** On L0H2’s leading 24 value-read directions, \(A_{31}B_{02}\) has normalized trace **2.907**, transpose symmetry **0.649**, and **95.0% positive symmetric eigenvalue mass**. The other final-layer heads lack a comparable predecessor-matching signature.
- **Corpus evidence is weak.** `W_proj.3` retains substantial singular values through dimensions 23–28: **10.367, 8.351, 8.064, 7.914, 7.461, 7.166**. There is no compelling small-alphabet cutoff, but these dimensions cannot simply be identified with characters.
## Heads
“Local” assignments below are tentative: the shared positional selector does not establish an exact attention distance.
L0H0: **broad** — Low-rank positional pooling candidate; effective QK norm 0.0546, OV positional-input fraction 0.858.
L0H1: **broad** — Shares the pooling pathway; QK norm 0.0295, OV positional-input fraction 0.881.
L0H2: **previous** — Strong, oscillatory, approximately shared query/key subspace; clearest shift candidate.
L0H3: **local** — Tentative positional-locality head; QK transpose symmetry 0.478 and substantial overlap with L0H2’s positional subspace.
L0H4: **broad** — Low-rank pooling pathway; OV cosine 0.903 with L0H0 and positional-input fraction 0.799.
L0H5: **broad** — Same positional-summary family; positional-input fraction 0.851 and dominant OV singular value 8.328.
L1H0: **local** — Common positional selector; bias/QK norm ratio 1.531.
L1H1: **local** — Same mechanism, with ratio 1.738; 53.7% of QK key-side energy reads the leading two summary-output directions.
L1H2: **local** — Common selector, ratio 1.671; no comparably strong induction signature.
L1H3: **local** — Common selector, ratio 1.711; strong dependence on the positional-summary pathway.
L1H4: **local** — Strongest two-dimensional summary-key overlap in L1, 60.3%; selector ratio 1.580.
L1H5: **local** — Common selector, ratio 1.312; summary-key overlap 54.9%.
L2H0: **local** — Selector cosine 0.806 with L1’s common direction; substantial positional-selection component.
L2H1: **unclear** — Weaker positional bias; strong downstream composition with L3H1 suggests a supporting context-retrieval role, but not a specific distance.
L2H2: **local** — Positional selector persists; leading two summary directions account for 17.7% of key-side energy.
L2H3: **local** — Selector cosine 0.814 and bias/QK ratio 0.701.
L2H4: **local** — Selector cosine 0.804 and ratio 0.673; no strong predecessor-matching signature.
L2H5: **local** — Selector cosine 0.814 and ratio 0.670.
L3H0: **unclear** — Positive direct matching, but not predecessor matching; this does not establish induction or attention distance.
L3H1: **copy** — Clear positive, relatively symmetric composition with L0H2’s OV pathway; strongest induction candidate.
L3H2: **local** — Tentative: retains the common positional selector, cosine 0.621, without strong induction composition.
L3H3: **local** — Tentative: selector cosine 0.652; predecessor matching is negligible.
L3H4: **local** — Tentative: selector cosine 0.602; no convincing induction signature.
L3H5: **unclear** — Strong OV and relatively weak QK could support broad aggregation, but do not establish its attention span.
## Reasoning
A likely predecessor channel feeding a later induction-like matcher favors **as-is** training: ordered repetitions provide a reason to match an earlier predecessor and retrieve its successor. Line shuffling would substantially weaken that benefit, though it would not eliminate all sequential structure.
**English is my narrow corpus preference**, followed by Python. The architecture suggests ordinary local character prediction plus reuse of earlier text, without a clearly identifiable indentation-specific mechanism or restricted-alphabet signature. Neither absence is decisive: the missing character interfaces prevent assigning these features to letters, spaces, or newlines. Corpus probabilities therefore remain deliberately modest.
corpus: **english**
order: **as is**
probabilities: python=0.36 romaji=0.20 english=0.44; as is=0.94 shuffled=0.06