Public export: private machine paths and host identifiers are replaced with placeholders. Recorded results and numerical weights are unchanged. Embedded source hashes identify the original files; PUBLICATION-MANIFEST.json identifies these public copies.
Built-in Codex tools were disabled. Harness execution is separately permitted only in Experiments 3 and 6. Text below is taken from saved records; it is not a new model run. Reasoning blocks are the summaries actually emitted by the CLI.
Answer the user's request.
Recorded prompt · Raw CLI events · Settings and audit
Below is a small character-level language model: a 4-layer transformer with 6 attention heads per layer, which reads up to 128 characters and outputs scores (logits) for the next character. This is exactly how it computes (numpy):
```python
import numpy as np
VOCAB = "\n " + "abcdefghijklmnopqrstuvwxyz" # index 0 = newline, 1 = space, 2..27 = a..z
HEADS, LAYERS = 6, 4
def layer_norm(x, w, b): # over the last axis, eps 1e-5
mu = x.mean(-1, keepdims=True)
var = ((x - mu) ** 2).mean(-1, keepdims=True)
return (x - mu) / np.sqrt(var + 1e-5) * w + b
def next_char_logits(text, p):
"""text: up to 128 characters (only the last 128 are used); p: dict of the weight arrays.
Returns the 28 logits for the character that comes next."""
ids = [VOCAB.index(c) for c in text[-128:]]
n, d = len(ids), p["tok_emb"].shape[1]
hd = d // HEADS
x = p["tok_emb"][ids] + p["pos_emb"][:n] # (n, d); positions 0..n-1 from the start of the text
for l in range(LAYERS): # layer l uses the tensors named <name>.<l>
h = layer_norm(x, p[f"ln1_w.{l}"], p[f"ln1_b.{l}"])
q, k, v = h @ p[f"W_q.{l}"].T, h @ p[f"W_k.{l}"].T, h @ p[f"W_v.{l}"].T
out = np.zeros((n, d))
for j in range(HEADS): # head j uses dimensions j*hd .. (j+1)*hd - 1
s = slice(j * hd, (j + 1) * hd)
att = q[:, s] @ k[:, s].T / np.sqrt(hd) # (n, n)
att = att + np.triu(np.full((n, n), -np.inf), 1) # causal: position i sees positions 0..i
att = np.exp(att - att.max(-1, keepdims=True))
att = att / att.sum(-1, keepdims=True)
out[:, s] = att @ v[:, s]
x = x + out @ p[f"W_o.{l}"].T
h = layer_norm(x, p[f"ln2_w.{l}"], p[f"ln2_b.{l}"])
x = x + np.maximum(0.0, h @ p[f"W_fc.{l}"].T) @ p[f"W_proj.{l}"].T # ReLU MLP, hidden 4 d
h = layer_norm(x, p["lnf_w"], p["lnf_b"])
return h[-1] @ p["W_out"].T + p["b_out"] # logits of the next character (28)
```
Training: from random initialization (every matrix entry drawn from a normal distribution with standard deviation 0.02, or 0.007071 for W_o and W_proj; LayerNorm weights 1 and biases 0; b_out 0), next-character prediction with AdamW (30,517 steps of 256 random 128-character windows, 1,000,000,000 characters in total; learning-rate warm-up then cosine decay; weight decay 0.01 on the matrices).
The training text was ONE of these three corpora:
- python: Python source code (about 36,600 files of Python libraries, docstrings and comments removed; 320 million characters)
- romaji: Japanese literature (the Aozora Bunko collection, about 17,000 works) written in romaji (Latin letters, Hepburn-style romanization; 564 million characters)
- english: English text (English Wikipedia articles; 500 million characters)
and it was used EITHER as is OR with the characters of every line randomly shuffled (each line keeps its letters and its length, the order of the characters inside the line is random, the order of the lines is kept).
Normalization (the same for all three corpora, before anything else): lowercased; in each line the leading spaces (indentation) were kept, every other character that is not a letter a-z became a space, runs of spaces were collapsed to one and trailing spaces removed. So the text contains only newlines, spaces and the letters a-z (the 28 characters of VOCAB).
WHAT YOU CAN USE. The weights inside the four transformer blocks, exactly as trained (float32): for layer l = 0, 1, 2 and 3, ln1_w.l, ln1_b.l, W_q.l, W_k.l, W_v.l, W_o.l (attention) and ln2_w.l, ln2_b.l, W_fc.l, W_proj.l (MLP). The input layer (tok_emb, the 28 x d character embeddings, and pos_emb, the 128 x d position embeddings) and the output layer (lnf_w, lnf_b, W_out, b_out) are NOT available. The d = 384 dimensions of the residual stream carry no labels. Head j of layer l uses rows 64j .. 64j+63 of W_q.l, W_k.l and W_v.l and columns 64j .. 64j+63 of W_o.l.
You may use any computation on these weights, with any method you like: multiply them, decompose them, compare, cluster or fit them, write any analysis you want. Two things are NOT allowed:
1. A forward pass: do not run the network or any of its layers (LayerNorm, attention with its softmax, the MLP with its ReLU) on any input, whether real, random or constructed, in code, in your head or on paper. Treating the weight matrices as linear-algebra objects (products, eigenvalues, singular vectors, applying one weight matrix to directions taken from other weights) is fine; simulating what the network does to an input is not.
2. Search: do not look anything up (there is no internet access in any case).
HOW YOU COMPUTE. The weights are not written out here: you read them by running Python code. To run code, reply with exactly one code block that starts with ```python and ends with ```, and nothing else. We run it and reply with its printed output. Available: numpy, scipy, scikit-learn, PyTorch (CPU), pandas, sympy, networkx. Before your code runs, numpy is imported as np and W is a dict of the weight arrays by name: W['W_q.0'] is 384 x 384, W['W_fc.0'] is 1536 x 384, W['W_proj.0'] is 384 x 1536, W['ln1_w.0'] has 384 entries, and so on; matrices are (out, in), as used in the code above. There is nothing else to read (no other files, no network). Each run is a fresh process (nothing is kept between runs), may take up to 600 seconds, and its output is cut at 20,000 characters. You may run code up to 100 times. When you are done, reply WITHOUT a code block, with your final answer in the order given in the question.
QUESTION. What was this network trained on, and how does it use the context? Answer in this order:
Observations: <what you see in the shown weights and what it suggests; quote the weights you rely on>
Heads: <one line per head, L0H0 .. L0H5, L1H0 .. L1H5, L2H0 .. L2H5, L3H0 .. L3H5: what it most likely does, as one of: 'previous' (attends to the previous character), 'local' (a few characters back), 'broad' (spread far back), 'copy' (induction: attends to the character right after an earlier occurrence of the current character or context, so the network can repeat earlier text), or 'unclear'; then your evidence>
Reasoning: <which training text fits these observations, and why>
corpus: <python | romaji | english>
order: <as is | shuffled>
probabilities: python=<p> romaji=<p> english=<p>; as is=<p> shuffled=<p>
Equal probabilities are not allowed. Even if the evidence is weak, give your best judgement.Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
def t(a):
return torch.as_tensor(np.asarray(a), dtype=torch.float64)
def spec(a):
s = torch.linalg.svdvals(a)
energy = s.square()
rank90 = int(torch.searchsorted(torch.cumsum(energy, 0), .9 * energy.sum())) + 1
return float(torch.linalg.norm(a)), float(s[0]), float(energy.sum()/energy[0]), rank90
print("All calculations below are on weights, not network activations.")
for l in range(4):
print("\nLAYER", l)
for name in ["ln1_w", "ln1_b", "ln2_w", "ln2_b"]:
a = np.asarray(W[f"{name}.{l}"])
print(name, "mean/std/min/max/norm",
np.round([a.mean(), a.std(), a.min(), a.max(), np.linalg.norm(a)], 4))
for name in ["W_q", "W_k", "W_v", "W_o", "W_fc", "W_proj"]:
print(name, "Frob/topSV/stableRank/rank90",
tuple(round(x, 3) if isinstance(x, float) else x
for x in spec(t(W[f"{name}.{l}"]))))
print("\nHEADS: raw QK = Q.T @ K / 8; OV = O @ V")
print("QK: Frob, topSV, stableRank, rank90, trace, symmetric min/max eigenvalue")
print("OV: Frob, topSV, stableRank, rank90, trace, largest-real eigenvalues")
for l in range(4):
for j in range(6):
sl = slice(64*j, 64*(j+1))
q = t(W[f"W_q.{l}"])[sl]
k = t(W[f"W_k.{l}"])[sl]
v = t(W[f"W_v.{l}"])[sl]
o = t(W[f"W_o.{l}"])[:, sl]
a = q.T @ k / 8
b = o @ v
ae = torch.linalg.eigvalsh((a+a.T)/2)
be = torch.linalg.eigvals(v @ o)
idx = torch.argsort(be.real, descending=True)[:4]
print(f"L{l}H{j}",
"QK", tuple(round(x, 3) if isinstance(x, float) else x for x in spec(a)),
"tr", round(float(torch.trace(a)), 3),
"sym", np.round([float(ae[0]), float(ae[-1])], 3),
"OV", tuple(round(x, 3) if isinstance(x, float) else x for x in spec(b)),
"tr", round(float(torch.trace(b)), 3),
"eig", [(round(float(be[z].real), 3), round(float(be[z].imag), 3)) for z in idx])
```
{
"t": 1792713203.2770932,
"s": 11.64,
"code": "import numpy as np\nimport torch\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\n\ndef t(a):\n return torch.as_tensor(np.asarray(a), dtype=torch.float64)\n\ndef spec(a):\n s = torch.linalg.svdvals(a)\n energy = s.square()\n rank90 = int(torch.searchsorted(torch.cumsum(energy, 0), .9 * energy.sum())) + 1\n return float(torch.linalg.norm(a)), float(s[0]), float(energy.sum()/energy[0]), rank90\n\nprint(\"All calculations below are on weights, not network activations.\")\nfor l in range(4):\n print(\"\\nLAYER\", l)\n for name in [\"ln1_w\", \"ln1_b\", \"ln2_w\", \"ln2_b\"]:\n a = np.asarray(W[f\"{name}.{l}\"])\n print(name, \"mean/std/min/max/norm\",\n np.round([a.mean(), a.std(), a.min(), a.max(), np.linalg.norm(a)], 4))\n for name in [\"W_q\", \"W_k\", \"W_v\", \"W_o\", \"W_fc\", \"W_proj\"]:\n print(name, \"Frob/topSV/stableRank/rank90\",\n tuple(round(x, 3) if isinstance(x, float) else x\n for x in spec(t(W[f\"{name}.{l}\"]))))\n\nprint(\"\\nHEADS: raw QK = Q.T @ K / 8; OV = O @ V\")\nprint(\"QK: Frob, topSV, stableRank, rank90, trace, symmetric min/max eigenvalue\")\nprint(\"OV: Frob, topSV, stableRank, rank90, trace, largest-real eigenvalues\")\nfor l in range(4):\n for j in range(6):\n sl = slice(64*j, 64*(j+1))\n q = t(W[f\"W_q.{l}\"])[sl]\n k = t(W[f\"W_k.{l}\"])[sl]\n v = t(W[f\"W_v.{l}\"])[sl]\n o = t(W[f\"W_o.{l}\"])[:, sl]\n a = q.T @ k / 8\n b = o @ v\n ae = torch.linalg.eigvalsh((a+a.T)/2)\n be = torch.linalg.eigvals(v @ o)\n idx = torch.argsort(be.real, descending=True)[:4]\n print(f\"L{l}H{j}\",\n \"QK\", tuple(round(x, 3) if isinstance(x, float) else x for x in spec(a)),\n \"tr\", round(float(torch.trace(a)), 3),\n \"sym\", np.round([float(ae[0]), float(ae[-1])], 3),\n \"OV\", tuple(round(x, 3) if isinstance(x, float) else x for x in spec(b)),\n \"tr\", round(float(torch.trace(b)), 3),\n \"eig\", [(round(float(be[z].real), 3), round(float(be[z].imag), 3)) for z in idx])\n",
"out": "All calculations below are on weights, not network activations.\n\nLAYER 0\nln1_w mean/std/min/max/norm [0.2677 0.2147 0.0165 1.0134 6.7241]\nln1_b mean/std/min/max/norm [ 0.0007 0.0366 -0.1215 0.1418 0.7172]\nln2_w mean/std/min/max/norm [ 0.8168 0.3312 0.0468 1.6412 17.2723]\nln2_b mean/std/min/max/norm [ 0.018 0.0667 -0.5537 0.1663 1.353 ]\nW_q Frob/topSV/stableRank/rank90 (22.704, 5.039, 20.305, 61)\nW_k Frob/topSV/stableRank/rank90 (23.214, 6.204, 14.001, 60)\nW_v Frob/topSV/stableRank/rank90 (18.197, 4.553, 15.972, 66)\nW_o Frob/topSV/stableRank/rank90 (19.6, 6.678, 8.614, 75)\nW_fc Frob/topSV/stableRank/rank90 (51.65, 18.11, 8.134, 87)\nW_proj Frob/topSV/stableRank/rank90 (45.806, 10.932, 17.557, 135)\n\nLAYER 1\nln1_w mean/std/min/max/norm [0.4754 0.0964 0.1979 1.2096 9.5046]\nln1_b mean/std/min/max/norm [ 0.022 0.1532 -1.0505 0.9102 3.0327]\nln2_w mean/std/min/max/norm [ 1.0111 0.1582 0.0139 1.4707 20.0543]\nln2_b mean/std/min/max/norm [ 0.0275 0.1442 -0.3744 0.3652 2.8769]\nW_q Frob/topSV/stableRank/rank90 (31.676, 8.984, 12.43, 66)\nW_k Frob/topSV/stableRank/rank90 (28.602, 10.967, 6.801, 68)\nW_v Frob/topSV/stableRank/rank90 (28.469, 5.35, 28.31, 84)\nW_o Frob/topSV/stableRank/rank90 (26.151, 4.19, 38.953, 119)\nW_fc Frob/topSV/stableRank/rank90 (80.261, 20.796, 14.896, 93)\nW_proj Frob/topSV/stableRank/rank90 (75.119, 7.625, 97.059, 280)\n\nLAYER 2\nln1_w mean/std/min/max/norm [ 0.7343 0.0712 0.3445 0.9047 14.4574]\nln1_b mean/std/min/max/norm [ 0.0244 0.1315 -0.9417 0.7326 2.6214]\nln2_w mean/std/min/max/norm [ 1.275 0.1126 0.0775 1.5018 25.082 ]\nln2_b mean/std/min/max/norm [ 0.0188 0.1474 -0.4157 0.3635 2.911 ]\nW_q Frob/topSV/stableRank/rank90 (37.483, 5.067, 54.718, 170)\nW_k Frob/topSV/stableRank/rank90 (36.356, 8.293, 19.216, 169)\nW_v Frob/topSV/stableRank/rank90 (34.746, 3.89, 79.785, 176)\nW_o Frob/topSV/stableRank/rank90 (34.205, 4.442, 59.308, 190)\nW_fc Frob/topSV/stableRank/rank90 (79.454, 14.829, 28.71, 252)\nW_proj Frob/topSV/stableRank/rank90 (77.477, 10.095, 58.906, 272)\n\nLAYER 3\nln1_w mean/std/min/max/norm [ 0.7694 0.0469 0.451 0.8778 15.1046]\nln1_b mean/std/min/max/norm [ 0.0102 0.0945 -0.5141 0.6285 1.8619]\nln2_w mean/std/min/max/norm [ 1.2552 0.0888 0.2619 1.4568 24.6574]\nln2_b mean/std/min/max/norm [ 0.014 0.1267 -0.5112 0.3808 2.4972]\nW_q Frob/topSV/stableRank/rank90 (33.94, 4.154, 66.743, 186)\nW_k Frob/topSV/stableRank/rank90 (33.221, 4.888, 46.201, 190)\nW_v Frob/topSV/stableRank/rank90 (38.939, 4.697, 68.734, 179)\nW_o Frob/topSV/stableRank/rank90 (39.017, 9.175, 18.083, 172)\nW_fc Frob/topSV/stableRank/rank90 (78.189, 14.373, 29.593, 279)\nW_proj Frob/topSV/stableRank/rank90 (85.089, 28.175, 9.12, 25)\n\nHEADS: raw QK = Q.T @ K / 8; OV = O @ V\nQK: Frob, topSV, stableRank, rank90, trace, symmetric min/max eigenvalue\nOV: Frob, topSV, stableRank, rank90, trace, largest-real eigenvalues\nL0H0 QK (0.807, 0.649, 1.545, 2) tr -0.529 sym [-0.482 0.207] OV (8.667, 6.898, 1.579, 4) tr 1.62 eig [(1.827, 0.0), (0.517, 0.0), (0.125, 0.058), (0.125, -0.058)]\nL0H1 QK (4.067, 2.153, 3.568, 14) tr -5.152 sym [-2.094 0.759] OV (17.067, 4.703, 13.166, 20) tr -0.82 eig [(1.036, 0.0), (0.285, 0.0), (0.252, -0.117), (0.252, 0.117)]\nL0H2 QK (1.136, 0.561, 4.101, 11) tr -0.964 sym [-0.356 0.328] OV (8.592, 4.786, 3.222, 7) tr -0.743 eig [(0.182, 0.101), (0.182, -0.101), (0.165, 0.0), (0.104, 0.152)]\nL0H3 QK (1.434, 0.959, 2.233, 7) tr -1.144 sym [-0.563 0.406] OV (5.288, 2.168, 5.949, 12) tr -0.071 eig [(0.406, 0.0), (0.154, 0.0), (0.125, 0.0), (0.092, -0.036)]\nL0H4 QK (0.097, 0.04, 5.859, 41) tr -0.059 sym [-0.027 0.013] OV (25.795, 18.815, 1.88, 3) tr 6.681 eig [(3.714, -0.514), (3.714, 0.514), (0.212, 0.2), (0.212, -0.2)]\nL0H5 QK (5.785, 1.76, 10.809, 18) tr 6.516 sym [-0.532 1.477] OV (18.558, 5.61, 10.941, 17) tr -11.398 eig [(0.45, 0.0), (0.124, 0.52), (0.124, -0.52), (0.079, 0.0)]\nL1H0 QK (2.471, 1.116, 4.901, 15) tr -1.587 sym [-0.688 0.475] OV (27.504, 9.927, 7.677, 16) tr -34.568 eig [(0.83"
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 1 of at most 100: ``` All calculations below are on weights, not network activations. LAYER 0 ln1_w mean/std/min/max/norm [0.2677 0.2147 0.0165 1.0134 6.7241] ln1_b mean/std/min/max/norm [ 0.0007 0.0366 -0.1215 0.1418 0.7172] ln2_w mean/std/min/max/norm [ 0.8168 0.3312 0.0468 1.6412 17.2723] ln2_b mean/std/min/max/norm [ 0.018 0.0667 -0.5537 0.1663 1.353 ] W_q Frob/topSV/stableRank/rank90 (22.704, 5.039, 20.305, 61) W_k Frob/topSV/stableRank/rank90 (23.214, 6.204, 14.001, 60) W_v Frob/topSV/stableRank/rank90 (18.197, 4.553, 15.972, 66) W_o Frob/topSV/stableRank/rank90 (19.6, 6.678, 8.614, 75) W_fc Frob/topSV/stableRank/rank90 (51.65, 18.11, 8.134, 87) W_proj Frob/topSV/stableRank/rank90 (45.806, 10.932, 17.557, 135) LAYER 1 ln1_w mean/std/min/max/norm [0.4754 0.0964 0.1979 1.2096 9.5046] ln1_b mean/std/min/max/norm [ 0.022 0.1532 -1.0505 0.9102 3.0327] ln2_w mean/std/min/max/norm [ 1.0111 0.1582 0.0139 1.4707 20.0543] ln2_b mean/std/min/max/norm [ 0.0275 0.1442 -0.3744 0.3652 2.8769] W_q Frob/topSV/stableRank/rank90 (31.676, 8.984, 12.43, 66) W_k Frob/topSV/stableRank/rank90 (28.602, 10.967, 6.801, 68) W_v Frob/topSV/stableRank/rank90 (28.469, 5.35, 28.31, 84) W_o Frob/topSV/stableRank/rank90 (26.151, 4.19, 38.953, 119) W_fc Frob/topSV/stableRank/rank90 (80.261, 20.796, 14.896, 93) W_proj Frob/topSV/stableRank/rank90 (75.119, 7.625, 97.059, 280) LAYER 2 ln1_w mean/std/min/max/norm [ 0.7343 0.0712 0.3445 0.9047 14.4574] ln1_b mean/std/min/max/norm [ 0.0244 0.1315 -0.9417 0.7326 2.6214] ln2_w mean/std/min/max/norm [ 1.275 0.1126 0.0775 1.5018 25.082 ] ln2_b mean/std/min/max/norm [ 0.0188 0.1474 -0.4157 0.3635 2.911 ] W_q Frob/topSV/stableRank/rank90 (37.483, 5.067, 54.718, 170) W_k Frob/topSV/stableRank/rank90 (36.356, 8.293, 19.216, 169) W_v Frob/topSV/stableRank/rank90 (34.746, 3.89, 79.785, 176) W_o Frob/topSV/stableRank/rank90 (34.205, 4.442, 59.308, 190) W_fc Frob/topSV/stableRank/rank90 (79.454, 14.829, 28.71, 252) W_proj Frob/topSV/stableRank/rank90 (77.477, 10.095, 58.906, 272) LAYER 3 ln1_w mean/std/min/max/norm [ 0.7694 0.0469 0.451 0.8778 15.1046] ln1_b mean/std/min/max/norm [ 0.0102 0.0945 -0.5141 0.6285 1.8619] ln2_w mean/std/min/max/norm [ 1.2552 0.0888 0.2619 1.4568 24.6574] ln2_b mean/std/min/max/norm [ 0.014 0.1267 -0.5112 0.3808 2.4972] W_q Frob/topSV/stableRank/rank90 (33.94, 4.154, 66.743, 186) W_k Frob/topSV/stableRank/rank90 (33.221, 4.888, 46.201, 190) W_v Frob/topSV/stableRank/rank90 (38.939, 4.697, 68.734, 179) W_o Frob/topSV/stableRank/rank90 (39.017, 9.175, 18.083, 172) W_fc Frob/topSV/stableRank/rank90 (78.189, 14.373, 29.593, 279) W_proj Frob/topSV/stableRank/rank90 (85.089, 28.175, 9.12, 25) HEADS: raw QK = Q.T @ K / 8; OV = O @ V QK: Frob, topSV, stableRank, rank90, trace, symmetric min/max eigenvalue OV: Frob, topSV, stableRank, rank90, trace, largest-real eigenvalues L0H0 QK (0.807, 0.649, 1.545, 2) tr -0.529 sym [-0.482 0.207] OV (8.667, 6.898, 1.579, 4) tr 1.62 eig [(1.827, 0.0), (0.517, 0.0), (0.125, 0.058), (0.125, -0.058)] L0H1 QK (4.067, 2.153, 3.568, 14) tr -5.152 sym [-2.094 0.759] OV (17.067, 4.703, 13.166, 20) tr -0.82 eig [(1.036, 0.0), (0.285, 0.0), (0.252, -0.117), (0.252, 0.117)] L0H2 QK (1.136, 0.561, 4.101, 11) tr -0.964 sym [-0.356 0.328] OV (8.592, 4.786, 3.222, 7) tr -0.743 eig [(0.182, 0.101), (0.182, -0.101), (0.165, 0.0), (0.104, 0.152)] L0H3 QK (1.434, 0.959, 2.233, 7) tr -1.144 sym [-0.563 0.406] OV (5.288, 2.168, 5.949, 12) tr -0.071 eig [(0.406, 0.0), (0.154, 0.0), (0.125, 0.0), (0.092, -0.036)] L0H4 QK (0.097, 0.04, 5.859, 41) tr -0.059 sym [-0.027 0.013] OV (25.795, 18.815, 1.88, 3) tr 6.681 eig [(3.714, -0.514), (3.714, 0.514), (0.212, 0.2), (0.212, -0.2)] L0H5 QK (5.785, 1.76, 10.809, 18) tr 6.516 sym [-0.532 1.477] OV (18.558, 5.61, 10.941, 17) tr -11.398 eig [(0.45, 0.0), (0.124, 0.52), (0.124, -0.52), (0.079, 0.0)] L1H0 QK (2.471, 1.116, 4.901, 15) tr -1.587 sym [-0.688 0.475] OV (27.504, 9.927, 7.677, 16) tr -34.568 eig [(0.831, 0.0), (0.245, 0.0), (0.22, -0.251), (0.22, 0.251)] L1H1 QK (4.754, 2.392, 3.95, 14) tr -5.939 sym [-1.371 1.106] OV (19.961, 6.211, 10.328, 21) tr -23.812 eig [(0.336, 0.0), (0.196, 0.225), (0.196, -0.225), (0.151, 0.023)] L1H2 QK (3.618, 1.811, 3.991, 14) tr -3.542 sym [-1.004 0.914] OV (23.71, 6.799, 12.16, 20) tr -31.928 eig [(0.374, 0.0), (0.244, 0.0), (0.14, -0.034), (0.14, 0.034)] L1H3 QK (3.776, 1.998, 3.57, 14) tr -3.287 sym [-0.925 1.063] OV (20.845, 7.119, 8.574, 18) tr -22.474 eig [(0.293, -0.287), (0.293, 0.287), (0.258, 0.0), (0.136, -0.179)] L1H4 QK (3.647, 1.853, 3.875, 14) tr -2.828 sym [-0.981 0.96 ] OV (19.572, 6.354, 9.489, 18) tr -21.931 eig [(0.212, -0.018), (0.212, 0.018), (0.13, 0.0), (0.07, 0.045)] L1H5 QK (4.695, 2.523, 3.463, 13) tr -5.56 sym [-1.391 1.164] OV (20.352, 5.781, 12.395, 22) tr -27.708 eig [(0.152, 0.062), (0.152, -0.062), (0.103, -0.126), (0.103, 0.126)] L2H0 QK (3.752, 0.935, 16.09, 39) tr -0.904 sym [-0.483 0.554] OV (25.165, 5.556, 20.518, 41) tr -48.29 eig [(0.37, -0.221), (0.37, 0.221), (0.207, 0.139), (0.207, -0.139)] L2H1 QK (3.61, 0.907, 15.831, 39) tr 4.721 sym [-0.466 0.5 ] OV (28.885, 8.652, 11.146, 39) tr -54.375 eig [(0.34, 0.0), (0.29, 0.0), (0.131, 0.155), (0.131, -0.155)] L2H2 QK (5.221, 1.354, 14.873, 31) tr 2.955 sym [-0.778 0.853] OV (22.619, 4.742, 22.749, 42) tr -39.941 eig [(0.468, 0.0), (0.371, 0.0), (0.329, 0.0), (0.207, 0.496)] L2H3 QK (3.648, 1.012, 13.006, 39) tr -0.606 sym [-0.481 0.545] OV (27.634, 6.179, 19.999, 42) tr -51.741 eig [(0.408, 0.0), (0.333, 0.301), (0.333, -0.301), (0.299, 0.588)] L2H4 QK (4.112, 1.451, 8.031, 36) tr 0.383 sym [-0.636 0.827] OV (24.785, 5.445, 20.721, 38) tr -52.123 eig [(0.167, 0.0), (0.127, 0.295), (0.127, -0.295), (0.044, -0.047)] L2H5 QK (3.041, 0.788, 14.895, 42) tr -1.284 sym [-0.415 0.452] OV (31.883, 8.467, 14.178, 40) tr -45.69 eig [(0.814, 0.0), (0.628, -0.255), (0.628, 0.255), (0.425, -0.399)] L3H0 QK (3.5, 0.699, 25.045, 48) tr 2.983 sym [-0.362 0.436] OV (31.618, 7.123, 19.702, 44) tr -62.391 eig [(0.372, -0.333), (0.372, 0.333), (0.314, 0.0), (0.276, -0.501)] L3H1 QK (2.865, 0.55, 27.15, 48) tr 3.811 sym [-0.314 0.345] OV (33.152, 9.373, 12.509, 37) tr -63.987 eig [(0.278, 0.0), (0.212, 0.109), (0.212, -0.109), (0.1, 0.025)] L3H2 QK (3.474, 0.781, 19.8, 48) tr 1.228 sym [-0.367 0.45 ] OV (34.637, 11.577, 8.952, 42) tr -65.099 eig [(0.409, -0.566), (0.409, 0.566), (0.362, 0.0), (0.358, 0.0)] L3H3 QK (1.784, 0.461, 14.988, 47) tr 0.003 sym [-0.228 0.25 ] OV (48.677, 18.625, 6.83, 32) tr -65.504 eig [(1.165, 0.0), (0.777, 0.248), (0.777, -0.248), (0.713, 0.451)] L3H4 QK (2.93, 0.715, 16.786, 47) tr 0.036 sym [-0.377 0.432] OV (36.644, 11.774, 9.686, 40) tr -45.155 eig [(1.042, 0.0), (0.709, -0.664), (0.709, 0.664), (0.555, 0.2)] L3H5 QK (3.581, 0.676, 28.066, 48) tr 13.669 sym [-0.261 0.546] OV (30.624, 13.046, 5.51, 26) tr -73.74 eig [(0.282, 0.0), (0.275, 0.12), (0.275, -0.12), (0.195, 0.0)] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T = lambda a: torch.as_tensor(np.asarray(a), dtype=torch.float64)
P = torch.eye(384) - torch.ones((384,384))/384
M, B = {}, {}
print("Effective linear maps include LayerNorm gain and mean-centering only.")
print("No normalization of inputs, attention, or MLP evaluation is performed.")
for l in range(4):
g = T(W[f"ln1_w.{l}"])
b = T(W[f"ln1_b.{l}"])
for j in range(6):
s = slice(j*64,(j+1)*64)
q, k, v = [T(W[f"{name}.{l}"])[s] for name in ["W_q","W_k","W_v"]]
o = T(W[f"W_o.{l}"])[:,s]
qe, ke, ve = (q*g)@P, (k*g)@P, (v*g)@P
a, c = qe.T@ke/8, P@o@ve
M[l,j], B[l,j] = a,c
sv = torch.linalg.svdvals(a)
cs = torch.linalg.svdvals(c)
keybias = ke.T@(q@b)/8
print(f"L{l}H{j}",
"QK_F/top/r90/tr",
np.round([float(a.norm()),float(sv[0]),
int(torch.searchsorted(sv.square().cumsum(0),.9*sv.square().sum()))+1,
float(torch.trace(a))],4),
"QK_svs", np.round(sv[:6].numpy(),3),
"OV_F/tr",np.round([float(c.norm()),float(torch.trace(c))],3),
"key_bias_norm",round(float(keybias.norm()),4))
print("\nWithin-layer QK cosine matrices:")
for l in range(4):
print("L",l)
print(np.round([[float((M[l,i]*M[l,j]).sum()/(M[l,i].norm()*M[l,j].norm()))
for j in range(6)] for i in range(6)],3))
print("\nKey-composition trace cosine: trace(QK_later @ OV_earlier)/(norms)")
for l in range(1,4):
for j in range(6):
a=M[l,j]
scores=[]
for p in range(l):
for h in range(6):
c=B[p,h]
z=float(torch.trace(a@c)/(a.norm()*c.norm()))
scores.append((z,f"L{p}H{h}"))
scores.sort(reverse=True)
print(f"L{l}H{j}", "highest",[(n,round(z,4)) for z,n in scores[:4]],
"lowest",[(n,round(z,4)) for z,n in scores[-2:]])
print("\nL0 gain extremes and corresponding linear-map column norms:")
g=np.asarray(W["ln1_w.0"])
idx=np.r_[np.argsort(g)[:12],np.argsort(g)[-12:]]
for i in idx:
print(int(i), "g1/g2",round(float(g[i]),4),round(float(W["ln2_w.0"][i]),4),
"q/k/v/fc", np.round([np.linalg.norm(W[n+".0"][:,i])
for n in ["W_q","W_k","W_v","W_fc"]],3))
print("\nLinear MLP weight products (not MLP evaluations):")
for l in range(4):
a=P@T(W[f"W_proj.{l}"])@(T(W[f"W_fc.{l}"])*T(W[f"ln2_w.{l}"]))@P
ev=torch.linalg.eigvals(a)
print(l,"Frob/trace",round(float(a.norm()),3),round(float(torch.trace(a)),3),
"eig real min/max",round(float(ev.real.min()),3),round(float(ev.real.max()),3))
```
{
"t": 1792713284.982677,
"s": 10.55,
"code": "import numpy as np\nimport torch\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT = lambda a: torch.as_tensor(np.asarray(a), dtype=torch.float64)\nP = torch.eye(384) - torch.ones((384,384))/384\nM, B = {}, {}\n\nprint(\"Effective linear maps include LayerNorm gain and mean-centering only.\")\nprint(\"No normalization of inputs, attention, or MLP evaluation is performed.\")\nfor l in range(4):\n g = T(W[f\"ln1_w.{l}\"])\n b = T(W[f\"ln1_b.{l}\"])\n for j in range(6):\n s = slice(j*64,(j+1)*64)\n q, k, v = [T(W[f\"{name}.{l}\"])[s] for name in [\"W_q\",\"W_k\",\"W_v\"]]\n o = T(W[f\"W_o.{l}\"])[:,s]\n qe, ke, ve = (q*g)@P, (k*g)@P, (v*g)@P\n a, c = qe.T@ke/8, P@o@ve\n M[l,j], B[l,j] = a,c\n sv = torch.linalg.svdvals(a)\n cs = torch.linalg.svdvals(c)\n keybias = ke.T@(q@b)/8\n print(f\"L{l}H{j}\",\n \"QK_F/top/r90/tr\",\n np.round([float(a.norm()),float(sv[0]),\n int(torch.searchsorted(sv.square().cumsum(0),.9*sv.square().sum()))+1,\n float(torch.trace(a))],4),\n \"QK_svs\", np.round(sv[:6].numpy(),3),\n \"OV_F/tr\",np.round([float(c.norm()),float(torch.trace(c))],3),\n \"key_bias_norm\",round(float(keybias.norm()),4))\n\nprint(\"\\nWithin-layer QK cosine matrices:\")\nfor l in range(4):\n print(\"L\",l)\n print(np.round([[float((M[l,i]*M[l,j]).sum()/(M[l,i].norm()*M[l,j].norm()))\n for j in range(6)] for i in range(6)],3))\n\nprint(\"\\nKey-composition trace cosine: trace(QK_later @ OV_earlier)/(norms)\")\nfor l in range(1,4):\n for j in range(6):\n a=M[l,j]\n scores=[]\n for p in range(l):\n for h in range(6):\n c=B[p,h]\n z=float(torch.trace(a@c)/(a.norm()*c.norm()))\n scores.append((z,f\"L{p}H{h}\"))\n scores.sort(reverse=True)\n print(f\"L{l}H{j}\", \"highest\",[(n,round(z,4)) for z,n in scores[:4]],\n \"lowest\",[(n,round(z,4)) for z,n in scores[-2:]])\n\nprint(\"\\nL0 gain extremes and corresponding linear-map column norms:\")\ng=np.asarray(W[\"ln1_w.0\"])\nidx=np.r_[np.argsort(g)[:12],np.argsort(g)[-12:]]\nfor i in idx:\n print(int(i), \"g1/g2\",round(float(g[i]),4),round(float(W[\"ln2_w.0\"][i]),4),\n \"q/k/v/fc\", np.round([np.linalg.norm(W[n+\".0\"][:,i])\n for n in [\"W_q\",\"W_k\",\"W_v\",\"W_fc\"]],3))\n\nprint(\"\\nLinear MLP weight products (not MLP evaluations):\")\nfor l in range(4):\n a=P@T(W[f\"W_proj.{l}\"])@(T(W[f\"W_fc.{l}\"])*T(W[f\"ln2_w.{l}\"]))@P\n ev=torch.linalg.eigvals(a)\n print(l,\"Frob/trace\",round(float(a.norm()),3),round(float(torch.trace(a)),3),\n \"eig real min/max\",round(float(ev.real.min()),3),round(float(ev.real.max()),3))\n",
"out": "Effective linear maps include LayerNorm gain and mean-centering only.\nNo normalization of inputs, attention, or MLP evaluation is performed.\nL0H0 QK_F/top/r90/tr [ 0.1137 0.0887 2. -0.0508] QK_svs [0.089 0.068 0.013 0.008 0.006 0.005] OV_F/tr [4.321 1.666] key_bias_norm 0.0185\nL0H1 QK_F/top/r90/tr [ 0.8775 0.6013 6. -0.4868] QK_svs [0.601 0.373 0.297 0.211 0.199 0.155] OV_F/tr [3.488 0.164] key_bias_norm 0.0736\nL0H2 QK_F/top/r90/tr [0.1745 0.132 5. 0.0546] QK_svs [0.132 0.08 0.044 0.031 0.028 0.025] OV_F/tr [2.023 0.015] key_bias_norm 0.0234\nL0H3 QK_F/top/r90/tr [ 0.2395 0.1645 3. -0.1871] QK_svs [0.165 0.14 0.081 0.031 0.029 0.021] OV_F/tr [1.621 0.242] key_bias_norm 0.057\nL0H4 QK_F/top/r90/tr [ 0.0093 0.0045 28. -0.0037] QK_svs [0.004 0.003 0.002 0.002 0.002 0.002] OV_F/tr [17.158 6.049] key_bias_norm 0.0009\nL0H5 QK_F/top/r90/tr [ 0.9704 0.4099 16. 1.2135] QK_svs [0.41 0.362 0.273 0.254 0.245 0.222] OV_F/tr [ 3.707 -2.463] key_bias_norm 0.0509\nL1H0 QK_F/top/r90/tr [ 0.644 0.3538 13. -0.308 ] QK_svs [0.354 0.252 0.218 0.167 0.158 0.139] OV_F/tr [ 13.436 -16.299] key_bias_norm 0.8392\nL1H1 QK_F/top/r90/tr [ 1.5843 1.1456 9. -1.2729] QK_svs [1.146 0.661 0.459 0.326 0.249 0.215] OV_F/tr [ 9.592 -11.205] key_bias_norm 3.1498\nL1H2 QK_F/top/r90/tr [ 1.2358 0.8069 8. -0.7401] QK_svs [0.807 0.693 0.332 0.205 0.18 0.159] OV_F/tr [ 11.496 -14.892] key_bias_norm 2.1998\nL1H3 QK_F/top/r90/tr [ 1.3094 0.9127 8. -0.6948] QK_svs [0.913 0.699 0.309 0.202 0.178 0.167] OV_F/tr [ 10.043 -10.571] key_bias_norm 2.7194\nL1H4 QK_F/top/r90/tr [ 1.2492 0.848 8. -0.5846] QK_svs [0.848 0.654 0.34 0.213 0.191 0.162] OV_F/tr [ 9.557 -10.446] key_bias_norm 2.518\nL1H5 QK_F/top/r90/tr [ 1.4614 1.0088 8. -1.2054] QK_svs [1.009 0.555 0.482 0.45 0.242 0.215] OV_F/tr [ 9.826 -13.045] key_bias_norm 2.4602\nL2H0 QK_F/top/r90/tr [ 2.0456 0.5184 39. -0.5814] QK_svs [0.518 0.485 0.459 0.416 0.407 0.397] OV_F/tr [ 18.698 -35.97 ] key_bias_norm 0.8945\nL2H1 QK_F/top/r90/tr [ 1.9454 0.4582 39. 2.5132] QK_svs [0.458 0.421 0.396 0.393 0.367 0.355] OV_F/tr [ 21.448 -40.225] key_bias_norm 0.6377\nL2H2 QK_F/top/r90/tr [ 2.804 0.7089 31. 1.4941] QK_svs [0.709 0.693 0.676 0.634 0.611 0.598] OV_F/tr [ 16.782 -29.462] key_bias_norm 1.5836\nL2H3 QK_F/top/r90/tr [ 1.9759 0.5708 39. -0.4059] QK_svs [0.571 0.423 0.412 0.392 0.388 0.37 ] OV_F/tr [ 20.617 -38.568] key_bias_norm 1.0368\nL2H4 QK_F/top/r90/tr [ 2.2016 0.7827 37. 0.0551] QK_svs [0.783 0.587 0.484 0.429 0.417 0.401] OV_F/tr [ 18.415 -38.753] key_bias_norm 1.1994\nL2H5 QK_F/top/r90/tr [ 1.6541 0.4194 42. -0.7004] QK_svs [0.419 0.414 0.359 0.32 0.315 0.296] OV_F/tr [ 23.766 -33.94 ] key_bias_norm 0.6361\nL3H0 QK_F/top/r90/tr [ 2.0777 0.4054 48. 1.7527] QK_svs [0.405 0.399 0.385 0.377 0.365 0.359] OV_F/tr [ 24.502 -48.319] key_bias_norm 0.2987\nL3H1 QK_F/top/r90/tr [ 1.6948 0.3223 48. 2.225 ] QK_svs [0.322 0.315 0.309 0.304 0.3 0.295] OV_F/tr [ 25.724 -49.613] key_bias_norm 0.2634\nL3H2 QK_F/top/r90/tr [ 2.0608 0.4337 48. 0.7176] QK_svs [0.434 0.411 0.392 0.362 0.358 0.353] OV_F/tr [ 26.86 -50.565] key_bias_norm 0.3422\nL3H3 QK_F/top/r90/tr [ 1.053 0.2624 47. 0.0028] QK_svs [0.262 0.237 0.208 0.202 0.193 0.191] OV_F/tr [ 37.938 -51.153] key_bias_norm 0.2583\nL3H4 QK_F/top/r90/tr [ 1.7344 0.4007 47. 0.0047] QK_svs [0.401 0.369 0.357 0.323 0.313 0.306] OV_F/tr [ 28.44 -35.013] key_bias_norm 0.3067\nL3H5 QK_F/top/r90/tr [ 2.1169 0.4041 48. 8.0447] QK_svs [0.404 0.39 0.384 0.37 0.365 0.362] OV_F/tr [ 23.687 -56.843] key_bias_norm 0.3101\n\nWithin-layer QK cosine matrices:\nL 0\n[[ 1. 0.179 0.112 0.206 0.063 0.093]\n [ 0.179 1. 0.21 -0.15 0.005 0.414]\n [ 0.112 0.21 1. -0.247 -0.006 0.114]\n [ 0.206 -0.15 -0.247 1. -0.027 -0.098]\n [ 0.063 0.005 -0.006 -0.027 1. -0.004]\n [ 0.093 0.414 0.114 -0.098 -0.004 1. ]]\nL 1\n[[1. 0.302 0.393 0.327 0.338 0.266]\n [0.3"
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 2 of at most 100:
```
Effective linear maps include LayerNorm gain and mean-centering only.
No normalization of inputs, attention, or MLP evaluation is performed.
L0H0 QK_F/top/r90/tr [ 0.1137 0.0887 2. -0.0508] QK_svs [0.089 0.068 0.013 0.008 0.006 0.005] OV_F/tr [4.321 1.666] key_bias_norm 0.0185
L0H1 QK_F/top/r90/tr [ 0.8775 0.6013 6. -0.4868] QK_svs [0.601 0.373 0.297 0.211 0.199 0.155] OV_F/tr [3.488 0.164] key_bias_norm 0.0736
L0H2 QK_F/top/r90/tr [0.1745 0.132 5. 0.0546] QK_svs [0.132 0.08 0.044 0.031 0.028 0.025] OV_F/tr [2.023 0.015] key_bias_norm 0.0234
L0H3 QK_F/top/r90/tr [ 0.2395 0.1645 3. -0.1871] QK_svs [0.165 0.14 0.081 0.031 0.029 0.021] OV_F/tr [1.621 0.242] key_bias_norm 0.057
L0H4 QK_F/top/r90/tr [ 0.0093 0.0045 28. -0.0037] QK_svs [0.004 0.003 0.002 0.002 0.002 0.002] OV_F/tr [17.158 6.049] key_bias_norm 0.0009
L0H5 QK_F/top/r90/tr [ 0.9704 0.4099 16. 1.2135] QK_svs [0.41 0.362 0.273 0.254 0.245 0.222] OV_F/tr [ 3.707 -2.463] key_bias_norm 0.0509
L1H0 QK_F/top/r90/tr [ 0.644 0.3538 13. -0.308 ] QK_svs [0.354 0.252 0.218 0.167 0.158 0.139] OV_F/tr [ 13.436 -16.299] key_bias_norm 0.8392
L1H1 QK_F/top/r90/tr [ 1.5843 1.1456 9. -1.2729] QK_svs [1.146 0.661 0.459 0.326 0.249 0.215] OV_F/tr [ 9.592 -11.205] key_bias_norm 3.1498
L1H2 QK_F/top/r90/tr [ 1.2358 0.8069 8. -0.7401] QK_svs [0.807 0.693 0.332 0.205 0.18 0.159] OV_F/tr [ 11.496 -14.892] key_bias_norm 2.1998
L1H3 QK_F/top/r90/tr [ 1.3094 0.9127 8. -0.6948] QK_svs [0.913 0.699 0.309 0.202 0.178 0.167] OV_F/tr [ 10.043 -10.571] key_bias_norm 2.7194
L1H4 QK_F/top/r90/tr [ 1.2492 0.848 8. -0.5846] QK_svs [0.848 0.654 0.34 0.213 0.191 0.162] OV_F/tr [ 9.557 -10.446] key_bias_norm 2.518
L1H5 QK_F/top/r90/tr [ 1.4614 1.0088 8. -1.2054] QK_svs [1.009 0.555 0.482 0.45 0.242 0.215] OV_F/tr [ 9.826 -13.045] key_bias_norm 2.4602
L2H0 QK_F/top/r90/tr [ 2.0456 0.5184 39. -0.5814] QK_svs [0.518 0.485 0.459 0.416 0.407 0.397] OV_F/tr [ 18.698 -35.97 ] key_bias_norm 0.8945
L2H1 QK_F/top/r90/tr [ 1.9454 0.4582 39. 2.5132] QK_svs [0.458 0.421 0.396 0.393 0.367 0.355] OV_F/tr [ 21.448 -40.225] key_bias_norm 0.6377
L2H2 QK_F/top/r90/tr [ 2.804 0.7089 31. 1.4941] QK_svs [0.709 0.693 0.676 0.634 0.611 0.598] OV_F/tr [ 16.782 -29.462] key_bias_norm 1.5836
L2H3 QK_F/top/r90/tr [ 1.9759 0.5708 39. -0.4059] QK_svs [0.571 0.423 0.412 0.392 0.388 0.37 ] OV_F/tr [ 20.617 -38.568] key_bias_norm 1.0368
L2H4 QK_F/top/r90/tr [ 2.2016 0.7827 37. 0.0551] QK_svs [0.783 0.587 0.484 0.429 0.417 0.401] OV_F/tr [ 18.415 -38.753] key_bias_norm 1.1994
L2H5 QK_F/top/r90/tr [ 1.6541 0.4194 42. -0.7004] QK_svs [0.419 0.414 0.359 0.32 0.315 0.296] OV_F/tr [ 23.766 -33.94 ] key_bias_norm 0.6361
L3H0 QK_F/top/r90/tr [ 2.0777 0.4054 48. 1.7527] QK_svs [0.405 0.399 0.385 0.377 0.365 0.359] OV_F/tr [ 24.502 -48.319] key_bias_norm 0.2987
L3H1 QK_F/top/r90/tr [ 1.6948 0.3223 48. 2.225 ] QK_svs [0.322 0.315 0.309 0.304 0.3 0.295] OV_F/tr [ 25.724 -49.613] key_bias_norm 0.2634
L3H2 QK_F/top/r90/tr [ 2.0608 0.4337 48. 0.7176] QK_svs [0.434 0.411 0.392 0.362 0.358 0.353] OV_F/tr [ 26.86 -50.565] key_bias_norm 0.3422
L3H3 QK_F/top/r90/tr [ 1.053 0.2624 47. 0.0028] QK_svs [0.262 0.237 0.208 0.202 0.193 0.191] OV_F/tr [ 37.938 -51.153] key_bias_norm 0.2583
L3H4 QK_F/top/r90/tr [ 1.7344 0.4007 47. 0.0047] QK_svs [0.401 0.369 0.357 0.323 0.313 0.306] OV_F/tr [ 28.44 -35.013] key_bias_norm 0.3067
L3H5 QK_F/top/r90/tr [ 2.1169 0.4041 48. 8.0447] QK_svs [0.404 0.39 0.384 0.37 0.365 0.362] OV_F/tr [ 23.687 -56.843] key_bias_norm 0.3101
Within-layer QK cosine matrices:
L 0
[[ 1. 0.179 0.112 0.206 0.063 0.093]
[ 0.179 1. 0.21 -0.15 0.005 0.414]
[ 0.112 0.21 1. -0.247 -0.006 0.114]
[ 0.206 -0.15 -0.247 1. -0.027 -0.098]
[ 0.063 0.005 -0.006 -0.027 1. -0.004]
[ 0.093 0.414 0.114 -0.098 -0.004 1. ]]
L 1
[[1. 0.302 0.393 0.327 0.338 0.266]
[0.302 1. 0.524 0.591 0.568 0.566]
[0.393 0.524 1. 0.538 0.578 0.45 ]
[0.327 0.591 0.538 1. 0.607 0.524]
[0.338 0.568 0.578 0.607 1. 0.479]
[0.266 0.566 0.45 0.524 0.479 1. ]]
L 2
[[1. 0.034 0.061 0.061 0.074 0.046]
[0.034 1. 0.049 0.03 0.035 0.031]
[0.061 0.049 1. 0.095 0.1 0.03 ]
[0.061 0.03 0.095 1. 0.076 0.045]
[0.074 0.035 0.1 0.076 1. 0.046]
[0.046 0.031 0.03 0.045 0.046 1. ]]
L 3
[[1. 0.084 0.044 0.045 0.042 0.073]
[0.084 1. 0.013 0.044 0.032 0.192]
[0.044 0.013 1. 0.038 0.053 0.021]
[0.045 0.044 0.038 1. 0.054 0.035]
[0.042 0.032 0.053 0.054 1. 0.037]
[0.073 0.192 0.021 0.035 0.037 1. ]]
Key-composition trace cosine: trace(QK_later @ OV_earlier)/(norms)
L1H0 highest [('L0H4', 0.0207), ('L0H0', 0.0065), ('L0H3', 0.005), ('L0H1', -0.0001)] lowest [('L0H5', -0.0034), ('L0H2', -0.0079)]
L1H1 highest [('L0H4', 0.0191), ('L0H5', 0.0154), ('L0H1', 0.007), ('L0H3', 0.0055)] lowest [('L0H0', -0.0046), ('L0H2', -0.0161)]
L1H2 highest [('L0H4', 0.0326), ('L0H3', 0.0092), ('L0H0', 0.0039), ('L0H1', 0.0007)] lowest [('L0H5', -0.0027), ('L0H2', -0.0168)]
L1H3 highest [('L0H4', 0.0421), ('L0H5', 0.0124), ('L0H0', 0.0077), ('L0H3', 0.0063)] lowest [('L0H1', -0.0026), ('L0H2', -0.0132)]
L1H4 highest [('L0H4', 0.0439), ('L0H3', 0.0114), ('L0H5', 0.0094), ('L0H0', 0.0093)] lowest [('L0H1', 0.0022), ('L0H2', -0.0181)]
L1H5 highest [('L0H5', 0.0128), ('L0H1', 0.0045), ('L0H3', -0.0004), ('L0H4', -0.0036)] lowest [('L0H2', -0.0134), ('L0H0', -0.0195)]
L2H0 highest [('L0H4', 0.0099), ('L0H3', 0.0078), ('L1H5', 0.0057), ('L0H0', 0.0049)] lowest [('L0H5', -0.0046), ('L1H0', -0.0126)]
L2H1 highest [('L0H5', 0.0776), ('L1H3', 0.0575), ('L1H2', 0.0482), ('L1H1', 0.0479)] lowest [('L0H3', 0.0025), ('L0H1', 0.0014)]
L2H2 highest [('L0H4', 0.02), ('L0H0', 0.0114), ('L0H3', 0.0015), ('L0H2', -0.0054)] lowest [('L1H0', -0.0418), ('L1H2', -0.0434)]
L2H3 highest [('L0H4', 0.0396), ('L0H0', 0.0177), ('L0H3', 0.0066), ('L1H1', 0.0049)] lowest [('L1H2', -0.0162), ('L1H0', -0.0197)]
L2H4 highest [('L0H4', 0.0246), ('L0H3', 0.0101), ('L0H0', 0.0079), ('L0H5', 0.0018)] lowest [('L1H2', -0.0115), ('L1H0', -0.0139)]
L2H5 highest [('L0H4', 0.0188), ('L1H2', 0.0095), ('L0H0', 0.009), ('L1H1', 0.0083)] lowest [('L1H0', 0.0001), ('L0H2', 0.0001)]
L3H0 highest [('L2H2', 0.0238), ('L0H5', 0.019), ('L1H2', 0.0101), ('L0H0', 0.0098)] lowest [('L2H5', -0.0088), ('L2H1', -0.0135)]
L3H1 highest [('L2H2', 0.1104), ('L0H5', 0.061), ('L1H4', 0.0456), ('L1H3', 0.0453)] lowest [('L2H5', -0.0002), ('L2H1', -0.0181)]
L3H2 highest [('L0H4', 0.013), ('L0H0', 0.0062), ('L0H2', 0.0015), ('L1H1', 0.0013)] lowest [('L1H2', -0.0092), ('L1H0', -0.0116)]
L3H3 highest [('L2H4', 0.0148), ('L2H3', 0.0103), ('L2H2', 0.009), ('L1H2', 0.0079)] lowest [('L1H3', -0.0004), ('L2H1', -0.0054)]
L3H4 highest [('L0H4', 0.0119), ('L2H4', 0.0103), ('L0H5', 0.0101), ('L0H0', 0.0085)] lowest [('L0H1', -0.0063), ('L2H1', -0.0083)]
L3H5 highest [('L2H2', 0.1836), ('L0H5', 0.0713), ('L1H3', 0.0683), ('L1H4', 0.0572)] lowest [('L2H5', -0.022), ('L2H1', -0.0599)]
L0 gain extremes and corresponding linear-map column norms:
365 g1/g2 0.0165 1.1172 q/k/v/fc [1.049 1.054 0.975 3.402]
187 g1/g2 0.0226 0.7165 q/k/v/fc [1.132 0.967 0.916 2.7 ]
55 g1/g2 0.0253 0.9217 q/k/v/fc [0.809 0.841 1.062 2.515]
24 g1/g2 0.0305 0.589 q/k/v/fc [1.035 0.996 0.845 2.668]
357 g1/g2 0.0323 1.1837 q/k/v/fc [0.925 0.895 1.155 3.137]
347 g1/g2 0.0337 1.1467 q/k/v/fc [0.889 0.975 1.032 2.508]
3 g1/g2 0.0358 0.917 q/k/v/fc [1.091 0.915 1.034 2.987]
248 g1/g2 0.041 1.1464 q/k/v/fc [0.889 0.892 1.053 3.335]
255 g1/g2 0.0412 1.0276 q/k/v/fc [1.056 1.148 1.033 2.583]
2 g1/g2 0.0455 1.4738 q/k/v/fc [0.75 0.743 1.054 3.661]
181 g1/g2 0.0456 0.862 q/k/v/fc [1.048 0.988 1.087 2.82 ]
92 g1/g2 0.0462 0.7997 q/k/v/fc [1.095 0.954 0.978 2.566]
184 g1/g2 0.8285 0.3596 q/k/v/fc [1.128 1.39 0.773 2.415]
359 g1/g2 0.8331 0.5308 q/k/v/fc [1.102 1.905 1.086 2.658]
307 g1/g2 0.8441 0.6557 q/k/v/fc [1.286 1.432 0.785 2.01 ]
48 g1/g2 0.8725 0.3104 q/k/v/fc [1.35 2.011 0.849 2.286]
228 g1/g2 0.9005 0.642 q/k/v/fc [1.07 2.112 1.049 2.281]
60 g1/g2 0.9031 0.4939 q/k/v/fc [1.232 2.132 1.076 2.338]
162 g1/g2 0.9129 0.4061 q/k/v/fc [1.295 2.002 0.951 2.428]
50 g1/g2 0.9335 0.5591 q/k/v/fc [1.225 1.803 0.924 2.606]
180 g1/g2 0.9483 0.5096 q/k/v/fc [1.255 2.308 0.843 2.416]
223 g1/g2 0.9731 0.3425 q/k/v/fc [1.198 2.437 1.17 2.354]
273 g1/g2 0.9813 0.4232 q/k/v/fc [1.273 2.123 1.29 1.452]
211 g1/g2 1.0134 0.5536 q/k/v/fc [1.119 2.735 1.166 2.481]
Linear MLP weight products (not MLP evaluations):
0 Frob/trace 203.535 89.182 eig real min/max -7.911 16.912
1 Frob/trace 173.124 -322.781 eig real min/max -14.512 17.842
2 Frob/trace 202.559 -791.202 eig real min/max -21.395 8.292
3 Frob/trace 232.961 -943.792 eig real min/max -22.338 17.566
```Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
from sklearn.cluster import KMeans
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T=lambda a: torch.as_tensor(np.asarray(a), dtype=torch.float64)
P=torch.eye(384)-torch.ones((384,384))/384
M,B={},{}
for l in range(4):
g=T(W[f"ln1_w.{l}"])
for j in range(6):
s=slice(j*64,(j+1)*64)
q,k,v=[(T(W[f"{n}.{l}"])[s]*g)@P for n in ["W_q","W_k","W_v"]]
o=P@T(W[f"W_o.{l}"])[:,s]
M[l,j]=q.T@k/8
B[l,j]=o@v
print("L0 effective OV singular spectra (first 32):")
for j in range(6):
print(j,np.round(torch.linalg.svdvals(B[0,j])[:32].numpy(),3))
print("\nL0H5 symmetric-QK eigenvalues, descending:")
e=torch.linalg.eigvalsh((M[0,5]+M[0,5].T)/2).flip(0)
print(np.round(e[:45].numpy(),4))
print("Smallest:",np.round(e[-15:].numpy(),4))
print("\nQK energy reading leading 3 OUTPUT directions of each L0 OV:")
U={}
for j in range(6):
U[j]=torch.linalg.svd(B[0,j],full_matrices=False).U[:,:3]
for l in range(1,4):
for j in range(6):
a=M[l,j]
print(f"L{l}H{j}",
"query",np.round([float((u.T@a).norm().square()/a.norm().square())
for u in U.values()],3),
"key",np.round([float((a@u).norm().square()/a.norm().square())
for u in U.values()],3))
print("\nSelected key-composition spectra, on weight products only:")
for later,earlier in [((2,1),(0,5)),((3,1),(2,2)),((3,5),(2,2)),
((1,4),(0,4)),((2,2),(0,1)),((2,2),(1,2))]:
a=M[later]@B[earlier]
ev=torch.linalg.eigvals(a)
se=torch.linalg.eigvalsh((a+a.T)/2)
sv=torch.linalg.svdvals(a)
print(later,earlier,"trace/Frob/nuclear",
np.round([float(torch.trace(a)),float(a.norm()),float(sv.sum())],4),
"sym_positive_fraction",round(float(se.clamp(min=0).sum()/se.abs().sum()),4),
"eig_top_real",np.round(torch.sort(ev.real,descending=True).values[:12].numpy(),4))
print("\nFirst MLP row-weight geometry:")
f=(T(W["W_fc.0"])*T(W["ln2_w.0"]))@P
u,s,vh=torch.linalg.svd(f,full_matrices=False)
print("effective FC0 singular values (first 65)",np.round(s[:65].numpy(),3))
# Whitening and clustering weight rows, not evaluating neurons.
for r in [24,32,48]:
z=u[:,:r].numpy()
z=z/(np.linalg.norm(z,axis=1,keepdims=True)+1e-12)
for k in [20,24,28,32,40]:
km=KMeans(n_clusters=k,n_init=5,random_state=0).fit(z)
centers=km.cluster_centers_
norms=np.linalg.norm(centers,axis=1)
counts=np.bincount(km.labels_,minlength=k)
print("rank",r,"k",k,"mean within-cluster cosine",
round(float(np.average(norms,weights=counts)),3),
"coherent clusters (centroid norm > .65)",
sorted([(int(counts[i]),round(float(norms[i]),3))
for i in range(k) if norms[i]>.65],reverse=True))
```
{
"t": 1792713406.669928,
"s": 21.92,
"code": "import numpy as np\nimport torch\nfrom sklearn.cluster import KMeans\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT=lambda a: torch.as_tensor(np.asarray(a), dtype=torch.float64)\nP=torch.eye(384)-torch.ones((384,384))/384\nM,B={},{}\n\nfor l in range(4):\n g=T(W[f\"ln1_w.{l}\"])\n for j in range(6):\n s=slice(j*64,(j+1)*64)\n q,k,v=[(T(W[f\"{n}.{l}\"])[s]*g)@P for n in [\"W_q\",\"W_k\",\"W_v\"]]\n o=P@T(W[f\"W_o.{l}\"])[:,s]\n M[l,j]=q.T@k/8\n B[l,j]=o@v\n\nprint(\"L0 effective OV singular spectra (first 32):\")\nfor j in range(6):\n print(j,np.round(torch.linalg.svdvals(B[0,j])[:32].numpy(),3))\nprint(\"\\nL0H5 symmetric-QK eigenvalues, descending:\")\ne=torch.linalg.eigvalsh((M[0,5]+M[0,5].T)/2).flip(0)\nprint(np.round(e[:45].numpy(),4))\nprint(\"Smallest:\",np.round(e[-15:].numpy(),4))\n\nprint(\"\\nQK energy reading leading 3 OUTPUT directions of each L0 OV:\")\nU={}\nfor j in range(6):\n U[j]=torch.linalg.svd(B[0,j],full_matrices=False).U[:,:3]\nfor l in range(1,4):\n for j in range(6):\n a=M[l,j]\n print(f\"L{l}H{j}\",\n \"query\",np.round([float((u.T@a).norm().square()/a.norm().square())\n for u in U.values()],3),\n \"key\",np.round([float((a@u).norm().square()/a.norm().square())\n for u in U.values()],3))\n\nprint(\"\\nSelected key-composition spectra, on weight products only:\")\nfor later,earlier in [((2,1),(0,5)),((3,1),(2,2)),((3,5),(2,2)),\n ((1,4),(0,4)),((2,2),(0,1)),((2,2),(1,2))]:\n a=M[later]@B[earlier]\n ev=torch.linalg.eigvals(a)\n se=torch.linalg.eigvalsh((a+a.T)/2)\n sv=torch.linalg.svdvals(a)\n print(later,earlier,\"trace/Frob/nuclear\",\n np.round([float(torch.trace(a)),float(a.norm()),float(sv.sum())],4),\n \"sym_positive_fraction\",round(float(se.clamp(min=0).sum()/se.abs().sum()),4),\n \"eig_top_real\",np.round(torch.sort(ev.real,descending=True).values[:12].numpy(),4))\n\nprint(\"\\nFirst MLP row-weight geometry:\")\nf=(T(W[\"W_fc.0\"])*T(W[\"ln2_w.0\"]))@P\nu,s,vh=torch.linalg.svd(f,full_matrices=False)\nprint(\"effective FC0 singular values (first 65)\",np.round(s[:65].numpy(),3))\n# Whitening and clustering weight rows, not evaluating neurons.\nfor r in [24,32,48]:\n z=u[:,:r].numpy()\n z=z/(np.linalg.norm(z,axis=1,keepdims=True)+1e-12)\n for k in [20,24,28,32,40]:\n km=KMeans(n_clusters=k,n_init=5,random_state=0).fit(z)\n centers=km.cluster_centers_\n norms=np.linalg.norm(centers,axis=1)\n counts=np.bincount(km.labels_,minlength=k)\n print(\"rank\",r,\"k\",k,\"mean within-cluster cosine\",\n round(float(np.average(norms,weights=counts)),3),\n \"coherent clusters (centroid norm > .65)\",\n sorted([(int(counts[i]),round(float(norms[i]),3))\n for i in range(k) if norms[i]>.65],reverse=True))\n",
"out": "L0 effective OV singular spectra (first 32):\n0 [3.424 2.469 0.529 0.401 0.346 0.25 0.205 0.199 0.178 0.135 0.114 0.101 0.094 0.086 0.079 0.073 0.07 0.06 0.057 0.053 0.051 0.048 0.046 0.042\n 0.041 0.039 0.037 0.035 0.033 0.03 0.029 0.027]\n1 [0.996 0.923 0.89 0.862 0.836 0.815 0.803 0.767 0.731 0.71 0.698 0.669 0.648 0.64 0.626 0.6 0.586 0.572 0.521 0.517 0.492 0.47 0.441 0.358\n 0.349 0.311 0.299 0.257 0.247 0.208 0.201 0.172]\n2 [1.055 0.901 0.807 0.66 0.537 0.47 0.373 0.347 0.277 0.236 0.21 0.163 0.133 0.12 0.105 0.097 0.089 0.081 0.073 0.068 0.061 0.058 0.053 0.051\n 0.048 0.041 0.038 0.036 0.035 0.034 0.033 0.029]\n3 [1.025 0.558 0.459 0.438 0.353 0.341 0.325 0.292 0.277 0.259 0.232 0.213 0.185 0.15 0.142 0.132 0.126 0.11 0.097 0.093 0.079 0.077 0.067 0.062\n 0.056 0.055 0.047 0.045 0.043 0.038 0.036 0.036]\n4 [13.877 9.644 2.507 0.968 0.594 0.554 0.483 0.433 0.345 0.295 0.28 0.228 0.205 0.187 0.16 0.126 0.111 0.091 0.079 0.074 0.069\n 0.068 0.063 0.061 0.05 0.047 0.044 0.042 0.039 0.038 0.033 0.033]\n5 [1.059 1.044 0.96 0.948 0.909 0.899 0.855 0.836 0.817 0.77 0.753 0.731 0.709 0.676 0.656 0.646 0.604 0.552 0.533 0.454 0.42 0.346 0.332 0.307\n 0.299 0.292 0.274 0.259 0.237 0.227 0.223 0.216]\n\nL0H5 symmetric-QK eigenvalues, descending:\n[0.2734 0.2195 0.2099 0.1954 0.1774 0.1566 0.1421 0.1202 0.1088 0.1023 0.0519 0.0495 0.0375 0.0297 0.029 0.0158 0.0133 0.0126 0.0107 0.0101 0.0092\n 0.0083 0.0074 0.0065 0.0058 0.005 0.0048 0.0038 0.0034 0.0033 0.0031 0.0025 0.0024 0.0021 0.002 0.0019 0.0016 0.0015 0.0013 0.0012 0.0012 0.0011\n 0.0009 0.0009 0.0009]\nSmallest: [-0.0174 -0.0197 -0.0224 -0.0268 -0.031 -0.0334 -0.0414 -0.0426 -0.046 -0.0496 -0.0502 -0.0549 -0.0642 -0.0776 -0.0826]\n\nQK energy reading leading 3 OUTPUT directions of each L0 OV:\nL1H0 query [0.043 0.021 0.035 0.022 0.061 0.015] key [0.272 0.012 0.012 0.051 0.342 0.012]\nL1H1 query [0.071 0.008 0.008 0.024 0.107 0.011] key [0.521 0.002 0.017 0.07 0.627 0.008]\nL1H2 query [0.063 0.009 0.009 0.027 0.108 0.014] key [0.489 0.003 0.017 0.072 0.631 0.009]\nL1H3 query [0.08 0.004 0.012 0.041 0.137 0.008] key [0.546 0.003 0.023 0.085 0.678 0.007]\nL1H4 query [0.075 0.006 0.008 0.031 0.119 0.019] key [0.516 0.003 0.024 0.079 0.645 0.013]\nL1H5 query [0.05 0.007 0.016 0.019 0.084 0.032] key [0.459 0.003 0.014 0.051 0.525 0.017]\nL2H0 query [0.009 0.008 0.009 0.008 0.01 0.01 ] key [0.046 0.009 0.011 0.012 0.051 0.014]\nL2H1 query [0.008 0.007 0.012 0.006 0.005 0.019] key [0.007 0.012 0.011 0.005 0.005 0.02 ]\nL2H2 query [0.013 0.007 0.006 0.01 0.014 0.023] key [0.057 0.006 0.01 0.015 0.07 0.019]\nL2H3 query [0.018 0.009 0.009 0.01 0.022 0.008] key [0.077 0.008 0.01 0.02 0.095 0.013]\nL2H4 query [0.018 0.008 0.006 0.012 0.023 0.009] key [0.079 0.006 0.027 0.02 0.095 0.011]\nL2H5 query [0.009 0.007 0.016 0.007 0.006 0.008] key [0.039 0.007 0.012 0.009 0.041 0.009]\nL3H0 query [0.004 0.006 0.005 0.005 0.002 0.01 ] key [0.016 0.007 0.011 0.006 0.017 0.01 ]\nL3H1 query [0.005 0.005 0.008 0.007 0.003 0.011] key [0.01 0.008 0.01 0.006 0.009 0.016]\nL3H2 query [0.004 0.007 0.01 0.006 0.003 0.01 ] key [0.022 0.007 0.01 0.007 0.025 0.01 ]\nL3H3 query [0.006 0.007 0.013 0.008 0.005 0.005] key [0.01 0.006 0.01 0.006 0.01 0.008]\nL3H4 query [0.01 0.007 0.012 0.006 0.004 0.006] key [0.025 0.008 0.01 0.007 0.025 0.008]\nL3H5 query [0.005 0.01 0.007 0.006 0.003 0.022] key [0.006 0.015 0.01 0.006 0.005 0.023]\n\nSelected key-composition spectra, on weight products only:\n(2, 1) (0, 5) trace/Frob/nuclear [0.5595 0.549 2.6366] sym_positive_fraction 0.6084 eig_top_real [0.0559 0.0559 0.0505 0.0501 0.0501 0.0379 0.0379 0.0331 0.0331 0.0242 0.0242 0.0225]\n(3, 1) (2, 2) trace/Frob/nuclear [ 3.1409 1.7435 10.9188] sym_positive_fraction 0.648 eig_top_real [0.2974 0.1824 0.1824 0.1799 0.1725 0.1725 0.116 0.1148 0.1148 0.1126 0.1126 0.0976]\n(3, 5) (2, 2) trace/Frob/nuclear [ 6.521 2.6779 16.7035] sym_positive_fraction 0.7003 eig_top_real [0.4 0.4 "
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 3 of at most 100: ``` L0 effective OV singular spectra (first 32): 0 [3.424 2.469 0.529 0.401 0.346 0.25 0.205 0.199 0.178 0.135 0.114 0.101 0.094 0.086 0.079 0.073 0.07 0.06 0.057 0.053 0.051 0.048 0.046 0.042 0.041 0.039 0.037 0.035 0.033 0.03 0.029 0.027] 1 [0.996 0.923 0.89 0.862 0.836 0.815 0.803 0.767 0.731 0.71 0.698 0.669 0.648 0.64 0.626 0.6 0.586 0.572 0.521 0.517 0.492 0.47 0.441 0.358 0.349 0.311 0.299 0.257 0.247 0.208 0.201 0.172] 2 [1.055 0.901 0.807 0.66 0.537 0.47 0.373 0.347 0.277 0.236 0.21 0.163 0.133 0.12 0.105 0.097 0.089 0.081 0.073 0.068 0.061 0.058 0.053 0.051 0.048 0.041 0.038 0.036 0.035 0.034 0.033 0.029] 3 [1.025 0.558 0.459 0.438 0.353 0.341 0.325 0.292 0.277 0.259 0.232 0.213 0.185 0.15 0.142 0.132 0.126 0.11 0.097 0.093 0.079 0.077 0.067 0.062 0.056 0.055 0.047 0.045 0.043 0.038 0.036 0.036] 4 [13.877 9.644 2.507 0.968 0.594 0.554 0.483 0.433 0.345 0.295 0.28 0.228 0.205 0.187 0.16 0.126 0.111 0.091 0.079 0.074 0.069 0.068 0.063 0.061 0.05 0.047 0.044 0.042 0.039 0.038 0.033 0.033] 5 [1.059 1.044 0.96 0.948 0.909 0.899 0.855 0.836 0.817 0.77 0.753 0.731 0.709 0.676 0.656 0.646 0.604 0.552 0.533 0.454 0.42 0.346 0.332 0.307 0.299 0.292 0.274 0.259 0.237 0.227 0.223 0.216] L0H5 symmetric-QK eigenvalues, descending: [0.2734 0.2195 0.2099 0.1954 0.1774 0.1566 0.1421 0.1202 0.1088 0.1023 0.0519 0.0495 0.0375 0.0297 0.029 0.0158 0.0133 0.0126 0.0107 0.0101 0.0092 0.0083 0.0074 0.0065 0.0058 0.005 0.0048 0.0038 0.0034 0.0033 0.0031 0.0025 0.0024 0.0021 0.002 0.0019 0.0016 0.0015 0.0013 0.0012 0.0012 0.0011 0.0009 0.0009 0.0009] Smallest: [-0.0174 -0.0197 -0.0224 -0.0268 -0.031 -0.0334 -0.0414 -0.0426 -0.046 -0.0496 -0.0502 -0.0549 -0.0642 -0.0776 -0.0826] QK energy reading leading 3 OUTPUT directions of each L0 OV: L1H0 query [0.043 0.021 0.035 0.022 0.061 0.015] key [0.272 0.012 0.012 0.051 0.342 0.012] L1H1 query [0.071 0.008 0.008 0.024 0.107 0.011] key [0.521 0.002 0.017 0.07 0.627 0.008] L1H2 query [0.063 0.009 0.009 0.027 0.108 0.014] key [0.489 0.003 0.017 0.072 0.631 0.009] L1H3 query [0.08 0.004 0.012 0.041 0.137 0.008] key [0.546 0.003 0.023 0.085 0.678 0.007] L1H4 query [0.075 0.006 0.008 0.031 0.119 0.019] key [0.516 0.003 0.024 0.079 0.645 0.013] L1H5 query [0.05 0.007 0.016 0.019 0.084 0.032] key [0.459 0.003 0.014 0.051 0.525 0.017] L2H0 query [0.009 0.008 0.009 0.008 0.01 0.01 ] key [0.046 0.009 0.011 0.012 0.051 0.014] L2H1 query [0.008 0.007 0.012 0.006 0.005 0.019] key [0.007 0.012 0.011 0.005 0.005 0.02 ] L2H2 query [0.013 0.007 0.006 0.01 0.014 0.023] key [0.057 0.006 0.01 0.015 0.07 0.019] L2H3 query [0.018 0.009 0.009 0.01 0.022 0.008] key [0.077 0.008 0.01 0.02 0.095 0.013] L2H4 query [0.018 0.008 0.006 0.012 0.023 0.009] key [0.079 0.006 0.027 0.02 0.095 0.011] L2H5 query [0.009 0.007 0.016 0.007 0.006 0.008] key [0.039 0.007 0.012 0.009 0.041 0.009] L3H0 query [0.004 0.006 0.005 0.005 0.002 0.01 ] key [0.016 0.007 0.011 0.006 0.017 0.01 ] L3H1 query [0.005 0.005 0.008 0.007 0.003 0.011] key [0.01 0.008 0.01 0.006 0.009 0.016] L3H2 query [0.004 0.007 0.01 0.006 0.003 0.01 ] key [0.022 0.007 0.01 0.007 0.025 0.01 ] L3H3 query [0.006 0.007 0.013 0.008 0.005 0.005] key [0.01 0.006 0.01 0.006 0.01 0.008] L3H4 query [0.01 0.007 0.012 0.006 0.004 0.006] key [0.025 0.008 0.01 0.007 0.025 0.008] L3H5 query [0.005 0.01 0.007 0.006 0.003 0.022] key [0.006 0.015 0.01 0.006 0.005 0.023] Selected key-composition spectra, on weight products only: (2, 1) (0, 5) trace/Frob/nuclear [0.5595 0.549 2.6366] sym_positive_fraction 0.6084 eig_top_real [0.0559 0.0559 0.0505 0.0501 0.0501 0.0379 0.0379 0.0331 0.0331 0.0242 0.0242 0.0225] (3, 1) (2, 2) trace/Frob/nuclear [ 3.1409 1.7435 10.9188] sym_positive_fraction 0.648 eig_top_real [0.2974 0.1824 0.1824 0.1799 0.1725 0.1725 0.116 0.1148 0.1148 0.1126 0.1126 0.0976] (3, 5) (2, 2) trace/Frob/nuclear [ 6.521 2.6779 16.7035] sym_positive_fraction 0.7003 eig_top_real [0.4 0.4 0.3195 0.3017 0.3017 0.2888 0.2888 0.2559 0.2559 0.2349 0.2349 0.2053] (1, 4) (0, 4) trace/Frob/nuclear [ 0.9408 11.2992 15.1822] sym_positive_fraction 0.5311 eig_top_real [0.4559 0.4559 0.0134 0.0076 0.0027 0.0022 0.0012 0.0012 0.0011 0.0008 0.0008 0.0004] (2, 2) (0, 1) trace/Frob/nuclear [-0.1618 0.4631 2.2341] sym_positive_fraction 0.4629 eig_top_real [0.0151 0.0102 0.0102 0.0069 0.0051 0.0051 0.0033 0.0033 0.0029 0.0029 0.002 0.002 ] (2, 2) (1, 2) trace/Frob/nuclear [-1.3976 1.8914 8.8391] sym_positive_fraction 0.4179 eig_top_real [0.0407 0.0407 0.0373 0.0326 0.0231 0.0231 0.015 0.0129 0.0129 0.0095 0.003 0.0027] First MLP row-weight geometry: effective FC0 singular values (first 65) [18.977 11.828 8.576 7.711 7.631 7.407 6.879 6.768 6.576 6.508 6.417 6.323 6.248 6.176 6.133 5.979 5.93 5.865 5.827 5.775 5.645 5.615 5.569 5.542 5.484 5.42 5.309 5.253 5.241 5.139 5.057 4.954 4.882 4.82 4.734 4.699 4.579 4.516 4.5 4.453 4.371 4.313 4.265 4.191 4.127 4.076 3.998 3.855 3.826 3.801 3.685 3.646 3.552 3.404 3.33 3.236 3.214 3.157 3.063 3.037 2.926 2.812 2.752 2.742 2.7 ] rank 24 k 20 mean within-cluster cosine 0.682 coherent clusters (centroid norm > .65) [(166, 0.758), (133, 0.767), (132, 0.733), (122, 0.746), (118, 0.734), (115, 0.74), (109, 0.767), (107, 0.716), (30, 0.663), (28, 0.773), (23, 0.75)] rank 24 k 24 mean within-cluster cosine 0.702 coherent clusters (centroid norm > .65) [(163, 0.771), (132, 0.746), (128, 0.781), (121, 0.77), (118, 0.746), (109, 0.737), (106, 0.753), (91, 0.691), (45, 0.713), (31, 0.654), (30, 0.673), (30, 0.671), (30, 0.662), (29, 0.663), (25, 0.692), (24, 0.685)] rank 24 k 28 mean within-cluster cosine 0.714 coherent clusters (centroid norm > .65) [(152, 0.75), (145, 0.786), (140, 0.777), (117, 0.74), (116, 0.757), (116, 0.739), (91, 0.744), (69, 0.67), (49, 0.719), (33, 0.732), (27, 0.709), (26, 0.688), (24, 0.708), (22, 0.733), (22, 0.723), (22, 0.679), (21, 0.756), (20, 0.697), (19, 0.783), (19, 0.699)] rank 24 k 32 mean within-cluster cosine 0.726 coherent clusters (centroid norm > .65) [(117, 0.781), (105, 0.781), (104, 0.769), (102, 0.759), (91, 0.771), (90, 0.771), (86, 0.767), (80, 0.737), (79, 0.756), (70, 0.72), (67, 0.723), (30, 0.749), (29, 0.676), (26, 0.695), (25, 0.671), (24, 0.714), (22, 0.742), (22, 0.731), (22, 0.722), (22, 0.684), (20, 0.769), (20, 0.685), (18, 0.759), (18, 0.695), (17, 0.782)] rank 24 k 40 mean within-cluster cosine 0.743 coherent clusters (centroid norm > .65) [(80, 0.79), (79, 0.791), (79, 0.777), (79, 0.771), (71, 0.794), (69, 0.756), (68, 0.788), (68, 0.784), (65, 0.771), (59, 0.722), (58, 0.758), (54, 0.778), (53, 0.76), (49, 0.767), (48, 0.765), (27, 0.692), (27, 0.657), (26, 0.702), (26, 0.661), (25, 0.67), (23, 0.736), (23, 0.666), (22, 0.716), (22, 0.711), (21, 0.815), (21, 0.741), (21, 0.704), (20, 0.764), (19, 0.789), (19, 0.745), (19, 0.68), (17, 0.71), (17, 0.657), (15, 0.745), (14, 0.783)] rank 32 k 20 mean within-cluster cosine 0.626 coherent clusters (centroid norm > .65) [(189, 0.694), (176, 0.706), (164, 0.687), (152, 0.681), (132, 0.679), (112, 0.661), (33, 0.705)] rank 32 k 24 mean within-cluster cosine 0.641 coherent clusters (centroid norm > .65) [(141, 0.719), (129, 0.705), (124, 0.68), (121, 0.691), (118, 0.707), (115, 0.693), (90, 0.653), (84, 0.654), (45, 0.655), (32, 0.683), (30, 0.718), (27, 0.659), (21, 0.714)] rank 32 k 28 mean within-cluster cosine 0.657 coherent clusters (centroid norm > .65) [(142, 0.716), (142, 0.693), (118, 0.707), (99, 0.715), (99, 0.704), (95, 0.685), (91, 0.689), (85, 0.675), (60, 0.657), (46, 0.656), (25, 0.669), (22, 0.703), (21, 0.713), (21, 0.691), (20, 0.752), (19, 0.708), (18, 0.723), (13, 0.714)] rank 32 k 32 mean within-cluster cosine 0.67 coherent clusters (centroid norm > .65) [(119, 0.72), (106, 0.723), (96, 0.719), (93, 0.711), (92, 0.689), (89, 0.69), (88, 0.691), (79, 0.683), (73, 0.703), (73, 0.683), (58, 0.683), (39, 0.673), (21, 0.703), (21, 0.691), (21, 0.678), (20, 0.752), (20, 0.73), (19, 0.733), (18, 0.717), (17, 0.733), (13, 0.714)] rank 32 k 40 mean within-cluster cosine 0.692 coherent clusters (centroid norm > .65) [(97, 0.745), (94, 0.749), (94, 0.702), (91, 0.712), (80, 0.725), (77, 0.705), (77, 0.678), (75, 0.712), (72, 0.693), (64, 0.705), (53, 0.688), (51, 0.689), (30, 0.672), (25, 0.756), (25, 0.661), (24, 0.685), (22, 0.69), (22, 0.66), (21, 0.709), (21, 0.692), (20, 0.733), (20, 0.682), (19, 0.744), (17, 0.729), (16, 0.748), (14, 0.814), (14, 0.747), (14, 0.725), (13, 0.736), (10, 0.766)] rank 48 k 20 mean within-cluster cosine 0.532 coherent clusters (centroid norm > .65) [] rank 48 k 24 mean within-cluster cosine 0.557 coherent clusters (centroid norm > .65) [] rank 48 k 28 mean within-cluster cosine 0.566 coherent clusters (centroid norm > .65) [(18, 0.675)] rank 48 k 32 mean within-cluster cosine 0.579 coherent clusters (centroid norm > .65) [(73, 0.65), (20, 0.66), (19, 0.664)] rank 48 k 40 mean within-cluster cosine 0.601 coherent clusters (centroid norm > .65) [(78, 0.655), (51, 0.663), (21, 0.683), (20, 0.651), (15, 0.736), (14, 0.696)] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)
P=torch.eye(384)-torch.ones((384,384))/384
M,B={},{}
for l in range(4):
g=T(W[f"ln1_w.{l}"])
for j in range(6):
s=slice(64*j,64*(j+1))
q,k,v=[(T(W[f"{n}.{l}"])[s]*g)@P for n in ["W_q","W_k","W_v"]]
M[l,j]=q.T@k/8
B[l,j]=P@T(W[f"W_o.{l}"])[:,s]@v
print("QK spectral structure:")
for l in range(4):
for j in range(6):
a=M[l,j]
u,s,vh=torch.linalg.svd(a)
ev=torch.linalg.eigvals(a)
ev=ev[torch.argsort(ev.abs(),descending=True)[:8]]
overlaps=torch.linalg.svdvals(u[:,:6].T@vh[:6].T)
print(f"L{l}H{j}",
"symmetric energy",round(float(((a+a.T)/2).norm().square()/a.norm().square()),3),
"top6 Q/K-subspace overlaps",np.round(overlaps.numpy(),2),
"eigenvalues",[(round(float(z.real),3),round(float(z.imag),3)) for z in ev])
u,s,vh=torch.linalg.svd(B[0,4])
r=vh[:3].T
print("\nL0H4 OV top-3 INPUT subspace and other L0 QK maps:")
for j in range(6):
a=M[0,j]
print(j,"query/key energy",
np.round([float((r.T@a).norm().square()/a.norm().square()),
float((a@r).norm().square()/a.norm().square())],4),
"3x3 restriction",np.round((r.T@a@r).numpy(),4))
print("\nDominant effective L0 OV input-subspace similarities (3 dims):")
rs=[torch.linalg.svd(B[0,j]).Vh[:3].T for j in range(6)]
print(np.round([[float((x.T@y).square().sum()/3) for y in rs] for x in rs],3))
print("\nJoint QK covariance spectra:")
for l in range(4):
c=sum(a.T@a+a@a.T for (ll,j),a in M.items() if ll==l)
e=torch.linalg.eigvalsh(c).flip(0)
print(l,np.round((e[:45]/e.sum()).numpy(),4))
print("\nL0 FC input and projection output mean directions:")
f=(T(W["W_fc.0"])*T(W["ln2_w.0"]))@P
o=P@T(W["W_proj.0"])
fn=f/f.norm(dim=1,keepdim=True)
on=o/o.norm(dim=0,keepdim=True)
print("FC row norm quantiles",np.round(np.quantile(f.norm(dim=1).numpy(),[0,.1,.25,.5,.75,.9,1]),3))
print("FC nearest-other-row cosine quantiles",
np.round(np.quantile((fn@fn.T-torch.eye(1536)*2).max(dim=1).values.numpy(),
[0,.1,.25,.5,.75,.9,1]),3))
print("Per-neuron paired input/output cosine quantiles",
np.round(np.quantile((fn*on.T).sum(1).numpy(),[0,.1,.25,.5,.75,.9,1]),3))
```
{
"t": 1792713495.932729,
"s": 12.03,
"code": "import numpy as np\nimport torch\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)\nP=torch.eye(384)-torch.ones((384,384))/384\nM,B={},{}\n\nfor l in range(4):\n g=T(W[f\"ln1_w.{l}\"])\n for j in range(6):\n s=slice(64*j,64*(j+1))\n q,k,v=[(T(W[f\"{n}.{l}\"])[s]*g)@P for n in [\"W_q\",\"W_k\",\"W_v\"]]\n M[l,j]=q.T@k/8\n B[l,j]=P@T(W[f\"W_o.{l}\"])[:,s]@v\n\nprint(\"QK spectral structure:\")\nfor l in range(4):\n for j in range(6):\n a=M[l,j]\n u,s,vh=torch.linalg.svd(a)\n ev=torch.linalg.eigvals(a)\n ev=ev[torch.argsort(ev.abs(),descending=True)[:8]]\n overlaps=torch.linalg.svdvals(u[:,:6].T@vh[:6].T)\n print(f\"L{l}H{j}\",\n \"symmetric energy\",round(float(((a+a.T)/2).norm().square()/a.norm().square()),3),\n \"top6 Q/K-subspace overlaps\",np.round(overlaps.numpy(),2),\n \"eigenvalues\",[(round(float(z.real),3),round(float(z.imag),3)) for z in ev])\n\nu,s,vh=torch.linalg.svd(B[0,4])\nr=vh[:3].T\nprint(\"\\nL0H4 OV top-3 INPUT subspace and other L0 QK maps:\")\nfor j in range(6):\n a=M[0,j]\n print(j,\"query/key energy\",\n np.round([float((r.T@a).norm().square()/a.norm().square()),\n float((a@r).norm().square()/a.norm().square())],4),\n \"3x3 restriction\",np.round((r.T@a@r).numpy(),4))\n\nprint(\"\\nDominant effective L0 OV input-subspace similarities (3 dims):\")\nrs=[torch.linalg.svd(B[0,j]).Vh[:3].T for j in range(6)]\nprint(np.round([[float((x.T@y).square().sum()/3) for y in rs] for x in rs],3))\n\nprint(\"\\nJoint QK covariance spectra:\")\nfor l in range(4):\n c=sum(a.T@a+a@a.T for (ll,j),a in M.items() if ll==l)\n e=torch.linalg.eigvalsh(c).flip(0)\n print(l,np.round((e[:45]/e.sum()).numpy(),4))\n\nprint(\"\\nL0 FC input and projection output mean directions:\")\nf=(T(W[\"W_fc.0\"])*T(W[\"ln2_w.0\"]))@P\no=P@T(W[\"W_proj.0\"])\nfn=f/f.norm(dim=1,keepdim=True)\non=o/o.norm(dim=0,keepdim=True)\nprint(\"FC row norm quantiles\",np.round(np.quantile(f.norm(dim=1).numpy(),[0,.1,.25,.5,.75,.9,1]),3))\nprint(\"FC nearest-other-row cosine quantiles\",\n np.round(np.quantile((fn@fn.T-torch.eye(1536)*2).max(dim=1).values.numpy(),\n [0,.1,.25,.5,.75,.9,1]),3))\nprint(\"Per-neuron paired input/output cosine quantiles\",\n np.round(np.quantile((fn*on.T).sum(1).numpy(),[0,.1,.25,.5,.75,.9,1]),3))\n",
"out": "QK spectral structure:\nL0H0 symmetric energy 0.591 top6 Q/K-subspace overlaps [0.69 0.43 0.38 0.34 0.26 0.14] eigenvalues [(-0.049, 0.0), (0.006, 0.009), (0.006, -0.009), (-0.003, 0.0), (-0.001, -0.002), (-0.001, 0.002), (-0.002, -0.0), (-0.002, 0.0)]\nL0H1 symmetric energy 0.45 top6 Q/K-subspace overlaps [0.97 0.96 0.95 0.91 0.82 0.63] eigenvalues [(0.218, -0.345), (0.218, 0.345), (-0.284, 0.0), (-0.123, 0.15), (-0.123, -0.15), (-0.097, 0.098), (-0.097, -0.098), (-0.032, 0.041)]\nL0H2 symmetric energy 0.701 top6 Q/K-subspace overlaps [0.9 0.82 0.65 0.59 0.2 0.05] eigenvalues [(0.091, 0.0), (0.053, 0.0), (-0.022, 0.0), (-0.013, 0.001), (-0.013, -0.001), (0.011, 0.0), (-0.01, 0.0), (-0.006, 0.004)]\nL0H3 symmetric energy 0.605 top6 Q/K-subspace overlaps [0.82 0.78 0.61 0.43 0.38 0.1 ] eigenvalues [(-0.078, -0.005), (-0.078, 0.005), (-0.02, 0.0), (0.006, 0.017), (0.006, -0.017), (0.007, 0.0), (-0.006, -0.002), (-0.006, 0.002)]\nL0H4 symmetric energy 0.512 top6 Q/K-subspace overlaps [0.5 0.35 0.29 0.2 0.11 0.01] eigenvalues [(-0.001, -0.001), (-0.001, 0.001), (-0.001, 0.0), (-0.001, -0.0), (0.0, 0.001), (0.0, -0.001), (-0.001, 0.0), (0.001, 0.0)]\nL0H5 symmetric energy 0.386 top6 Q/K-subspace overlaps [0.93 0.91 0.85 0.79 0.67 0.52] eigenvalues [(0.207, -0.275), (0.207, 0.275), (0.114, -0.172), (0.114, 0.172), (0.178, 0.0), (0.155, 0.084), (0.155, -0.084), (0.035, 0.166)]\nL1H0 symmetric energy 0.51 top6 Q/K-subspace overlaps [0.43 0.38 0.27 0.14 0.08 0.04] eigenvalues [(0.028, -0.051), (0.028, 0.051), (-0.051, -0.023), (-0.051, 0.023), (-0.051, 0.0), (-0.041, 0.013), (-0.041, -0.013), (-0.042, 0.0)]\nL1H1 symmetric energy 0.498 top6 Q/K-subspace overlaps [0.69 0.54 0.43 0.37 0.29 0.21] eigenvalues [(0.08, -0.26), (0.08, 0.26), (-0.144, -0.009), (-0.144, 0.009), (-0.092, 0.079), (-0.092, -0.079), (-0.102, 0.0), (-0.099, 0.019)]\nL1H2 symmetric energy 0.492 top6 Q/K-subspace overlaps [0.4 0.36 0.32 0.27 0.17 0.15] eigenvalues [(0.057, -0.202), (0.057, 0.202), (-0.108, 0.0), (-0.07, 0.05), (-0.07, -0.05), (-0.079, 0.004), (-0.079, -0.004), (-0.064, 0.0)]\nL1H3 symmetric energy 0.478 top6 Q/K-subspace overlaps [0.51 0.4 0.36 0.33 0.22 0.06] eigenvalues [(0.076, -0.272), (0.076, 0.272), (-0.094, 0.0), (0.092, 0.0), (-0.08, 0.024), (-0.08, -0.024), (-0.077, 0.0), (-0.072, 0.0)]\nL1H4 symmetric energy 0.481 top6 Q/K-subspace overlaps [0.52 0.38 0.37 0.22 0.12 0. ] eigenvalues [(0.071, -0.239), (0.071, 0.239), (-0.085, -0.014), (-0.085, 0.014), (-0.078, 0.0), (-0.065, 0.028), (-0.065, -0.028), (-0.07, 0.0)]\nL1H5 symmetric energy 0.505 top6 Q/K-subspace overlaps [0.53 0.44 0.41 0.29 0.14 0.11] eigenvalues [(0.072, -0.204), (0.072, 0.204), (-0.156, 0.0), (-0.124, 0.0), (-0.083, -0.053), (-0.083, 0.053), (-0.08, -0.03), (-0.08, 0.03)]\nL2H0 symmetric energy 0.499 top6 Q/K-subspace overlaps [0.29 0.19 0.1 0.08 0.06 0.02] eigenvalues [(0.033, -0.132), (0.033, 0.132), (-0.117, 0.0), (0.091, -0.074), (0.091, 0.074), (0.037, 0.104), (0.037, -0.104), (-0.103, -0.031)]\nL2H1 symmetric energy 0.52 top6 Q/K-subspace overlaps [0.34 0.32 0.26 0.17 0.12 0.02] eigenvalues [(0.142, 0.036), (0.142, -0.036), (0.142, 0.0), (0.141, 0.011), (0.141, -0.011), (0.131, 0.022), (0.131, -0.022), (0.103, 0.079)]\nL2H2 symmetric energy 0.516 top6 Q/K-subspace overlaps [0.28 0.26 0.24 0.2 0.09 0.04] eigenvalues [(0.259, 0.0), (0.08, 0.216), (0.08, -0.216), (0.193, 0.058), (0.193, -0.058), (0.195, -0.026), (0.195, 0.026), (-0.172, 0.022)]\nL2H3 symmetric energy 0.494 top6 Q/K-subspace overlaps [0.43 0.34 0.17 0.12 0.05 0. ] eigenvalues [(0.058, -0.185), (0.058, 0.185), (0.002, 0.106), (0.002, -0.106), (-0.055, -0.085), (-0.055, 0.085), (0.097, -0.026), (0.097, 0.026)]\nL2H4 symmetric energy 0.5 top6 Q/K-subspace overlaps [0.32 0.26 0.22 0.17 0.07 0.01] eigenvalues [(0.08, -0.22), (0.08, 0.22), (0.154, 0.0), (-0.121, 0.0), (0.086, 0.077), (0.086, -0.077), (-0.111, 0.028), (-0.111, -0.028)]\nL2H5 symmetric energy 0.497 top6 Q/K-subspace overlaps [0.23 0.17 0.14 0.09 0.04 0.02"
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 4 of at most 100: ``` QK spectral structure: L0H0 symmetric energy 0.591 top6 Q/K-subspace overlaps [0.69 0.43 0.38 0.34 0.26 0.14] eigenvalues [(-0.049, 0.0), (0.006, 0.009), (0.006, -0.009), (-0.003, 0.0), (-0.001, -0.002), (-0.001, 0.002), (-0.002, -0.0), (-0.002, 0.0)] L0H1 symmetric energy 0.45 top6 Q/K-subspace overlaps [0.97 0.96 0.95 0.91 0.82 0.63] eigenvalues [(0.218, -0.345), (0.218, 0.345), (-0.284, 0.0), (-0.123, 0.15), (-0.123, -0.15), (-0.097, 0.098), (-0.097, -0.098), (-0.032, 0.041)] L0H2 symmetric energy 0.701 top6 Q/K-subspace overlaps [0.9 0.82 0.65 0.59 0.2 0.05] eigenvalues [(0.091, 0.0), (0.053, 0.0), (-0.022, 0.0), (-0.013, 0.001), (-0.013, -0.001), (0.011, 0.0), (-0.01, 0.0), (-0.006, 0.004)] L0H3 symmetric energy 0.605 top6 Q/K-subspace overlaps [0.82 0.78 0.61 0.43 0.38 0.1 ] eigenvalues [(-0.078, -0.005), (-0.078, 0.005), (-0.02, 0.0), (0.006, 0.017), (0.006, -0.017), (0.007, 0.0), (-0.006, -0.002), (-0.006, 0.002)] L0H4 symmetric energy 0.512 top6 Q/K-subspace overlaps [0.5 0.35 0.29 0.2 0.11 0.01] eigenvalues [(-0.001, -0.001), (-0.001, 0.001), (-0.001, 0.0), (-0.001, -0.0), (0.0, 0.001), (0.0, -0.001), (-0.001, 0.0), (0.001, 0.0)] L0H5 symmetric energy 0.386 top6 Q/K-subspace overlaps [0.93 0.91 0.85 0.79 0.67 0.52] eigenvalues [(0.207, -0.275), (0.207, 0.275), (0.114, -0.172), (0.114, 0.172), (0.178, 0.0), (0.155, 0.084), (0.155, -0.084), (0.035, 0.166)] L1H0 symmetric energy 0.51 top6 Q/K-subspace overlaps [0.43 0.38 0.27 0.14 0.08 0.04] eigenvalues [(0.028, -0.051), (0.028, 0.051), (-0.051, -0.023), (-0.051, 0.023), (-0.051, 0.0), (-0.041, 0.013), (-0.041, -0.013), (-0.042, 0.0)] L1H1 symmetric energy 0.498 top6 Q/K-subspace overlaps [0.69 0.54 0.43 0.37 0.29 0.21] eigenvalues [(0.08, -0.26), (0.08, 0.26), (-0.144, -0.009), (-0.144, 0.009), (-0.092, 0.079), (-0.092, -0.079), (-0.102, 0.0), (-0.099, 0.019)] L1H2 symmetric energy 0.492 top6 Q/K-subspace overlaps [0.4 0.36 0.32 0.27 0.17 0.15] eigenvalues [(0.057, -0.202), (0.057, 0.202), (-0.108, 0.0), (-0.07, 0.05), (-0.07, -0.05), (-0.079, 0.004), (-0.079, -0.004), (-0.064, 0.0)] L1H3 symmetric energy 0.478 top6 Q/K-subspace overlaps [0.51 0.4 0.36 0.33 0.22 0.06] eigenvalues [(0.076, -0.272), (0.076, 0.272), (-0.094, 0.0), (0.092, 0.0), (-0.08, 0.024), (-0.08, -0.024), (-0.077, 0.0), (-0.072, 0.0)] L1H4 symmetric energy 0.481 top6 Q/K-subspace overlaps [0.52 0.38 0.37 0.22 0.12 0. ] eigenvalues [(0.071, -0.239), (0.071, 0.239), (-0.085, -0.014), (-0.085, 0.014), (-0.078, 0.0), (-0.065, 0.028), (-0.065, -0.028), (-0.07, 0.0)] L1H5 symmetric energy 0.505 top6 Q/K-subspace overlaps [0.53 0.44 0.41 0.29 0.14 0.11] eigenvalues [(0.072, -0.204), (0.072, 0.204), (-0.156, 0.0), (-0.124, 0.0), (-0.083, -0.053), (-0.083, 0.053), (-0.08, -0.03), (-0.08, 0.03)] L2H0 symmetric energy 0.499 top6 Q/K-subspace overlaps [0.29 0.19 0.1 0.08 0.06 0.02] eigenvalues [(0.033, -0.132), (0.033, 0.132), (-0.117, 0.0), (0.091, -0.074), (0.091, 0.074), (0.037, 0.104), (0.037, -0.104), (-0.103, -0.031)] L2H1 symmetric energy 0.52 top6 Q/K-subspace overlaps [0.34 0.32 0.26 0.17 0.12 0.02] eigenvalues [(0.142, 0.036), (0.142, -0.036), (0.142, 0.0), (0.141, 0.011), (0.141, -0.011), (0.131, 0.022), (0.131, -0.022), (0.103, 0.079)] L2H2 symmetric energy 0.516 top6 Q/K-subspace overlaps [0.28 0.26 0.24 0.2 0.09 0.04] eigenvalues [(0.259, 0.0), (0.08, 0.216), (0.08, -0.216), (0.193, 0.058), (0.193, -0.058), (0.195, -0.026), (0.195, 0.026), (-0.172, 0.022)] L2H3 symmetric energy 0.494 top6 Q/K-subspace overlaps [0.43 0.34 0.17 0.12 0.05 0. ] eigenvalues [(0.058, -0.185), (0.058, 0.185), (0.002, 0.106), (0.002, -0.106), (-0.055, -0.085), (-0.055, 0.085), (0.097, -0.026), (0.097, 0.026)] L2H4 symmetric energy 0.5 top6 Q/K-subspace overlaps [0.32 0.26 0.22 0.17 0.07 0.01] eigenvalues [(0.08, -0.22), (0.08, 0.22), (0.154, 0.0), (-0.121, 0.0), (0.086, 0.077), (0.086, -0.077), (-0.111, 0.028), (-0.111, -0.028)] L2H5 symmetric energy 0.497 top6 Q/K-subspace overlaps [0.23 0.17 0.14 0.09 0.04 0.02] eigenvalues [(-0.107, 0.0), (-0.007, 0.101), (-0.007, -0.101), (-0.079, 0.045), (-0.079, -0.045), (0.09, 0.0), (-0.067, -0.055), (-0.067, 0.055)] L3H0 symmetric energy 0.51 top6 Q/K-subspace overlaps [0.25 0.2 0.16 0.14 0.11 0.03] eigenvalues [(0.13, 0.0), (0.119, 0.04), (0.119, -0.04), (0.119, 0.018), (0.119, -0.018), (0.109, 0.0), (0.016, 0.105), (0.016, -0.105)] L3H1 symmetric energy 0.511 top6 Q/K-subspace overlaps [0.26 0.22 0.18 0.12 0.06 0.04] eigenvalues [(0.11, 0.027), (0.11, -0.027), (0.112, 0.0), (0.075, -0.071), (0.075, 0.071), (0.102, 0.0), (0.087, -0.04), (0.087, 0.04)] L3H2 symmetric energy 0.497 top6 Q/K-subspace overlaps [0.27 0.18 0.13 0.09 0.06 0.01] eigenvalues [(0.036, -0.114), (0.036, 0.114), (0.114, -0.026), (0.114, 0.026), (0.003, 0.11), (0.003, -0.11), (0.106, 0.0), (0.07, 0.077)] L3H3 symmetric energy 0.496 top6 Q/K-subspace overlaps [0.26 0.21 0.15 0.12 0.08 0. ] eigenvalues [(0.027, -0.061), (0.027, 0.061), (0.06, 0.0), (0.026, 0.049), (0.026, -0.049), (-0.002, 0.051), (-0.002, -0.051), (-0.042, 0.026)] L3H4 symmetric energy 0.502 top6 Q/K-subspace overlaps [0.36 0.22 0.17 0.13 0.08 0.02] eigenvalues [(-0.105, 0.0), (0.093, 0.0), (0.066, -0.059), (0.066, 0.059), (0.029, -0.079), (0.029, 0.079), (-0.002, -0.083), (-0.002, 0.083)] L3H5 symmetric energy 0.611 top6 Q/K-subspace overlaps [0.69 0.68 0.65 0.63 0.6 0.57] eigenvalues [(0.195, -0.117), (0.195, 0.117), (0.213, 0.074), (0.213, -0.074), (0.2, 0.104), (0.2, -0.104), (0.202, 0.076), (0.202, -0.076)] L0H4 OV top-3 INPUT subspace and other L0 QK maps: 0 query/key energy [0.755 0.2233] 3x3 restriction [[ 0.0104 0.0028 0.0239] [ 0.0242 -0.0183 0.0044] [ 0.0073 -0.0178 -0.0149]] 1 query/key energy [0.5672 0.6076] 3x3 restriction [[ 0.2271 -0.2086 0.1563] [ 0.4972 0.1801 -0.0339] [ 0.0301 -0.0462 0.0169]] 2 query/key energy [0.6801 0.5109] 3x3 restriction [[ 0.0939 0.0278 -0.0051] [ 0.01 0.044 -0.031 ] [ 0.0166 0.0038 0.0031]] 3 query/key energy [0.6231 0.4144] 3x3 restriction [[-0.0666 0.0016 -0.0272] [ 0.0056 -0.0797 0.0348] [ 0.0026 -0.0321 0.0293]] 4 query/key energy [0.03 0.0859] 3x3 restriction [[-0. 0.0001 0.0002] [-0. 0.0002 0.0002] [ 0.0001 -0.0004 -0.0005]] 5 query/key energy [0.2486 0.2813] 3x3 restriction [[ 0.1923 -0.1792 0.1387] [ 0.2756 0.1381 -0.0439] [-0.0232 -0.114 0.0519]] Dominant effective L0 OV input-subspace similarities (3 dims): [[1. 0.019 0.13 0.339 0.707 0.021] [0.019 1. 0.089 0.031 0.004 0.234] [0.13 0.089 1. 0.07 0.085 0.154] [0.339 0.031 0.07 1. 0.301 0.013] [0.707 0.004 0.085 0.301 1. 0.005] [0.021 0.234 0.154 0.013 0.005 1. ]] Joint QK covariance spectra: 0 [0.2536 0.1755 0.0546 0.0402 0.0389 0.0378 0.033 0.0316 0.0278 0.0267 0.0245 0.0235 0.0232 0.0214 0.0204 0.0171 0.0166 0.016 0.0145 0.0126 0.0112 0.0098 0.0083 0.0068 0.0051 0.0046 0.0044 0.0032 0.003 0.0026 0.0023 0.0019 0.0018 0.0016 0.0015 0.0014 0.0013 0.0011 0.0011 0.001 0.0009 0.0008 0.0008 0.0008 0.0007] 1 [0.247 0.2019 0.0933 0.0706 0.0354 0.0251 0.0184 0.0175 0.0171 0.0146 0.0125 0.0117 0.0105 0.0096 0.0093 0.0085 0.0076 0.0074 0.0066 0.0064 0.0061 0.0059 0.0056 0.0055 0.0051 0.0049 0.0047 0.0043 0.0043 0.0042 0.0039 0.0038 0.0037 0.0036 0.0033 0.0032 0.0031 0.0031 0.0029 0.0028 0.0028 0.0027 0.0026 0.0026 0.0026] 2 [0.0328 0.0242 0.0184 0.0154 0.0149 0.0146 0.0133 0.0131 0.0127 0.0121 0.0116 0.0109 0.0105 0.0104 0.0102 0.01 0.0097 0.0095 0.0092 0.0091 0.009 0.0088 0.0087 0.0084 0.0083 0.0082 0.0082 0.0081 0.0079 0.0076 0.0074 0.0074 0.0073 0.0071 0.007 0.0068 0.0068 0.0066 0.0065 0.0064 0.0064 0.0063 0.0061 0.0061 0.006 ] 3 [0.0146 0.0131 0.0111 0.0105 0.01 0.0099 0.0098 0.0096 0.0093 0.0091 0.009 0.0088 0.0086 0.0085 0.0084 0.0082 0.0081 0.008 0.0079 0.0079 0.0077 0.0076 0.0075 0.0073 0.0073 0.0072 0.0071 0.0069 0.0068 0.0068 0.0067 0.0066 0.0066 0.0065 0.0065 0.0064 0.0063 0.0063 0.0062 0.0061 0.0061 0.0061 0.006 0.0059 0.0058] L0 FC input and projection output mean directions: FC row norm quantiles [0.291 0.329 0.351 0.394 1.71 2.224 4.325] FC nearest-other-row cosine quantiles [0.233 0.394 0.423 0.463 0.554 0.819 0.985] Per-neuron paired input/output cosine quantiles [-0.635 -0.108 -0.064 -0.022 0.033 0.109 0.544] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)
P=torch.eye(384)-torch.ones((384,384))/384
print("Output-side singular spectra; possible alphabet-sized bottlenecks:")
for l in range(4):
for name in ["W_proj","W_o"]:
a=P@T(W[f"{name}.{l}"])
s=torch.linalg.svdvals(a)
print(name,l,"first 65",np.round(s[:65].numpy(),3))
ratios=s[:-1]/s[1:]
ids=torch.argsort(ratios[:80],descending=True)[:8]
print("largest early gaps (rank, ratio)",
[(int(i)+1,round(float(ratios[i]),3)) for i in ids])
a=torch.cat([P@T(W["W_o.3"]),P@T(W["W_proj.3"])],dim=1)
u,s,vh=torch.linalg.svd(a,full_matrices=False)
print("\nCombined final-block outputs first 65 SVs",np.round(s[:65].numpy(),3))
print("\nMLP weight-row norm distributions and low-norm fractions:")
for l in range(4):
f=T(W[f"W_fc.{l}"])
o=T(W[f"W_proj.{l}"])
fn=f.norm(dim=1).numpy()
on=o.norm(dim=0).numpy()
print(l,"FC norms",np.round(np.quantile(fn,[0,.1,.25,.5,.75,.9,.99,1]),3),
"Proj norms",np.round(np.quantile(on,[0,.1,.25,.5,.75,.9,.99,1]),3),
"FC < .6",round(float((fn<.6).mean()),3))
print("\nL0 positional-versus-value subspace separation:")
g=T(W["ln1_w.0"])
q=(T(W["W_q.0"])*g)@P
k=(T(W["W_k.0"])*g)@P
v=(T(W["W_v.0"])*g)@P
uq,sq,vq=torch.linalg.svd(torch.cat([q,k],dim=0),full_matrices=False)
uv,sv,vv=torch.linalg.svd(v,full_matrices=False)
print("QK input singular spectrum",np.round(sq[:60].numpy(),3))
print("V input singular spectrum",np.round(sv[:60].numpy(),3))
for r in [2,4,8,16,24,28,32,48]:
z=torch.linalg.svdvals(vq[:r]@vv[:r].T)
print("rank",r,"subspace overlaps",np.round(z.numpy(),3))
```
{
"t": 1792713588.760692,
"s": 10.29,
"code": "import numpy as np\nimport torch\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)\nP=torch.eye(384)-torch.ones((384,384))/384\n\nprint(\"Output-side singular spectra; possible alphabet-sized bottlenecks:\")\nfor l in range(4):\n for name in [\"W_proj\",\"W_o\"]:\n a=P@T(W[f\"{name}.{l}\"])\n s=torch.linalg.svdvals(a)\n print(name,l,\"first 65\",np.round(s[:65].numpy(),3))\n ratios=s[:-1]/s[1:]\n ids=torch.argsort(ratios[:80],descending=True)[:8]\n print(\"largest early gaps (rank, ratio)\",\n [(int(i)+1,round(float(ratios[i]),3)) for i in ids])\n\na=torch.cat([P@T(W[\"W_o.3\"]),P@T(W[\"W_proj.3\"])],dim=1)\nu,s,vh=torch.linalg.svd(a,full_matrices=False)\nprint(\"\\nCombined final-block outputs first 65 SVs\",np.round(s[:65].numpy(),3))\n\nprint(\"\\nMLP weight-row norm distributions and low-norm fractions:\")\nfor l in range(4):\n f=T(W[f\"W_fc.{l}\"])\n o=T(W[f\"W_proj.{l}\"])\n fn=f.norm(dim=1).numpy()\n on=o.norm(dim=0).numpy()\n print(l,\"FC norms\",np.round(np.quantile(fn,[0,.1,.25,.5,.75,.9,.99,1]),3),\n \"Proj norms\",np.round(np.quantile(on,[0,.1,.25,.5,.75,.9,.99,1]),3),\n \"FC < .6\",round(float((fn<.6).mean()),3))\n\nprint(\"\\nL0 positional-versus-value subspace separation:\")\ng=T(W[\"ln1_w.0\"])\nq=(T(W[\"W_q.0\"])*g)@P\nk=(T(W[\"W_k.0\"])*g)@P\nv=(T(W[\"W_v.0\"])*g)@P\nuq,sq,vq=torch.linalg.svd(torch.cat([q,k],dim=0),full_matrices=False)\nuv,sv,vv=torch.linalg.svd(v,full_matrices=False)\nprint(\"QK input singular spectrum\",np.round(sq[:60].numpy(),3))\nprint(\"V input singular spectrum\",np.round(sv[:60].numpy(),3))\nfor r in [2,4,8,16,24,28,32,48]:\n z=torch.linalg.svdvals(vq[:r]@vv[:r].T)\n print(\"rank\",r,\"subspace overlaps\",np.round(z.numpy(),3))\n",
"out": "Output-side singular spectra; possible alphabet-sized bottlenecks:\nW_proj 0 first 65 [10.931 7.576 6.628 6.293 6.166 5.924 5.886 5.768 5.706 5.61 5.495 5.462 5.404 5.313 5.193 5.161 5.119 5.07 5.003 4.937 4.877\n 4.793 4.692 4.664 4.651 4.637 4.548 4.526 4.487 4.404 4.329 4.283 4.246 4.206 4.178 4.141 4.12 4.105 4.061 4.023 3.997 3.958\n 3.934 3.904 3.857 3.85 3.797 3.758 3.709 3.694 3.668 3.622 3.618 3.591 3.573 3.516 3.476 3.445 3.421 3.387 3.363 3.343 3.335\n 3.268 3.248]\nlargest early gaps (rank, ratio) [(1, 1.443), (2, 1.143), (3, 1.053), (5, 1.041), (77, 1.025), (14, 1.023), (72, 1.022), (22, 1.021)]\nW_o 0 first 65 [6.676 5.235 2.834 2.679 2.619 2.581 2.549 2.515 2.503 2.477 2.449 2.443 2.424 2.404 2.365 2.361 2.319 2.291 2.269 2.259 2.243 2.217 2.2 2.196\n 2.173 2.14 2.13 2.116 2.099 2.086 2.039 2.029 2.012 1.995 1.979 1.972 1.934 1.923 1.885 1.876 1.829 1.823 1.816 1.8 1.751 1.736 1.711 1.692\n 1.654 1.642 1.613 1.608 1.589 1.572 1.551 1.536 1.498 1.483 1.474 1.437 1.415 1.384 1.358 1.355 1.346]\nlargest early gaps (rank, ratio) [(2, 1.847), (1, 1.275), (3, 1.058), (44, 1.028), (80, 1.028), (65, 1.027), (40, 1.026), (59, 1.025)]\nW_proj 1 first 65 [7.625 7.359 6.907 6.489 6.344 6.223 6.192 6.02 5.949 5.902 5.82 5.756 5.728 5.692 5.674 5.65 5.633 5.608 5.581 5.567 5.544 5.523 5.479 5.457\n 5.45 5.408 5.404 5.39 5.374 5.341 5.338 5.309 5.285 5.267 5.257 5.223 5.197 5.193 5.179 5.159 5.154 5.127 5.125 5.112 5.094 5.083 5.082 5.071\n 5.055 5.045 5.022 5.014 5.007 4.985 4.975 4.958 4.944 4.929 4.923 4.908 4.894 4.885 4.878 4.863 4.848]\nlargest early gaps (rank, ratio) [(2, 1.065), (3, 1.064), (1, 1.036), (7, 1.029), (4, 1.023), (5, 1.019), (10, 1.014), (8, 1.012)]\nW_o 1 first 65 [4.19 3.651 3.385 3.352 3.24 3.216 3.141 3.115 3.109 3.086 3.031 3.005 2.981 2.95 2.917 2.902 2.877 2.862 2.85 2.829 2.805 2.797 2.78 2.762\n 2.723 2.71 2.704 2.691 2.678 2.661 2.656 2.651 2.624 2.612 2.61 2.575 2.559 2.534 2.527 2.52 2.506 2.491 2.472 2.448 2.438 2.428 2.417 2.388\n 2.359 2.341 2.327 2.319 2.279 2.268 2.246 2.238 2.228 2.215 2.181 2.157 2.135 2.126 2.099 2.09 2.058]\nlargest early gaps (rank, ratio) [(1, 1.148), (2, 1.079), (4, 1.034), (6, 1.024), (74, 1.021), (10, 1.018), (52, 1.018), (64, 1.015)]\nW_proj 2 first 65 [10.095 8.985 8.075 7.909 7.791 7.656 7.588 7.375 7.212 7.144 7.089 7.034 7.003 6.951 6.73 6.709 6.63 6.572 6.452 6.385 6.341\n 6.321 6.252 6.207 6.087 6. 5.905 5.842 5.789 5.765 5.718 5.716 5.679 5.636 5.599 5.562 5.505 5.458 5.415 5.393 5.37 5.36\n 5.305 5.281 5.267 5.252 5.238 5.213 5.172 5.164 5.141 5.106 5.081 5.078 5.052 5.037 5.024 5.003 4.989 4.968 4.947 4.943 4.935\n 4.924 4.908]\nlargest early gaps (rank, ratio) [(1, 1.124), (2, 1.113), (14, 1.033), (7, 1.029), (8, 1.023), (3, 1.021), (24, 1.02), (18, 1.019)]\nW_o 2 first 65 [4.442 4.073 3.768 3.455 3.44 3.365 3.35 3.297 3.254 3.233 3.178 3.174 3.165 3.149 3.122 3.11 3.08 3.052 3.047 3.012 2.997 2.988 2.971 2.959\n 2.956 2.94 2.922 2.91 2.903 2.892 2.879 2.875 2.849 2.833 2.83 2.822 2.811 2.802 2.792 2.775 2.756 2.748 2.739 2.736 2.721 2.703 2.693 2.684\n 2.682 2.668 2.667 2.647 2.632 2.63 2.613 2.608 2.599 2.595 2.578 2.576 2.557 2.553 2.534 2.52 2.512]\nlargest early gaps (rank, ratio) [(1, 1.091), (3, 1.09), (2, 1.081), (5, 1.022), (10, 1.017), (7, 1.016), (8, 1.013), (19, 1.011)]\nW_proj 3 first 65 [28.175 20.293 19.292 19.135 18.708 18.57 18.139 17.776 17.484 16.54 15.913 15.61 14.975 14.872 14.417 14.313 13.327 12.971 12.593 12.37 12.224\n 12.019 11.322 11.115 10.768 9.226 8.573 7.929 6.499 5.271 3.966 3.184 2.791 2.697 2.653 2.554 2.51 2.47 2.378 2.29 2.25 2.215\n 2.117 2.081 2.035 2.01 1.984 1.969 1.951 1.926 1.888 1.868 1.851 1.833 1.816 1.795 1.791 1.781 1.763 1.735 1.721 1.712 1.704\n 1.692 1.681]\nlargest early gaps (rank, ratio) [(1, 1.388), (30, 1.329), (31, "
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 5 of at most 100: ``` Output-side singular spectra; possible alphabet-sized bottlenecks: W_proj 0 first 65 [10.931 7.576 6.628 6.293 6.166 5.924 5.886 5.768 5.706 5.61 5.495 5.462 5.404 5.313 5.193 5.161 5.119 5.07 5.003 4.937 4.877 4.793 4.692 4.664 4.651 4.637 4.548 4.526 4.487 4.404 4.329 4.283 4.246 4.206 4.178 4.141 4.12 4.105 4.061 4.023 3.997 3.958 3.934 3.904 3.857 3.85 3.797 3.758 3.709 3.694 3.668 3.622 3.618 3.591 3.573 3.516 3.476 3.445 3.421 3.387 3.363 3.343 3.335 3.268 3.248] largest early gaps (rank, ratio) [(1, 1.443), (2, 1.143), (3, 1.053), (5, 1.041), (77, 1.025), (14, 1.023), (72, 1.022), (22, 1.021)] W_o 0 first 65 [6.676 5.235 2.834 2.679 2.619 2.581 2.549 2.515 2.503 2.477 2.449 2.443 2.424 2.404 2.365 2.361 2.319 2.291 2.269 2.259 2.243 2.217 2.2 2.196 2.173 2.14 2.13 2.116 2.099 2.086 2.039 2.029 2.012 1.995 1.979 1.972 1.934 1.923 1.885 1.876 1.829 1.823 1.816 1.8 1.751 1.736 1.711 1.692 1.654 1.642 1.613 1.608 1.589 1.572 1.551 1.536 1.498 1.483 1.474 1.437 1.415 1.384 1.358 1.355 1.346] largest early gaps (rank, ratio) [(2, 1.847), (1, 1.275), (3, 1.058), (44, 1.028), (80, 1.028), (65, 1.027), (40, 1.026), (59, 1.025)] W_proj 1 first 65 [7.625 7.359 6.907 6.489 6.344 6.223 6.192 6.02 5.949 5.902 5.82 5.756 5.728 5.692 5.674 5.65 5.633 5.608 5.581 5.567 5.544 5.523 5.479 5.457 5.45 5.408 5.404 5.39 5.374 5.341 5.338 5.309 5.285 5.267 5.257 5.223 5.197 5.193 5.179 5.159 5.154 5.127 5.125 5.112 5.094 5.083 5.082 5.071 5.055 5.045 5.022 5.014 5.007 4.985 4.975 4.958 4.944 4.929 4.923 4.908 4.894 4.885 4.878 4.863 4.848] largest early gaps (rank, ratio) [(2, 1.065), (3, 1.064), (1, 1.036), (7, 1.029), (4, 1.023), (5, 1.019), (10, 1.014), (8, 1.012)] W_o 1 first 65 [4.19 3.651 3.385 3.352 3.24 3.216 3.141 3.115 3.109 3.086 3.031 3.005 2.981 2.95 2.917 2.902 2.877 2.862 2.85 2.829 2.805 2.797 2.78 2.762 2.723 2.71 2.704 2.691 2.678 2.661 2.656 2.651 2.624 2.612 2.61 2.575 2.559 2.534 2.527 2.52 2.506 2.491 2.472 2.448 2.438 2.428 2.417 2.388 2.359 2.341 2.327 2.319 2.279 2.268 2.246 2.238 2.228 2.215 2.181 2.157 2.135 2.126 2.099 2.09 2.058] largest early gaps (rank, ratio) [(1, 1.148), (2, 1.079), (4, 1.034), (6, 1.024), (74, 1.021), (10, 1.018), (52, 1.018), (64, 1.015)] W_proj 2 first 65 [10.095 8.985 8.075 7.909 7.791 7.656 7.588 7.375 7.212 7.144 7.089 7.034 7.003 6.951 6.73 6.709 6.63 6.572 6.452 6.385 6.341 6.321 6.252 6.207 6.087 6. 5.905 5.842 5.789 5.765 5.718 5.716 5.679 5.636 5.599 5.562 5.505 5.458 5.415 5.393 5.37 5.36 5.305 5.281 5.267 5.252 5.238 5.213 5.172 5.164 5.141 5.106 5.081 5.078 5.052 5.037 5.024 5.003 4.989 4.968 4.947 4.943 4.935 4.924 4.908] largest early gaps (rank, ratio) [(1, 1.124), (2, 1.113), (14, 1.033), (7, 1.029), (8, 1.023), (3, 1.021), (24, 1.02), (18, 1.019)] W_o 2 first 65 [4.442 4.073 3.768 3.455 3.44 3.365 3.35 3.297 3.254 3.233 3.178 3.174 3.165 3.149 3.122 3.11 3.08 3.052 3.047 3.012 2.997 2.988 2.971 2.959 2.956 2.94 2.922 2.91 2.903 2.892 2.879 2.875 2.849 2.833 2.83 2.822 2.811 2.802 2.792 2.775 2.756 2.748 2.739 2.736 2.721 2.703 2.693 2.684 2.682 2.668 2.667 2.647 2.632 2.63 2.613 2.608 2.599 2.595 2.578 2.576 2.557 2.553 2.534 2.52 2.512] largest early gaps (rank, ratio) [(1, 1.091), (3, 1.09), (2, 1.081), (5, 1.022), (10, 1.017), (7, 1.016), (8, 1.013), (19, 1.011)] W_proj 3 first 65 [28.175 20.293 19.292 19.135 18.708 18.57 18.139 17.776 17.484 16.54 15.913 15.61 14.975 14.872 14.417 14.313 13.327 12.971 12.593 12.37 12.224 12.019 11.322 11.115 10.768 9.226 8.573 7.929 6.499 5.271 3.966 3.184 2.791 2.697 2.653 2.554 2.51 2.47 2.378 2.29 2.25 2.215 2.117 2.081 2.035 2.01 1.984 1.969 1.951 1.926 1.888 1.868 1.851 1.833 1.816 1.795 1.791 1.781 1.763 1.735 1.721 1.712 1.704 1.692 1.681] largest early gaps (rank, ratio) [(1, 1.388), (30, 1.329), (31, 1.246), (29, 1.233), (28, 1.22), (25, 1.167), (32, 1.141), (27, 1.081)] W_o 3 first 65 [9.173 6.63 4.442 4.111 4.054 4.04 4.006 3.975 3.959 3.942 3.877 3.859 3.845 3.81 3.791 3.775 3.76 3.753 3.713 3.7 3.648 3.64 3.615 3.606 3.572 3.563 3.53 3.527 3.513 3.475 3.455 3.446 3.407 3.389 3.385 3.364 3.346 3.323 3.319 3.297 3.253 3.234 3.221 3.191 3.183 3.178 3.149 3.137 3.12 3.087 3.074 3.062 3.048 3.035 3.014 2.981 2.978 2.949 2.933 2.901 2.896 2.88 2.859 2.848 2.831] largest early gaps (rank, ratio) [(2, 1.493), (1, 1.384), (3, 1.081), (10, 1.017), (20, 1.014), (4, 1.014), (40, 1.014), (32, 1.012)] Combined final-block outputs first 65 SVs [28.583 20.47 19.507 19.381 18.958 18.841 18.369 18.033 17.725 16.838 16.187 15.891 15.33 15.191 14.751 14.663 13.733 13.356 12.975 12.723 12.62 12.373 11.68 11.568 11.239 10.383 9.596 8.97 8.36 6.522 5.305 4.821 4.647 4.479 4.455 4.367 4.295 4.207 4.084 4.002 3.977 3.915 3.906 3.834 3.746 3.717 3.681 3.623 3.601 3.58 3.57 3.529 3.51 3.472 3.453 3.419 3.39 3.374 3.354 3.335 3.32 3.311 3.282 3.279 3.25 ] MLP weight-row norm distributions and low-norm fractions: 0 FC norms [0.33 0.371 0.394 0.437 1.766 2.393 3.213 4.287] Proj norms [0.127 0.174 0.201 0.263 1.71 2.098 2.672 3.366] FC < .6 0.635 1 FC norms [0.441 1.869 1.942 2.023 2.11 2.206 2.45 8.68 ] Proj norms [0.325 1.776 1.839 1.908 1.98 2.056 2.257 2.909] FC < .6 0.003 2 FC norms [1.678 1.933 1.979 2.024 2.073 2.121 2.209 2.731] Proj norms [1.597 1.837 1.899 1.965 2.048 2.12 2.25 2.85 ] FC < .6 0.0 3 FC norms [1.746 1.902 1.954 1.995 2.036 2.072 2.176 2.947] Proj norms [1.305 1.799 1.96 2.132 2.328 2.511 2.932 4.621] FC < .6 0.0 L0 positional-versus-value subspace separation: QK input singular spectrum [5.682 4.22 2.647 2.597 2.532 2.414 2.342 2.272 2.202 2.166 2.134 2.056 2.02 1.963 1.931 1.871 1.813 1.794 1.745 1.683 1.591 1.53 1.473 1.436 1.268 1.229 1.194 1.125 1.085 1.01 0.982 0.941 0.931 0.881 0.854 0.837 0.818 0.799 0.786 0.775 0.745 0.725 0.714 0.694 0.662 0.657 0.65 0.63 0.629 0.606 0.589 0.587 0.564 0.559 0.554 0.542 0.539 0.535 0.524 0.514] V input singular spectrum [3.195 2.304 1.316 0.88 0.809 0.771 0.73 0.704 0.685 0.662 0.638 0.629 0.623 0.621 0.604 0.595 0.582 0.567 0.554 0.55 0.542 0.541 0.522 0.513 0.496 0.487 0.479 0.475 0.464 0.457 0.447 0.431 0.428 0.424 0.41 0.399 0.389 0.383 0.373 0.369 0.358 0.345 0.334 0.333 0.319 0.313 0.308 0.305 0.302 0.3 0.295 0.292 0.286 0.281 0.278 0.275 0.27 0.265 0.263 0.258] rank 2 subspace overlaps [0.974 0.8 ] rank 4 subspace overlaps [0.984 0.937 0.185 0.064] rank 8 subspace overlaps [0.986 0.947 0.626 0.335 0.166 0.09 0.051 0.005] rank 16 subspace overlaps [0.99 0.971 0.851 0.535 0.309 0.292 0.262 0.236 0.21 0.148 0.146 0.1 0.086 0.065 0.029 0.005] rank 24 subspace overlaps [0.992 0.979 0.906 0.691 0.498 0.452 0.396 0.364 0.321 0.294 0.272 0.244 0.232 0.194 0.179 0.162 0.129 0.122 0.089 0.078 0.069 0.048 0.025 0.01 ] rank 28 subspace overlaps [0.993 0.985 0.948 0.886 0.664 0.559 0.527 0.5 0.452 0.439 0.35 0.3 0.295 0.275 0.261 0.237 0.201 0.177 0.163 0.142 0.121 0.115 0.102 0.089 0.047 0.044 0.013 0.007] rank 32 subspace overlaps [0.995 0.987 0.969 0.905 0.894 0.84 0.733 0.641 0.564 0.523 0.503 0.455 0.416 0.402 0.338 0.302 0.275 0.261 0.24 0.227 0.198 0.184 0.176 0.17 0.145 0.134 0.102 0.095 0.071 0.061 0.036 0.008] rank 48 subspace overlaps [0.998 0.995 0.986 0.977 0.976 0.97 0.968 0.963 0.962 0.957 0.954 0.947 0.943 0.933 0.931 0.921 0.909 0.906 0.902 0.889 0.873 0.869 0.84 0.799 0.785 0.756 0.742 0.665 0.644 0.578 0.463 0.449 0.396 0.342 0.308 0.282 0.236 0.209 0.194 0.173 0.16 0.133 0.114 0.087 0.07 0.052 0.027 0.011] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
from sklearn.cluster import KMeans
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)
P=torch.eye(384)-torch.ones((384,384))/384
g=T(W["ln1_w.0"])
vv={}
for j in [1,5]:
vv[j]=(T(W["W_v.0"])[64*j:64*(j+1)]*g)@P
print("V effective head",j,"singular values",np.round(torch.linalg.svdvals(vv[j])[:45].numpy(),3))
a=torch.linalg.svd(vv[1],full_matrices=False).Vh
b=torch.linalg.svd(vv[5],full_matrices=False).Vh
print("\nCommon L0H1/L0H5 value-input subspaces:")
for r in [12,20,24,28,32,40,48,64]:
print(r,np.round(torch.linalg.svdvals(a[:r]@b[:r].T).numpy(),3))
# Analyze trained first-MLP rows after restriction to a weight-derived subspace.
f=(T(W["W_fc.0"])*T(W["ln2_w.0"]))@P
active=np.linalg.norm(W["W_fc.0"],axis=1)>.8
c=vv[1].T@vv[1]+vv[5].T@vv[5]
e,u=torch.linalg.eigh(c)
u=u.flip(1)
print("\nJoint value-input eigenvalues",np.round(e.flip(0)[:50].numpy(),3))
for r in [20,24,28,32]:
z=(f@u[:,:r]).numpy()[active]
norms=np.linalg.norm(z,axis=1)
z=z/(norms[:,None]+1e-12)
sim=z@z.T
np.fill_diagonal(sim,-1)
print("\nProjected FC0 rank",r,"active",len(z),
"nearest-row cosine quantiles",
np.round(np.quantile(sim.max(1),[.1,.25,.5,.75,.9]),3))
for k in [20,24,28,32]:
km=KMeans(n_clusters=k,n_init=10,random_state=1).fit(z)
counts=np.bincount(km.labels_,minlength=k)
coherence=np.linalg.norm(km.cluster_centers_,axis=1)
print("k",k,"average coherence",round(float(np.average(coherence,weights=counts)),3),
"cluster sizes",sorted(counts.tolist(),reverse=True),
"coherence",np.round(np.sort(coherence),2))
print("\nFinal MLP output-column geometry in top 28 output directions:")
o=P@T(W["W_proj.3"])
u,s,vh=torch.linalg.svd(o,full_matrices=False)
z=vh[:28].T.numpy()
z=z/(np.linalg.norm(z,axis=1,keepdims=True)+1e-12)
for k in [20,24,28,32,40]:
km=KMeans(n_clusters=k,n_init=10,random_state=1).fit(z)
counts=np.bincount(km.labels_,minlength=k)
coherence=np.linalg.norm(km.cluster_centers_,axis=1)
print("k",k,"average coherence",round(float(np.average(coherence,weights=counts)),3),
"cluster sizes",sorted(counts.tolist(),reverse=True),
"coherence",np.round(np.sort(coherence),2))
```
{
"t": 1792713714.9762762,
"s": 20.58,
"code": "import numpy as np\nimport torch\nfrom sklearn.cluster import KMeans\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)\nP=torch.eye(384)-torch.ones((384,384))/384\ng=T(W[\"ln1_w.0\"])\nvv={}\nfor j in [1,5]:\n vv[j]=(T(W[\"W_v.0\"])[64*j:64*(j+1)]*g)@P\n print(\"V effective head\",j,\"singular values\",np.round(torch.linalg.svdvals(vv[j])[:45].numpy(),3))\na=torch.linalg.svd(vv[1],full_matrices=False).Vh\nb=torch.linalg.svd(vv[5],full_matrices=False).Vh\nprint(\"\\nCommon L0H1/L0H5 value-input subspaces:\")\nfor r in [12,20,24,28,32,40,48,64]:\n print(r,np.round(torch.linalg.svdvals(a[:r]@b[:r].T).numpy(),3))\n\n# Analyze trained first-MLP rows after restriction to a weight-derived subspace.\nf=(T(W[\"W_fc.0\"])*T(W[\"ln2_w.0\"]))@P\nactive=np.linalg.norm(W[\"W_fc.0\"],axis=1)>.8\nc=vv[1].T@vv[1]+vv[5].T@vv[5]\ne,u=torch.linalg.eigh(c)\nu=u.flip(1)\nprint(\"\\nJoint value-input eigenvalues\",np.round(e.flip(0)[:50].numpy(),3))\nfor r in [20,24,28,32]:\n z=(f@u[:,:r]).numpy()[active]\n norms=np.linalg.norm(z,axis=1)\n z=z/(norms[:,None]+1e-12)\n sim=z@z.T\n np.fill_diagonal(sim,-1)\n print(\"\\nProjected FC0 rank\",r,\"active\",len(z),\n \"nearest-row cosine quantiles\",\n np.round(np.quantile(sim.max(1),[.1,.25,.5,.75,.9]),3))\n for k in [20,24,28,32]:\n km=KMeans(n_clusters=k,n_init=10,random_state=1).fit(z)\n counts=np.bincount(km.labels_,minlength=k)\n coherence=np.linalg.norm(km.cluster_centers_,axis=1)\n print(\"k\",k,\"average coherence\",round(float(np.average(coherence,weights=counts)),3),\n \"cluster sizes\",sorted(counts.tolist(),reverse=True),\n \"coherence\",np.round(np.sort(coherence),2))\n\nprint(\"\\nFinal MLP output-column geometry in top 28 output directions:\")\no=P@T(W[\"W_proj.3\"])\nu,s,vh=torch.linalg.svd(o,full_matrices=False)\nz=vh[:28].T.numpy()\nz=z/(np.linalg.norm(z,axis=1,keepdims=True)+1e-12)\nfor k in [20,24,28,32,40]:\n km=KMeans(n_clusters=k,n_init=10,random_state=1).fit(z)\n counts=np.bincount(km.labels_,minlength=k)\n coherence=np.linalg.norm(km.cluster_centers_,axis=1)\n print(\"k\",k,\"average coherence\",round(float(np.average(coherence,weights=counts)),3),\n \"cluster sizes\",sorted(counts.tolist(),reverse=True),\n \"coherence\",np.round(np.sort(coherence),2))\n",
"out": "V effective head 1 singular values [0.48 0.455 0.448 0.429 0.421 0.41 0.403 0.385 0.379 0.373 0.37 0.36 0.353 0.341 0.33 0.327 0.322 0.313 0.309 0.303 0.295 0.287 0.281 0.278\n 0.27 0.255 0.253 0.241 0.226 0.216 0.209 0.201 0.198 0.171 0.164 0.159 0.155 0.148 0.145 0.143 0.137 0.134 0.126 0.124 0.117]\nV effective head 5 singular values [0.493 0.473 0.446 0.437 0.426 0.408 0.404 0.393 0.386 0.376 0.363 0.361 0.356 0.345 0.333 0.329 0.322 0.316 0.31 0.304 0.299 0.289 0.285 0.277\n 0.274 0.267 0.252 0.245 0.241 0.235 0.233 0.221 0.216 0.212 0.208 0.203 0.199 0.194 0.188 0.179 0.169 0.164 0.157 0.143 0.138]\n\nCommon L0H1/L0H5 value-input subspaces:\n12 [0.923 0.908 0.879 0.832 0.787 0.707 0.659 0.531 0.416 0.281 0.231 0.023]\n20 [0.963 0.943 0.935 0.924 0.921 0.907 0.891 0.883 0.87 0.868 0.856 0.84 0.798 0.773 0.581 0.559 0.506 0.353 0.134 0.034]\n24 [0.966 0.958 0.957 0.946 0.936 0.932 0.923 0.912 0.909 0.895 0.891 0.874 0.862 0.843 0.839 0.785 0.675 0.639 0.587 0.549 0.425 0.287 0.156 0.005]\n28 [0.972 0.966 0.959 0.957 0.946 0.94 0.938 0.934 0.931 0.918 0.913 0.9 0.889 0.883 0.872 0.849 0.836 0.81 0.8 0.788 0.702 0.615 0.55 0.482\n 0.427 0.356 0.193 0.048]\n32 [0.977 0.969 0.961 0.958 0.951 0.946 0.945 0.943 0.939 0.931 0.925 0.917 0.912 0.905 0.885 0.875 0.872 0.858 0.839 0.82 0.787 0.727 0.708 0.666\n 0.61 0.567 0.487 0.385 0.262 0.133 0.083 0.013]\n40 [0.981 0.975 0.971 0.966 0.963 0.958 0.956 0.952 0.948 0.947 0.946 0.936 0.933 0.927 0.918 0.913 0.898 0.894 0.886 0.883 0.864 0.839 0.825 0.804\n 0.792 0.756 0.74 0.694 0.671 0.633 0.548 0.531 0.502 0.471 0.344 0.304 0.232 0.088 0.057 0.017]\n48 [0.984 0.98 0.974 0.973 0.97 0.967 0.964 0.962 0.956 0.955 0.949 0.946 0.942 0.939 0.929 0.927 0.925 0.911 0.899 0.896 0.885 0.866 0.866 0.854\n 0.835 0.82 0.802 0.786 0.762 0.747 0.729 0.706 0.693 0.647 0.614 0.6 0.518 0.476 0.445 0.418 0.319 0.317 0.289 0.258 0.15 0.134 0.059 0.049]\n64 [0.987 0.984 0.98 0.98 0.977 0.974 0.972 0.97 0.97 0.966 0.961 0.96 0.958 0.953 0.95 0.945 0.938 0.935 0.928 0.924 0.917 0.902 0.897 0.892\n 0.881 0.879 0.856 0.853 0.833 0.828 0.812 0.795 0.785 0.781 0.743 0.727 0.71 0.69 0.657 0.65 0.632 0.609 0.571 0.564 0.555 0.504 0.478 0.461\n 0.426 0.415 0.395 0.382 0.36 0.339 0.306 0.253 0.241 0.209 0.194 0.12 0.073 0.067 0.053 0.023]\n\nJoint value-input eigenvalues [0.408 0.386 0.353 0.348 0.326 0.295 0.289 0.28 0.266 0.264 0.244 0.239 0.225 0.222 0.218 0.205 0.201 0.198 0.19 0.178 0.173 0.157 0.145 0.145\n 0.131 0.126 0.125 0.112 0.107 0.095 0.089 0.083 0.082 0.076 0.075 0.067 0.066 0.064 0.059 0.056 0.053 0.051 0.05 0.049 0.041 0.037 0.033 0.032\n 0.03 0.029]\n\nProjected FC0 rank 20 active 531 nearest-row cosine quantiles [0.649 0.71 0.796 0.922 0.991]\nk 20 average coherence 0.692 cluster sizes [42, 38, 37, 37, 36, 30, 29, 28, 26, 26, 24, 24, 23, 21, 21, 20, 19, 18, 18, 14] coherence [0.58 0.59 0.62 0.63 0.64 0.66 0.66 0.67 0.7 0.7 0.71 0.71 0.72 0.73 0.74 0.76 0.76 0.76 0.79 0.84]\nk 24 average coherence 0.723 cluster sizes [36, 35, 34, 27, 26, 25, 25, 24, 24, 22, 22, 21, 21, 20, 20, 19, 19, 19, 17, 17, 15, 15, 15, 13] coherence [0.63 0.64 0.65 0.66 0.66 0.68 0.68 0.68 0.7 0.7 0.71 0.73 0.73 0.73 0.75 0.75 0.76 0.76 0.78 0.79 0.79 0.81 0.81 0.85]\nk 28 average coherence 0.746 cluster sizes [37, 32, 26, 25, 23, 21, 20, 20, 20, 20, 19, 19, 19, 18, 18, 18, 18, 18, 17, 17, 16, 16, 16, 14, 13, 12, 12, 7] coherence [0.66 0.67 0.68 0.69 0.7 0.7 0.7 0.7 0.72 0.72 0.73 0.73 0.74 0.75 0.75 0.75 0.76 0.76 0.77 0.77 0.77 0.77 0.78 0.8 0.8 0.81 0.84 0.89]\nk 32 average coherence 0.759 cluster sizes [35, 33, 22, 22, 21, 20, 20, 19, 19, 18, 18, 18, 17, 17, 17, 16, 16, 16, 16, 15, 14, 13, 13, 13, 12, 11, 11, 11, 11, 10, 9, 8] coherence [0.62 0.67 0.69 0.7 0.7 0.7 0.7 0.7 0.71 0.72 0.72 0.73 0.74 0.74 0.75 0.76 0.76 0.77 0.77 0.78 0.78 0.78 0.78 0.8 0.81 0.82 0.83 0.83 0.84\n 0.85 0.87 0.88]\n\nProjected FC0 rank 24 active 531 nearest-row cosine quantiles [0.629 0.698 0.792 0.928 0.993]\nk 20 a"
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 6 of at most 100: ``` V effective head 1 singular values [0.48 0.455 0.448 0.429 0.421 0.41 0.403 0.385 0.379 0.373 0.37 0.36 0.353 0.341 0.33 0.327 0.322 0.313 0.309 0.303 0.295 0.287 0.281 0.278 0.27 0.255 0.253 0.241 0.226 0.216 0.209 0.201 0.198 0.171 0.164 0.159 0.155 0.148 0.145 0.143 0.137 0.134 0.126 0.124 0.117] V effective head 5 singular values [0.493 0.473 0.446 0.437 0.426 0.408 0.404 0.393 0.386 0.376 0.363 0.361 0.356 0.345 0.333 0.329 0.322 0.316 0.31 0.304 0.299 0.289 0.285 0.277 0.274 0.267 0.252 0.245 0.241 0.235 0.233 0.221 0.216 0.212 0.208 0.203 0.199 0.194 0.188 0.179 0.169 0.164 0.157 0.143 0.138] Common L0H1/L0H5 value-input subspaces: 12 [0.923 0.908 0.879 0.832 0.787 0.707 0.659 0.531 0.416 0.281 0.231 0.023] 20 [0.963 0.943 0.935 0.924 0.921 0.907 0.891 0.883 0.87 0.868 0.856 0.84 0.798 0.773 0.581 0.559 0.506 0.353 0.134 0.034] 24 [0.966 0.958 0.957 0.946 0.936 0.932 0.923 0.912 0.909 0.895 0.891 0.874 0.862 0.843 0.839 0.785 0.675 0.639 0.587 0.549 0.425 0.287 0.156 0.005] 28 [0.972 0.966 0.959 0.957 0.946 0.94 0.938 0.934 0.931 0.918 0.913 0.9 0.889 0.883 0.872 0.849 0.836 0.81 0.8 0.788 0.702 0.615 0.55 0.482 0.427 0.356 0.193 0.048] 32 [0.977 0.969 0.961 0.958 0.951 0.946 0.945 0.943 0.939 0.931 0.925 0.917 0.912 0.905 0.885 0.875 0.872 0.858 0.839 0.82 0.787 0.727 0.708 0.666 0.61 0.567 0.487 0.385 0.262 0.133 0.083 0.013] 40 [0.981 0.975 0.971 0.966 0.963 0.958 0.956 0.952 0.948 0.947 0.946 0.936 0.933 0.927 0.918 0.913 0.898 0.894 0.886 0.883 0.864 0.839 0.825 0.804 0.792 0.756 0.74 0.694 0.671 0.633 0.548 0.531 0.502 0.471 0.344 0.304 0.232 0.088 0.057 0.017] 48 [0.984 0.98 0.974 0.973 0.97 0.967 0.964 0.962 0.956 0.955 0.949 0.946 0.942 0.939 0.929 0.927 0.925 0.911 0.899 0.896 0.885 0.866 0.866 0.854 0.835 0.82 0.802 0.786 0.762 0.747 0.729 0.706 0.693 0.647 0.614 0.6 0.518 0.476 0.445 0.418 0.319 0.317 0.289 0.258 0.15 0.134 0.059 0.049] 64 [0.987 0.984 0.98 0.98 0.977 0.974 0.972 0.97 0.97 0.966 0.961 0.96 0.958 0.953 0.95 0.945 0.938 0.935 0.928 0.924 0.917 0.902 0.897 0.892 0.881 0.879 0.856 0.853 0.833 0.828 0.812 0.795 0.785 0.781 0.743 0.727 0.71 0.69 0.657 0.65 0.632 0.609 0.571 0.564 0.555 0.504 0.478 0.461 0.426 0.415 0.395 0.382 0.36 0.339 0.306 0.253 0.241 0.209 0.194 0.12 0.073 0.067 0.053 0.023] Joint value-input eigenvalues [0.408 0.386 0.353 0.348 0.326 0.295 0.289 0.28 0.266 0.264 0.244 0.239 0.225 0.222 0.218 0.205 0.201 0.198 0.19 0.178 0.173 0.157 0.145 0.145 0.131 0.126 0.125 0.112 0.107 0.095 0.089 0.083 0.082 0.076 0.075 0.067 0.066 0.064 0.059 0.056 0.053 0.051 0.05 0.049 0.041 0.037 0.033 0.032 0.03 0.029] Projected FC0 rank 20 active 531 nearest-row cosine quantiles [0.649 0.71 0.796 0.922 0.991] k 20 average coherence 0.692 cluster sizes [42, 38, 37, 37, 36, 30, 29, 28, 26, 26, 24, 24, 23, 21, 21, 20, 19, 18, 18, 14] coherence [0.58 0.59 0.62 0.63 0.64 0.66 0.66 0.67 0.7 0.7 0.71 0.71 0.72 0.73 0.74 0.76 0.76 0.76 0.79 0.84] k 24 average coherence 0.723 cluster sizes [36, 35, 34, 27, 26, 25, 25, 24, 24, 22, 22, 21, 21, 20, 20, 19, 19, 19, 17, 17, 15, 15, 15, 13] coherence [0.63 0.64 0.65 0.66 0.66 0.68 0.68 0.68 0.7 0.7 0.71 0.73 0.73 0.73 0.75 0.75 0.76 0.76 0.78 0.79 0.79 0.81 0.81 0.85] k 28 average coherence 0.746 cluster sizes [37, 32, 26, 25, 23, 21, 20, 20, 20, 20, 19, 19, 19, 18, 18, 18, 18, 18, 17, 17, 16, 16, 16, 14, 13, 12, 12, 7] coherence [0.66 0.67 0.68 0.69 0.7 0.7 0.7 0.7 0.72 0.72 0.73 0.73 0.74 0.75 0.75 0.75 0.76 0.76 0.77 0.77 0.77 0.77 0.78 0.8 0.8 0.81 0.84 0.89] k 32 average coherence 0.759 cluster sizes [35, 33, 22, 22, 21, 20, 20, 19, 19, 18, 18, 18, 17, 17, 17, 16, 16, 16, 16, 15, 14, 13, 13, 13, 12, 11, 11, 11, 11, 10, 9, 8] coherence [0.62 0.67 0.69 0.7 0.7 0.7 0.7 0.7 0.71 0.72 0.72 0.73 0.74 0.74 0.75 0.76 0.76 0.77 0.77 0.78 0.78 0.78 0.78 0.8 0.81 0.82 0.83 0.83 0.84 0.85 0.87 0.88] Projected FC0 rank 24 active 531 nearest-row cosine quantiles [0.629 0.698 0.792 0.928 0.993] k 20 average coherence 0.688 cluster sizes [55, 43, 40, 33, 31, 30, 27, 25, 25, 24, 23, 22, 21, 21, 21, 20, 19, 19, 18, 14] coherence [0.54 0.55 0.62 0.63 0.65 0.68 0.68 0.69 0.69 0.7 0.7 0.71 0.73 0.73 0.74 0.75 0.75 0.75 0.76 0.86] k 24 average coherence 0.718 cluster sizes [56, 37, 30, 29, 29, 25, 24, 23, 23, 22, 22, 21, 21, 20, 19, 17, 17, 15, 15, 14, 14, 13, 13, 12] coherence [0.57 0.59 0.63 0.66 0.67 0.68 0.68 0.69 0.71 0.71 0.72 0.74 0.75 0.75 0.75 0.75 0.76 0.78 0.78 0.81 0.81 0.82 0.82 0.87] k 28 average coherence 0.734 cluster sizes [49, 31, 29, 27, 25, 23, 22, 22, 22, 21, 20, 20, 20, 19, 18, 18, 16, 14, 14, 14, 14, 14, 12, 12, 12, 12, 8, 3] coherence [0.55 0.64 0.66 0.69 0.69 0.7 0.71 0.71 0.71 0.71 0.72 0.73 0.73 0.73 0.73 0.74 0.76 0.76 0.76 0.77 0.77 0.77 0.79 0.79 0.83 0.85 0.85 0.86] k 32 average coherence 0.752 cluster sizes [50, 29, 24, 21, 21, 20, 20, 19, 18, 18, 17, 17, 17, 16, 16, 16, 16, 15, 15, 15, 14, 13, 13, 13, 13, 13, 12, 11, 10, 7, 6, 6] coherence [0.64 0.67 0.68 0.68 0.69 0.7 0.71 0.71 0.72 0.72 0.72 0.73 0.74 0.74 0.74 0.75 0.76 0.76 0.76 0.76 0.76 0.77 0.77 0.78 0.79 0.79 0.8 0.8 0.8 0.84 0.87 0.87] Projected FC0 rank 28 active 531 nearest-row cosine quantiles [0.625 0.696 0.792 0.931 0.993] k 20 average coherence 0.671 cluster sizes [70, 52, 36, 29, 28, 28, 27, 25, 25, 24, 23, 22, 21, 20, 19, 18, 17, 17, 16, 14] coherence [0.5 0.51 0.56 0.63 0.63 0.64 0.65 0.66 0.68 0.69 0.7 0.72 0.72 0.72 0.74 0.74 0.75 0.78 0.8 0.87] k 24 average coherence 0.707 cluster sizes [61, 35, 29, 27, 27, 26, 25, 24, 23, 22, 21, 21, 21, 21, 17, 16, 16, 15, 15, 15, 15, 13, 13, 13] coherence [0.53 0.58 0.6 0.63 0.64 0.69 0.7 0.7 0.71 0.72 0.73 0.73 0.74 0.74 0.75 0.75 0.75 0.75 0.75 0.76 0.76 0.81 0.82 0.87] k 28 average coherence 0.732 cluster sizes [66, 27, 25, 22, 22, 21, 21, 20, 20, 20, 20, 19, 18, 17, 17, 17, 16, 15, 15, 14, 14, 14, 14, 13, 13, 13, 10, 8] coherence [0.57 0.63 0.65 0.65 0.69 0.69 0.69 0.7 0.71 0.72 0.73 0.73 0.73 0.73 0.74 0.74 0.74 0.76 0.76 0.76 0.77 0.78 0.78 0.79 0.82 0.82 0.83 0.87] k 32 average coherence 0.747 cluster sizes [32, 27, 26, 24, 22, 21, 21, 21, 21, 19, 19, 18, 18, 18, 18, 17, 16, 16, 15, 14, 13, 12, 12, 12, 12, 12, 11, 11, 11, 9, 7, 6] coherence [0.66 0.67 0.67 0.67 0.68 0.68 0.69 0.7 0.71 0.72 0.73 0.73 0.73 0.74 0.74 0.74 0.75 0.75 0.75 0.75 0.75 0.76 0.76 0.77 0.79 0.8 0.8 0.82 0.86 0.87 0.87 0.89] Projected FC0 rank 32 active 531 nearest-row cosine quantiles [0.609 0.684 0.793 0.929 0.992] k 20 average coherence 0.665 cluster sizes [65, 58, 30, 29, 28, 28, 27, 27, 25, 24, 24, 23, 20, 20, 19, 19, 18, 17, 16, 14] coherence [0.44 0.55 0.59 0.63 0.65 0.66 0.66 0.66 0.66 0.67 0.67 0.69 0.7 0.71 0.74 0.75 0.76 0.77 0.8 0.84] k 24 average coherence 0.7 cluster sizes [46, 34, 31, 29, 28, 27, 26, 25, 25, 24, 23, 23, 23, 20, 20, 18, 17, 16, 16, 15, 13, 13, 12, 7] coherence [0.55 0.56 0.57 0.61 0.61 0.66 0.67 0.67 0.68 0.69 0.7 0.73 0.73 0.73 0.73 0.75 0.77 0.79 0.8 0.8 0.8 0.84 0.85 0.86] k 28 average coherence 0.725 cluster sizes [43, 28, 28, 27, 26, 23, 22, 21, 21, 20, 20, 19, 19, 18, 17, 17, 16, 15, 15, 14, 14, 14, 14, 13, 13, 13, 12, 9] coherence [0.54 0.64 0.65 0.66 0.66 0.66 0.67 0.69 0.7 0.7 0.71 0.72 0.72 0.73 0.74 0.74 0.74 0.75 0.75 0.75 0.76 0.76 0.77 0.79 0.81 0.82 0.84 0.85] k 32 average coherence 0.746 cluster sizes [62, 29, 28, 22, 22, 22, 19, 18, 18, 18, 17, 17, 16, 16, 16, 15, 14, 14, 13, 13, 13, 12, 12, 12, 11, 11, 11, 10, 9, 8, 7, 6] coherence [0.61 0.62 0.66 0.66 0.68 0.68 0.68 0.68 0.71 0.71 0.72 0.73 0.74 0.75 0.75 0.75 0.75 0.76 0.76 0.77 0.78 0.79 0.8 0.8 0.8 0.81 0.83 0.83 0.84 0.85 0.9 0.92] Final MLP output-column geometry in top 28 output directions: k 20 average coherence 0.436 cluster sizes [89, 86, 84, 84, 83, 82, 81, 81, 81, 80, 79, 79, 77, 76, 76, 76, 72, 66, 56, 48] coherence [0.39 0.4 0.41 0.41 0.41 0.41 0.41 0.41 0.42 0.42 0.42 0.43 0.43 0.44 0.44 0.44 0.45 0.53 0.54 0.58] k 24 average coherence 0.453 cluster sizes [81, 76, 74, 71, 71, 70, 69, 69, 68, 67, 67, 67, 66, 66, 64, 64, 63, 62, 60, 58, 52, 50, 49, 32] coherence [0.41 0.41 0.42 0.42 0.43 0.43 0.43 0.43 0.44 0.44 0.44 0.44 0.44 0.45 0.45 0.45 0.45 0.45 0.46 0.46 0.49 0.55 0.61 0.65] k 28 average coherence 0.47 cluster sizes [69, 68, 64, 64, 63, 63, 61, 60, 59, 59, 58, 58, 57, 57, 57, 55, 55, 52, 52, 51, 51, 49, 48, 48, 46, 46, 38, 28] coherence [0.43 0.43 0.44 0.44 0.44 0.45 0.45 0.45 0.45 0.45 0.45 0.45 0.45 0.46 0.46 0.46 0.46 0.47 0.47 0.47 0.48 0.48 0.49 0.49 0.49 0.51 0.61 0.68] k 32 average coherence 0.483 cluster sizes [60, 58, 58, 57, 56, 55, 55, 54, 53, 53, 52, 52, 51, 51, 49, 49, 49, 48, 46, 44, 44, 43, 42, 41, 41, 41, 41, 40, 40, 40, 38, 35] coherence [0.44 0.44 0.45 0.45 0.45 0.45 0.46 0.46 0.46 0.46 0.47 0.47 0.47 0.47 0.47 0.47 0.47 0.47 0.47 0.48 0.48 0.48 0.49 0.49 0.49 0.5 0.5 0.51 0.51 0.55 0.63 0.65] k 40 average coherence 0.504 cluster sizes [46, 45, 44, 44, 44, 44, 44, 44, 43, 42, 41, 41, 41, 40, 40, 40, 40, 39, 39, 39, 38, 38, 38, 38, 38, 38, 37, 36, 36, 36, 36, 35, 35, 34, 32, 32, 32, 29, 29, 29] coherence [0.47 0.47 0.47 0.47 0.48 0.48 0.48 0.48 0.48 0.49 0.49 0.49 0.49 0.49 0.49 0.49 0.49 0.49 0.49 0.49 0.49 0.49 0.49 0.49 0.5 0.5 0.5 0.5 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.52 0.53 0.64 0.65 0.67] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
from scipy.linalg import eig
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)
P=torch.eye(384)-torch.ones((384,384))/384
print("Final MLP output weights after linear regression on input weights.")
print("This is weight-space regression only, not evaluation of an MLP.")
f=T(W["W_fc.3"])
o=P@T(W["W_proj.3"])
alpha=-float((o*f.T).sum()/f.square().sum())
print("Best scalar cancellation coefficient",round(alpha,5))
u,s,vh=torch.linalg.svd(f,full_matrices=False)
for method in ["scalar","regress"]:
if method=="scalar":
r=o+alpha*(P@f.T)
else:
r=o-(o@u)@u.T
sv=torch.linalg.svdvals(r)
print(method,"first 55 singular values",np.round(sv[:55].numpy(),3))
ratios=sv[:-1]/sv[1:]
ids=torch.argsort(ratios[15:45],descending=True)[:8]+15
print("gaps",[(int(i)+1,round(float(ratios[i]),3)) for i in ids])
g=T(W["ln1_w.0"])
m={}
for j in [1,5]:
sl=slice(64*j,64*(j+1))
q=(T(W["W_q.0"])[sl]*g)@P
k=(T(W["W_k.0"])[sl]*g)@P
m[j]=q.T@k/8
c=sum(a@a.T+a.T@a for a in m.values())
ee,uu=torch.linalg.eigh(c)
uu=uu.flip(1)
print("\nShared QK subspace: congruence-invariant phase diagnostics.")
for r in [8,12,16,20,24,28]:
z=uu[:,:r]
a=(z.T@m[1]@z).numpy()
b=(z.T@m[5]@z).numpy()
print("rank",r)
for label,x,y in [("H1 versus transpose",a,a.T),
("H5 versus transpose",b,b.T),
("H1 versus H5",a,b)]:
ev=eig(x,y,right=False)
good=ev[np.isfinite(ev)]
positive=good[good.imag>1e-5]
positive=positive[np.argsort(np.angle(positive))]
print(label,[(round(float(np.angle(v)),3),round(float(abs(v)),3)) for v in positive])
```
{
"t": 1792713790.2425113,
"s": 11.25,
"code": "import numpy as np\nimport torch\nfrom scipy.linalg import eig\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)\nP=torch.eye(384)-torch.ones((384,384))/384\n\nprint(\"Final MLP output weights after linear regression on input weights.\")\nprint(\"This is weight-space regression only, not evaluation of an MLP.\")\nf=T(W[\"W_fc.3\"])\no=P@T(W[\"W_proj.3\"])\nalpha=-float((o*f.T).sum()/f.square().sum())\nprint(\"Best scalar cancellation coefficient\",round(alpha,5))\nu,s,vh=torch.linalg.svd(f,full_matrices=False)\nfor method in [\"scalar\",\"regress\"]:\n if method==\"scalar\":\n r=o+alpha*(P@f.T)\n else:\n r=o-(o@u)@u.T\n sv=torch.linalg.svdvals(r)\n print(method,\"first 55 singular values\",np.round(sv[:55].numpy(),3))\n ratios=sv[:-1]/sv[1:]\n ids=torch.argsort(ratios[15:45],descending=True)[:8]+15\n print(\"gaps\",[(int(i)+1,round(float(ratios[i]),3)) for i in ids])\n\ng=T(W[\"ln1_w.0\"])\nm={}\nfor j in [1,5]:\n sl=slice(64*j,64*(j+1))\n q=(T(W[\"W_q.0\"])[sl]*g)@P\n k=(T(W[\"W_k.0\"])[sl]*g)@P\n m[j]=q.T@k/8\nc=sum(a@a.T+a.T@a for a in m.values())\nee,uu=torch.linalg.eigh(c)\nuu=uu.flip(1)\n\nprint(\"\\nShared QK subspace: congruence-invariant phase diagnostics.\")\nfor r in [8,12,16,20,24,28]:\n z=uu[:,:r]\n a=(z.T@m[1]@z).numpy()\n b=(z.T@m[5]@z).numpy()\n print(\"rank\",r)\n for label,x,y in [(\"H1 versus transpose\",a,a.T),\n (\"H5 versus transpose\",b,b.T),\n (\"H1 versus H5\",a,b)]:\n ev=eig(x,y,right=False)\n good=ev[np.isfinite(ev)]\n positive=good[good.imag>1e-5]\n positive=positive[np.argsort(np.angle(positive))]\n print(label,[(round(float(np.angle(v)),3),round(float(abs(v)),3)) for v in positive])\n",
"out": "Final MLP output weights after linear regression on input weights.\nThis is weight-space regression only, not evaluation of an MLP.\nBest scalar cancellation coefficient 0.1226\nscalar first 55 singular values [28.283 20.308 19.323 19.114 18.692 18.549 18.137 17.786 17.483 16.527 15.894 15.6 14.962 14.885 14.406 14.312 13.34 12.985 12.591 12.374 12.256\n 12.037 11.345 11.107 10.768 9.222 8.598 7.927 6.518 5.299 3.919 3.262 2.783 2.656 2.402 2.166 2.125 1.959 1.933 1.864 1.809 1.786\n 1.757 1.74 1.704 1.659 1.646 1.626 1.579 1.576 1.56 1.535 1.513 1.502 1.49 ]\ngaps [(30, 1.352), (29, 1.23), (28, 1.216), (31, 1.201), (32, 1.172), (25, 1.168), (35, 1.109), (34, 1.106)]\nregress first 55 singular values [18.909 18.247 17.385 16.993 16.749 16.475 16.143 15.907 15.758 14.826 14.425 14.017 13.553 13.176 12.916 12.74 11.888 11.44 11.079 10.939 10.762\n 10.254 9.966 9.931 9.492 8.276 7.586 7.012 4.48 3.376 2.688 2.222 1.946 1.805 1.713 1.473 1.43 1.362 1.343 1.332 1.314 1.303\n 1.276 1.239 1.228 1.205 1.187 1.181 1.172 1.163 1.155 1.15 1.141 1.13 1.119]\ngaps [(28, 1.565), (29, 1.327), (30, 1.256), (31, 1.21), (35, 1.163), (25, 1.147), (32, 1.141), (26, 1.091)]\n\nShared QK subspace: congruence-invariant phase diagnostics.\nrank 8\nH1 versus transpose [(1.721, 1.0), (2.465, 1.0)]\nH5 versus transpose [(1.281, 1.0), (1.896, 1.0), (2.669, 1.0)]\nH1 versus H5 [(1.138, 0.121), (2.248, 1.244)]\nrank 12\nH1 versus transpose [(1.166, 1.0), (1.595, 1.0), (1.88, 0.795), (1.88, 1.257)]\nH5 versus transpose [(1.119, 1.0), (1.909, 1.0), (1.957, 1.0), (2.47, 1.135), (2.47, 0.881)]\nH1 versus H5 [(0.748, 0.1), (1.525, 0.156), (2.197, 1.26), (2.834, 0.57)]\nrank 16\nH1 versus transpose [(0.134, 1.0), (0.828, 1.932), (0.828, 0.518), (1.476, 1.0), (1.8, 1.0), (1.993, 1.258), (1.993, 0.795)]\nH5 versus transpose [(0.944, 1.0), (1.503, 1.0), (2.092, 1.0), (2.489, 0.779), (2.489, 1.284)]\nH1 versus H5 [(0.478, 0.176), (0.823, 0.201), (1.357, 0.155), (1.611, 0.223), (2.133, 1.253), (2.637, 0.951)]\nrank 20\nH1 versus transpose [(0.067, 1.0), (0.819, 1.0), (1.214, 1.0), (1.627, 1.0), (1.937, 0.792), (1.937, 1.263), (1.94, 1.0)]\nH5 versus transpose [(0.753, 1.0), (1.356, 1.0), (1.898, 1.0), (2.226, 1.038), (2.226, 0.963), (2.437, 1.0), (2.79, 1.0), (3.023, 1.0)]\nH1 versus H5 [(0.678, 1.291), (0.716, 0.193), (0.925, 0.186), (1.349, 0.164), (1.654, 0.19), (1.995, 1.047), (2.123, 1.336), (2.171, 0.186)]\nrank 24\nH1 versus transpose [(0.024, 1.0), (0.333, 1.0), (0.958, 4.242), (0.958, 0.236), (1.368, 1.0), (1.563, 1.0), (1.854, 1.0), (1.982, 1.0), (2.652, 1.702), (2.652, 0.587), (2.924, 1.0)]\nH5 versus transpose [(1.055, 1.0), (1.713, 1.0), (1.88, 1.0), (2.232, 1.227), (2.232, 0.815), (2.394, 1.0), (2.661, 0.978), (2.661, 1.023)]\nH1 versus H5 [(0.114, 1.404), (0.126, 4.66), (0.299, 0.459), (0.823, 0.164), (0.892, 0.169), (1.455, 0.155), (1.584, 0.19), (1.899, 0.212), (2.021, 0.969), (2.132, 1.328), (2.209, 0.431)]\nrank 28\nH1 versus transpose [(0.084, 1.0), (0.426, 1.0), (0.978, 1.0), (1.517, 1.0), (1.714, 1.0), (1.926, 1.0), (2.355, 1.0), (2.82, 0.144), (2.82, 6.929), (2.933, 1.0)]\nH5 versus transpose [(1.13, 1.0), (1.523, 0.136), (1.523, 7.341), (1.6, 1.0), (2.216, 0.603), (2.216, 1.658), (2.37, 1.0), (2.559, 0.908), (2.559, 1.101), (2.735, 0.868), (2.735, 1.152)]\nH1 versus H5 [(0.671, 0.373), (0.693, 0.209), (0.915, 0.33), (1.347, 0.142), (1.374, 0.102), (1.58, 0.244), (1.644, 0.135), (2.013, 1.369), (2.014, 0.208), (2.112, 0.49), (2.469, 1.38)]\n"
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 7 of at most 100: ``` Final MLP output weights after linear regression on input weights. This is weight-space regression only, not evaluation of an MLP. Best scalar cancellation coefficient 0.1226 scalar first 55 singular values [28.283 20.308 19.323 19.114 18.692 18.549 18.137 17.786 17.483 16.527 15.894 15.6 14.962 14.885 14.406 14.312 13.34 12.985 12.591 12.374 12.256 12.037 11.345 11.107 10.768 9.222 8.598 7.927 6.518 5.299 3.919 3.262 2.783 2.656 2.402 2.166 2.125 1.959 1.933 1.864 1.809 1.786 1.757 1.74 1.704 1.659 1.646 1.626 1.579 1.576 1.56 1.535 1.513 1.502 1.49 ] gaps [(30, 1.352), (29, 1.23), (28, 1.216), (31, 1.201), (32, 1.172), (25, 1.168), (35, 1.109), (34, 1.106)] regress first 55 singular values [18.909 18.247 17.385 16.993 16.749 16.475 16.143 15.907 15.758 14.826 14.425 14.017 13.553 13.176 12.916 12.74 11.888 11.44 11.079 10.939 10.762 10.254 9.966 9.931 9.492 8.276 7.586 7.012 4.48 3.376 2.688 2.222 1.946 1.805 1.713 1.473 1.43 1.362 1.343 1.332 1.314 1.303 1.276 1.239 1.228 1.205 1.187 1.181 1.172 1.163 1.155 1.15 1.141 1.13 1.119] gaps [(28, 1.565), (29, 1.327), (30, 1.256), (31, 1.21), (35, 1.163), (25, 1.147), (32, 1.141), (26, 1.091)] Shared QK subspace: congruence-invariant phase diagnostics. rank 8 H1 versus transpose [(1.721, 1.0), (2.465, 1.0)] H5 versus transpose [(1.281, 1.0), (1.896, 1.0), (2.669, 1.0)] H1 versus H5 [(1.138, 0.121), (2.248, 1.244)] rank 12 H1 versus transpose [(1.166, 1.0), (1.595, 1.0), (1.88, 0.795), (1.88, 1.257)] H5 versus transpose [(1.119, 1.0), (1.909, 1.0), (1.957, 1.0), (2.47, 1.135), (2.47, 0.881)] H1 versus H5 [(0.748, 0.1), (1.525, 0.156), (2.197, 1.26), (2.834, 0.57)] rank 16 H1 versus transpose [(0.134, 1.0), (0.828, 1.932), (0.828, 0.518), (1.476, 1.0), (1.8, 1.0), (1.993, 1.258), (1.993, 0.795)] H5 versus transpose [(0.944, 1.0), (1.503, 1.0), (2.092, 1.0), (2.489, 0.779), (2.489, 1.284)] H1 versus H5 [(0.478, 0.176), (0.823, 0.201), (1.357, 0.155), (1.611, 0.223), (2.133, 1.253), (2.637, 0.951)] rank 20 H1 versus transpose [(0.067, 1.0), (0.819, 1.0), (1.214, 1.0), (1.627, 1.0), (1.937, 0.792), (1.937, 1.263), (1.94, 1.0)] H5 versus transpose [(0.753, 1.0), (1.356, 1.0), (1.898, 1.0), (2.226, 1.038), (2.226, 0.963), (2.437, 1.0), (2.79, 1.0), (3.023, 1.0)] H1 versus H5 [(0.678, 1.291), (0.716, 0.193), (0.925, 0.186), (1.349, 0.164), (1.654, 0.19), (1.995, 1.047), (2.123, 1.336), (2.171, 0.186)] rank 24 H1 versus transpose [(0.024, 1.0), (0.333, 1.0), (0.958, 4.242), (0.958, 0.236), (1.368, 1.0), (1.563, 1.0), (1.854, 1.0), (1.982, 1.0), (2.652, 1.702), (2.652, 0.587), (2.924, 1.0)] H5 versus transpose [(1.055, 1.0), (1.713, 1.0), (1.88, 1.0), (2.232, 1.227), (2.232, 0.815), (2.394, 1.0), (2.661, 0.978), (2.661, 1.023)] H1 versus H5 [(0.114, 1.404), (0.126, 4.66), (0.299, 0.459), (0.823, 0.164), (0.892, 0.169), (1.455, 0.155), (1.584, 0.19), (1.899, 0.212), (2.021, 0.969), (2.132, 1.328), (2.209, 0.431)] rank 28 H1 versus transpose [(0.084, 1.0), (0.426, 1.0), (0.978, 1.0), (1.517, 1.0), (1.714, 1.0), (1.926, 1.0), (2.355, 1.0), (2.82, 0.144), (2.82, 6.929), (2.933, 1.0)] H5 versus transpose [(1.13, 1.0), (1.523, 0.136), (1.523, 7.341), (1.6, 1.0), (2.216, 0.603), (2.216, 1.658), (2.37, 1.0), (2.559, 0.908), (2.559, 1.101), (2.735, 0.868), (2.735, 1.152)] H1 versus H5 [(0.671, 0.373), (0.693, 0.209), (0.915, 0.33), (1.347, 0.142), (1.374, 0.102), (1.58, 0.244), (1.644, 0.135), (2.013, 1.369), (2.014, 0.208), (2.112, 0.49), (2.469, 1.38)] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)
P=torch.eye(384)-torch.ones((384,384))/384
M,B={},{}
for l in range(4):
g=T(W[f"ln1_w.{l}"])
for j in range(6):
sl=slice(64*j,64*(j+1))
q,k,v=[(T(W[f"{n}.{l}"])[sl]*g)@P for n in ["W_q","W_k","W_v"]]
M[l,j]=q.T@k/8
B[l,j]=P@T(W[f"W_o.{l}"])[:,sl]@v
print("Directional composition: key trace(M B), query trace(M B.T), each divided by ||M||||B||.")
print("A substantially key-specific positive match is more suggestive of an induction circuit.")
for l in range(1,4):
for j in range(6):
a=M[l,j]
rows=[]
for p in range(l):
for h in range(6):
b=B[p,h]
den=a.norm()*b.norm()
key=float(torch.trace(a@b)/den)
query=float(torch.trace(a@b.T)/den)
rows.append((key-query,key,query,p,h))
rows.sort(reverse=True)
print(f"L{l}H{j}",[(f"L{p}H{h}",round(k,4),round(q,4),round(d,4))
for d,k,q,p,h in rows[:3]])
print("\nL0 QK energy in a value-defined candidate character subspace:")
g=T(W["ln1_w.0"])
v1=(T(W["W_v.0"])[64:128]*g)@P
v5=(T(W["W_v.0"])[320:384]*g)@P
c=v1.T@v1+v5.T@v5
ev,u=torch.linalg.eigh(c)
u=u.flip(1)
for r in [16,24,32]:
z=u[:,:r]
print("subspace dimension",r)
for j in range(6):
a=M[0,j]
den=a.norm().square()
qe=float((z.T@a).norm().square()/den)
ke=float((a@z).norm().square()/den)
both=float((z.T@a@z).norm().square()/den)
print(j,"query/key/both",np.round([qe,ke,both],4))
print("\nL1 attention query-bias key directions: pairwise cosine.")
kb=[]
for j in range(6):
sl=slice(64*j,64*(j+1))
q=T(W["W_q.1"])[sl]
k=(T(W["W_k.1"])[sl]*T(W["ln1_w.1"]))@P
v=k.T@(q@T(W["ln1_b.1"]))/8
kb.append(v/v.norm())
print(np.round(torch.stack(kb).numpy()@torch.stack(kb).numpy().T,3))
```
{
"t": 1792713914.9889407,
"s": 10.88,
"code": "import numpy as np\nimport torch\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)\nP=torch.eye(384)-torch.ones((384,384))/384\nM,B={},{}\n\nfor l in range(4):\n g=T(W[f\"ln1_w.{l}\"])\n for j in range(6):\n sl=slice(64*j,64*(j+1))\n q,k,v=[(T(W[f\"{n}.{l}\"])[sl]*g)@P for n in [\"W_q\",\"W_k\",\"W_v\"]]\n M[l,j]=q.T@k/8\n B[l,j]=P@T(W[f\"W_o.{l}\"])[:,sl]@v\n\nprint(\"Directional composition: key trace(M B), query trace(M B.T), each divided by ||M||||B||.\")\nprint(\"A substantially key-specific positive match is more suggestive of an induction circuit.\")\nfor l in range(1,4):\n for j in range(6):\n a=M[l,j]\n rows=[]\n for p in range(l):\n for h in range(6):\n b=B[p,h]\n den=a.norm()*b.norm()\n key=float(torch.trace(a@b)/den)\n query=float(torch.trace(a@b.T)/den)\n rows.append((key-query,key,query,p,h))\n rows.sort(reverse=True)\n print(f\"L{l}H{j}\",[(f\"L{p}H{h}\",round(k,4),round(q,4),round(d,4))\n for d,k,q,p,h in rows[:3]])\n\nprint(\"\\nL0 QK energy in a value-defined candidate character subspace:\")\ng=T(W[\"ln1_w.0\"])\nv1=(T(W[\"W_v.0\"])[64:128]*g)@P\nv5=(T(W[\"W_v.0\"])[320:384]*g)@P\nc=v1.T@v1+v5.T@v5\nev,u=torch.linalg.eigh(c)\nu=u.flip(1)\nfor r in [16,24,32]:\n z=u[:,:r]\n print(\"subspace dimension\",r)\n for j in range(6):\n a=M[0,j]\n den=a.norm().square()\n qe=float((z.T@a).norm().square()/den)\n ke=float((a@z).norm().square()/den)\n both=float((z.T@a@z).norm().square()/den)\n print(j,\"query/key/both\",np.round([qe,ke,both],4))\n\nprint(\"\\nL1 attention query-bias key directions: pairwise cosine.\")\nkb=[]\nfor j in range(6):\n sl=slice(64*j,64*(j+1))\n q=T(W[\"W_q.1\"])[sl]\n k=(T(W[\"W_k.1\"])[sl]*T(W[\"ln1_w.1\"]))@P\n v=k.T@(q@T(W[\"ln1_b.1\"]))/8\n kb.append(v/v.norm())\nprint(np.round(torch.stack(kb).numpy()@torch.stack(kb).numpy().T,3))\n",
"out": "Directional composition: key trace(M B), query trace(M B.T), each divided by ||M||||B||.\nA substantially key-specific positive match is more suggestive of an induction circuit.\nL1H0 [('L0H5', -0.0034, -0.0471, 0.0437), ('L0H4', 0.0207, -0.0068, 0.0276), ('L0H1', -0.0001, -0.0222, 0.0221)]\nL1H1 [('L0H4', 0.0191, -0.0246, 0.0437), ('L0H3', 0.0055, -0.0119, 0.0174), ('L0H5', 0.0154, 0.0072, 0.0082)]\nL1H2 [('L0H4', 0.0326, -0.0233, 0.056), ('L0H5', -0.0027, -0.0332, 0.0305), ('L0H3', 0.0092, -0.0086, 0.0178)]\nL1H3 [('L0H4', 0.0421, -0.0282, 0.0703), ('L0H3', 0.0063, -0.0094, 0.0157), ('L0H5', 0.0124, 0.0092, 0.0032)]\nL1H4 [('L0H4', 0.0439, -0.0262, 0.0701), ('L0H5', 0.0094, -0.0234, 0.0328), ('L0H3', 0.0114, -0.0045, 0.0159)]\nL1H5 [('L0H5', 0.0128, -0.0122, 0.025), ('L0H4', -0.0036, -0.0203, 0.0167), ('L0H3', -0.0004, -0.0077, 0.0073)]\nL2H0 [('L0H4', 0.0099, -0.0047, 0.0146), ('L1H2', -0.0003, -0.0129, 0.0127), ('L0H3', 0.0078, 0.0008, 0.007)]\nL2H1 [('L0H5', 0.0776, -0.0398, 0.1174), ('L1H2', 0.0482, -0.0636, 0.1118), ('L1H4', 0.0476, -0.0495, 0.0971)]\nL2H2 [('L0H4', 0.02, -0.0077, 0.0276), ('L0H0', 0.0114, 0.0063, 0.005), ('L0H3', 0.0015, -0.0021, 0.0035)]\nL2H3 [('L0H4', 0.0396, -0.009, 0.0486), ('L0H0', 0.0177, 0.0052, 0.0126), ('L0H3', 0.0066, -0.0018, 0.0084)]\nL2H4 [('L0H4', 0.0246, -0.0048, 0.0294), ('L0H3', 0.0101, -0.0014, 0.0115), ('L0H5', 0.0018, -0.0006, 0.0024)]\nL2H5 [('L1H2', 0.0095, -0.012, 0.0215), ('L0H4', 0.0188, -0.0023, 0.0211), ('L1H5', 0.0068, -0.0092, 0.016)]\nL3H0 [('L2H4', 0.0044, -0.0182, 0.0225), ('L0H5', 0.019, 0.0037, 0.0153), ('L0H4', 0.0098, -0.0003, 0.0101)]\nL3H1 [('L2H2', 0.1104, -0.0164, 0.1268), ('L0H5', 0.061, -0.0198, 0.0808), ('L1H2', 0.038, -0.0323, 0.0703)]\nL3H2 [('L0H4', 0.013, -0.0025, 0.0155), ('L0H0', 0.0062, 0.0017, 0.0045), ('L0H3', -0.0008, -0.0014, 0.0006)]\nL3H3 [('L2H5', 0.0045, -0.0134, 0.0179), ('L2H3', 0.0103, -0.0069, 0.0172), ('L2H4', 0.0148, 0.0019, 0.0129)]\nL3H4 [('L0H5', 0.0101, -0.0033, 0.0134), ('L0H4', 0.0119, 0.0004, 0.0115), ('L1H1', 0.0069, -0.0039, 0.0108)]\nL3H5 [('L2H2', 0.1836, -0.0354, 0.2189), ('L2H3', 0.055, -0.0604, 0.1154), ('L1H3', 0.0683, -0.0422, 0.1104)]\n\nL0 QK energy in a value-defined candidate character subspace:\nsubspace dimension 16\n0 query/key/both [0.0224 0.2603 0.0044]\n1 query/key/both [0.0286 0.0327 0.0015]\n2 query/key/both [0.0605 0.1709 0.0173]\n3 query/key/both [0.0436 0.1638 0.0092]\n4 query/key/both [0.0547 0.0596 0.0032]\n5 query/key/both [0.0308 0.0319 0.0011]\nsubspace dimension 24\n0 query/key/both [0.03 0.3361 0.0076]\n1 query/key/both [0.0649 0.0706 0.0128]\n2 query/key/both [0.0876 0.2359 0.0348]\n3 query/key/both [0.0651 0.2143 0.0168]\n4 query/key/both [0.0781 0.091 0.0077]\n5 query/key/both [0.0536 0.057 0.0038]\nsubspace dimension 32\n0 query/key/both [0.0502 0.3927 0.0154]\n1 query/key/both [0.1885 0.2035 0.0842]\n2 query/key/both [0.1574 0.3177 0.0624]\n3 query/key/both [0.1216 0.3235 0.0365]\n4 query/key/both [0.113 0.1314 0.0162]\n5 query/key/both [0.1712 0.1751 0.0323]\n\nL1 attention query-bias key directions: pairwise cosine.\n[[1. 0.798 0.808 0.801 0.81 0.757]\n [0.798 1. 0.923 0.924 0.921 0.897]\n [0.808 0.923 1. 0.899 0.9 0.868]\n [0.801 0.924 0.899 1. 0.926 0.899]\n [0.81 0.921 0.9 0.926 1. 0.883]\n [0.757 0.897 0.868 0.899 0.883 1. ]]\n"
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 8 of at most 100:
```
Directional composition: key trace(M B), query trace(M B.T), each divided by ||M||||B||.
A substantially key-specific positive match is more suggestive of an induction circuit.
L1H0 [('L0H5', -0.0034, -0.0471, 0.0437), ('L0H4', 0.0207, -0.0068, 0.0276), ('L0H1', -0.0001, -0.0222, 0.0221)]
L1H1 [('L0H4', 0.0191, -0.0246, 0.0437), ('L0H3', 0.0055, -0.0119, 0.0174), ('L0H5', 0.0154, 0.0072, 0.0082)]
L1H2 [('L0H4', 0.0326, -0.0233, 0.056), ('L0H5', -0.0027, -0.0332, 0.0305), ('L0H3', 0.0092, -0.0086, 0.0178)]
L1H3 [('L0H4', 0.0421, -0.0282, 0.0703), ('L0H3', 0.0063, -0.0094, 0.0157), ('L0H5', 0.0124, 0.0092, 0.0032)]
L1H4 [('L0H4', 0.0439, -0.0262, 0.0701), ('L0H5', 0.0094, -0.0234, 0.0328), ('L0H3', 0.0114, -0.0045, 0.0159)]
L1H5 [('L0H5', 0.0128, -0.0122, 0.025), ('L0H4', -0.0036, -0.0203, 0.0167), ('L0H3', -0.0004, -0.0077, 0.0073)]
L2H0 [('L0H4', 0.0099, -0.0047, 0.0146), ('L1H2', -0.0003, -0.0129, 0.0127), ('L0H3', 0.0078, 0.0008, 0.007)]
L2H1 [('L0H5', 0.0776, -0.0398, 0.1174), ('L1H2', 0.0482, -0.0636, 0.1118), ('L1H4', 0.0476, -0.0495, 0.0971)]
L2H2 [('L0H4', 0.02, -0.0077, 0.0276), ('L0H0', 0.0114, 0.0063, 0.005), ('L0H3', 0.0015, -0.0021, 0.0035)]
L2H3 [('L0H4', 0.0396, -0.009, 0.0486), ('L0H0', 0.0177, 0.0052, 0.0126), ('L0H3', 0.0066, -0.0018, 0.0084)]
L2H4 [('L0H4', 0.0246, -0.0048, 0.0294), ('L0H3', 0.0101, -0.0014, 0.0115), ('L0H5', 0.0018, -0.0006, 0.0024)]
L2H5 [('L1H2', 0.0095, -0.012, 0.0215), ('L0H4', 0.0188, -0.0023, 0.0211), ('L1H5', 0.0068, -0.0092, 0.016)]
L3H0 [('L2H4', 0.0044, -0.0182, 0.0225), ('L0H5', 0.019, 0.0037, 0.0153), ('L0H4', 0.0098, -0.0003, 0.0101)]
L3H1 [('L2H2', 0.1104, -0.0164, 0.1268), ('L0H5', 0.061, -0.0198, 0.0808), ('L1H2', 0.038, -0.0323, 0.0703)]
L3H2 [('L0H4', 0.013, -0.0025, 0.0155), ('L0H0', 0.0062, 0.0017, 0.0045), ('L0H3', -0.0008, -0.0014, 0.0006)]
L3H3 [('L2H5', 0.0045, -0.0134, 0.0179), ('L2H3', 0.0103, -0.0069, 0.0172), ('L2H4', 0.0148, 0.0019, 0.0129)]
L3H4 [('L0H5', 0.0101, -0.0033, 0.0134), ('L0H4', 0.0119, 0.0004, 0.0115), ('L1H1', 0.0069, -0.0039, 0.0108)]
L3H5 [('L2H2', 0.1836, -0.0354, 0.2189), ('L2H3', 0.055, -0.0604, 0.1154), ('L1H3', 0.0683, -0.0422, 0.1104)]
L0 QK energy in a value-defined candidate character subspace:
subspace dimension 16
0 query/key/both [0.0224 0.2603 0.0044]
1 query/key/both [0.0286 0.0327 0.0015]
2 query/key/both [0.0605 0.1709 0.0173]
3 query/key/both [0.0436 0.1638 0.0092]
4 query/key/both [0.0547 0.0596 0.0032]
5 query/key/both [0.0308 0.0319 0.0011]
subspace dimension 24
0 query/key/both [0.03 0.3361 0.0076]
1 query/key/both [0.0649 0.0706 0.0128]
2 query/key/both [0.0876 0.2359 0.0348]
3 query/key/both [0.0651 0.2143 0.0168]
4 query/key/both [0.0781 0.091 0.0077]
5 query/key/both [0.0536 0.057 0.0038]
subspace dimension 32
0 query/key/both [0.0502 0.3927 0.0154]
1 query/key/both [0.1885 0.2035 0.0842]
2 query/key/both [0.1574 0.3177 0.0624]
3 query/key/both [0.1216 0.3235 0.0365]
4 query/key/both [0.113 0.1314 0.0162]
5 query/key/both [0.1712 0.1751 0.0323]
L1 attention query-bias key directions: pairwise cosine.
[[1. 0.798 0.808 0.801 0.81 0.757]
[0.798 1. 0.923 0.924 0.921 0.897]
[0.808 0.923 1. 0.899 0.9 0.868]
[0.801 0.924 0.899 1. 0.926 0.899]
[0.81 0.921 0.9 0.926 1. 0.883]
[0.757 0.897 0.868 0.899 0.883 1. ]]
```Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
from scipy.linalg import eig
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)
P=torch.eye(384)-torch.ones((384,384))/384
g=T(W["ln1_w.0"])
m={}
for j in [1,5]:
sl=slice(64*j,64*(j+1))
q=(T(W["W_q.0"])[sl]*g)@P
k=(T(W["W_k.0"])[sl]*g)@P
m[j]=q.T@k/8
c=sum(a@a.T+a.T@a for a in m.values())
_,u=torch.linalg.eigh(c)
u=u.flip(1)
print("Relative-shift diagnostic using only generalized eigenvectors of QK weight products.")
print("For shared rotational modes, phase(H1) approximately k*phase(H5) would suggest a lag ratio k.")
for r in [12,16,20,24,28,32,40]:
z=u[:,:r]
a=(z.T@m[1]@z).numpy()
b=(z.T@m[5]@z).numpy()
ev,vec=eig(b.T,b)
rows=[]
for i,x in enumerate(ev):
if not np.isfinite(x) or x.imag<=1e-5 or abs(abs(x)-1)>.12:
continue
v=vec[:,i]
av=a@v
atv=a.T@v
overlap=np.vdot(av,atv)/(np.linalg.norm(av)*np.linalg.norm(atv)+1e-15)
rows.append((float(np.angle(x)),float(np.angle(overlap)),float(abs(overlap))))
print("\nrank",r,"(H5 phase,H1 phase,mode agreement)",np.round(rows,3).tolist())
if rows:
p,q,w=np.array(rows).T
print("phase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4:",
[(k,round(float(np.average(np.cos(q-k*p),weights=w**4)),3))
for k in [-3,-2,-1,1,2,3,4]])
```
{
"t": 1792713980.5300128,
"s": 11.68,
"code": "import numpy as np\nimport torch\nfrom scipy.linalg import eig\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)\nP=torch.eye(384)-torch.ones((384,384))/384\ng=T(W[\"ln1_w.0\"])\nm={}\nfor j in [1,5]:\n sl=slice(64*j,64*(j+1))\n q=(T(W[\"W_q.0\"])[sl]*g)@P\n k=(T(W[\"W_k.0\"])[sl]*g)@P\n m[j]=q.T@k/8\nc=sum(a@a.T+a.T@a for a in m.values())\n_,u=torch.linalg.eigh(c)\nu=u.flip(1)\n\nprint(\"Relative-shift diagnostic using only generalized eigenvectors of QK weight products.\")\nprint(\"For shared rotational modes, phase(H1) approximately k*phase(H5) would suggest a lag ratio k.\")\nfor r in [12,16,20,24,28,32,40]:\n z=u[:,:r]\n a=(z.T@m[1]@z).numpy()\n b=(z.T@m[5]@z).numpy()\n ev,vec=eig(b.T,b)\n rows=[]\n for i,x in enumerate(ev):\n if not np.isfinite(x) or x.imag<=1e-5 or abs(abs(x)-1)>.12:\n continue\n v=vec[:,i]\n av=a@v\n atv=a.T@v\n overlap=np.vdot(av,atv)/(np.linalg.norm(av)*np.linalg.norm(atv)+1e-15)\n rows.append((float(np.angle(x)),float(np.angle(overlap)),float(abs(overlap))))\n print(\"\\nrank\",r,\"(H5 phase,H1 phase,mode agreement)\",np.round(rows,3).tolist())\n if rows:\n p,q,w=np.array(rows).T\n print(\"phase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4:\",\n [(k,round(float(np.average(np.cos(q-k*p),weights=w**4)),3))\n for k in [-3,-2,-1,1,2,3,4]])\n",
"out": "Relative-shift diagnostic using only generalized eigenvectors of QK weight products.\nFor shared rotational modes, phase(H1) approximately k*phase(H5) would suggest a lag ratio k.\n\nrank 12 (H5 phase,H1 phase,mode agreement) [[1.119, 0.703, 0.559], [1.909, 2.094, 0.746], [1.957, 2.244, 0.67], [2.47, -1.649, 0.722]]\nphase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.154), (-2, 0.141), (-1, -0.158), (1, 0.493), (2, 0.222), (3, -0.899), (4, 0.502)]\n\nrank 16 (H5 phase,H1 phase,mode agreement) [[0.944, -1.064, 0.252], [1.503, 0.037, 0.209], [2.092, 2.497, 0.759]]\nphase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, -0.784), (-2, 0.908), (-1, -0.109), (1, 0.898), (2, -0.131), (3, -0.8), (4, 0.907)]\n\nrank 20 (H5 phase,H1 phase,mode agreement) [[0.753, 0.182, 0.552], [1.356, 0.74, 0.57], [1.898, 2.17, 0.748], [2.226, -1.942, 0.736], [2.226, -1.956, 0.642], [2.437, -1.628, 0.898], [3.023, -2.646, 0.413], [2.79, -1.664, 0.348]]\nphase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.313), (-2, -0.522), (-1, 0.445), (1, -0.086), (2, 0.627), (3, -0.797), (4, 0.171)]\n\nrank 24 (H5 phase,H1 phase,mode agreement) [[1.055, -3.122, 0.399], [1.713, 2.482, 0.595], [1.88, -2.537, 0.09], [2.394, -1.529, 0.91], [2.661, -1.741, 0.721], [2.661, -1.943, 0.626]]\nphase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.815), (-2, -0.75), (-1, 0.516), (1, -0.404), (2, 0.833), (3, -0.791), (4, 0.369)]\n\nrank 28 (H5 phase,H1 phase,mode agreement) [[1.13, -2.598, 0.491], [1.6, 0.918, 0.418], [2.37, -1.723, 0.899], [2.559, -1.791, 0.762], [2.559, -2.104, 0.367]]\nphase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.729), (-2, -0.875), (-1, 0.694), (1, -0.478), (2, 0.834), (3, -0.787), (4, 0.458)]\n\nrank 32 (H5 phase,H1 phase,mode agreement) [[1.081, -2.056, 0.212], [2.508, -1.578, 0.689], [2.937, -0.866, 0.295], [2.777, -1.781, 0.33], [2.508, -1.675, 0.617]]\nphase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.907), (-2, -0.926), (-1, 0.603), (1, -0.549), (2, 0.917), (3, -0.945), (4, 0.626)]\n\nrank 40 (H5 phase,H1 phase,mode agreement) [[1.714, -2.585, 0.567], [2.52, -1.388, 0.773]]\nphase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.582), (-2, -0.527), (-1, 0.475), (1, -0.648), (2, 0.984), (3, -0.66), (4, 0.127)]\n"
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 9 of at most 100: ``` Relative-shift diagnostic using only generalized eigenvectors of QK weight products. For shared rotational modes, phase(H1) approximately k*phase(H5) would suggest a lag ratio k. rank 12 (H5 phase,H1 phase,mode agreement) [[1.119, 0.703, 0.559], [1.909, 2.094, 0.746], [1.957, 2.244, 0.67], [2.47, -1.649, 0.722]] phase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.154), (-2, 0.141), (-1, -0.158), (1, 0.493), (2, 0.222), (3, -0.899), (4, 0.502)] rank 16 (H5 phase,H1 phase,mode agreement) [[0.944, -1.064, 0.252], [1.503, 0.037, 0.209], [2.092, 2.497, 0.759]] phase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, -0.784), (-2, 0.908), (-1, -0.109), (1, 0.898), (2, -0.131), (3, -0.8), (4, 0.907)] rank 20 (H5 phase,H1 phase,mode agreement) [[0.753, 0.182, 0.552], [1.356, 0.74, 0.57], [1.898, 2.17, 0.748], [2.226, -1.942, 0.736], [2.226, -1.956, 0.642], [2.437, -1.628, 0.898], [3.023, -2.646, 0.413], [2.79, -1.664, 0.348]] phase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.313), (-2, -0.522), (-1, 0.445), (1, -0.086), (2, 0.627), (3, -0.797), (4, 0.171)] rank 24 (H5 phase,H1 phase,mode agreement) [[1.055, -3.122, 0.399], [1.713, 2.482, 0.595], [1.88, -2.537, 0.09], [2.394, -1.529, 0.91], [2.661, -1.741, 0.721], [2.661, -1.943, 0.626]] phase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.815), (-2, -0.75), (-1, 0.516), (1, -0.404), (2, 0.833), (3, -0.791), (4, 0.369)] rank 28 (H5 phase,H1 phase,mode agreement) [[1.13, -2.598, 0.491], [1.6, 0.918, 0.418], [2.37, -1.723, 0.899], [2.559, -1.791, 0.762], [2.559, -2.104, 0.367]] phase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.729), (-2, -0.875), (-1, 0.694), (1, -0.478), (2, 0.834), (3, -0.787), (4, 0.458)] rank 32 (H5 phase,H1 phase,mode agreement) [[1.081, -2.056, 0.212], [2.508, -1.578, 0.689], [2.937, -0.866, 0.295], [2.777, -1.781, 0.33], [2.508, -1.675, 0.617]] phase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.907), (-2, -0.926), (-1, 0.603), (1, -0.549), (2, 0.917), (3, -0.945), (4, 0.626)] rank 40 (H5 phase,H1 phase,mode agreement) [[1.714, -2.585, 0.567], [2.52, -1.388, 0.773]] phase agreement for H1/H5 lag ratios -3,-2,-1,1,2,3,4: [(-3, 0.582), (-2, -0.527), (-1, 0.475), (1, -0.648), (2, 0.984), (3, -0.66), (4, 0.127)] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
from sklearn.cluster import DBSCAN
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)
P=torch.eye(384)-torch.ones((384,384))/384
g=T(W["ln1_w.0"])
vs={}
bs={}
for j in [1,5]:
sl=slice(64*j,64*(j+1))
vs[j]=(T(W["W_v.0"])[sl]*g)@P
bs[j]=P@T(W["W_o.0"])[:,sl]@vs[j]
_,u=torch.linalg.eigh(sum(v.T@v for v in vs.values()))
u=u.flip(1)
f=(T(W["W_fc.0"])*T(W["ln2_w.0"]))@P
active=np.linalg.norm(W["W_fc.0"],axis=1)>.8
ids=np.flatnonzero(active)
print("Static MLP read-direction alignments for current / H5-value / H1-value channels.")
print("These are products of weight matrices, with no inputs or nonlinearities.")
for r in [20,28,40]:
z=u[:,:r]
arr=[(a@z).numpy()[active] for a in [f,f@bs[5],f@bs[1]]]
norms=[np.linalg.norm(a,axis=1) for a in arr]
nn=[a/(n[:,None]+1e-12) for a,n in zip(arr,norms)]
cs=np.stack([(nn[0]*nn[1]).sum(1),(nn[0]*nn[2]).sum(1),
(nn[1]*nn[2]).sum(1)],1)
print("\nrank",r)
print("Channel norm quantiles",np.round([np.quantile(n,[.1,.5,.9]) for n in norms],3))
print("Pair cosine quantiles",np.round(np.quantile(cs,[.01,.1,.5,.9,.99],axis=0),3))
for threshold in [.5,.65,.8]:
mask=cs.min(1)>threshold
print("all-three pair cosines >",threshold,"count",int(mask.sum()))
order=np.argsort(cs.min(1))[::-1][:15]
print("best same-direction neurons (index, 3 cosines, 3 norms):")
for i in order:
print(int(ids[i]),np.round(cs[i],3).tolist(),
np.round([n[i] for n in norms],3).tolist())
if r==28:
selected=np.flatnonzero(cs.min(1)>.5)
if len(selected):
sim=nn[0][selected]@nn[0][selected].T
print("Similarities among strongest repeated-direction candidates:")
print(np.round(sim[:20,:20],2))
for a,b,label in [(0,1,"current/H5"),(0,2,"current/H1"),(1,2,"H5/H1")]:
# Permutation baseline for row correspondence.
rng=np.random.default_rng(0)
baseline=np.sum(nn[a]*nn[b][rng.permutation(len(ids))],axis=1)
actual=np.sum(nn[a]*nn[b],axis=1)
print(label,"actual/random mean",
round(float(actual.mean()),4),round(float(baseline.mean()),4),
"actual/random fraction > .65",
round(float((actual>.65).mean()),4),round(float((baseline>.65).mean()),4))
```
{
"t": 1792714087.0696974,
"s": 21.3,
"code": "import numpy as np\nimport torch\nfrom sklearn.cluster import DBSCAN\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)\nP=torch.eye(384)-torch.ones((384,384))/384\ng=T(W[\"ln1_w.0\"])\nvs={}\nbs={}\nfor j in [1,5]:\n sl=slice(64*j,64*(j+1))\n vs[j]=(T(W[\"W_v.0\"])[sl]*g)@P\n bs[j]=P@T(W[\"W_o.0\"])[:,sl]@vs[j]\n_,u=torch.linalg.eigh(sum(v.T@v for v in vs.values()))\nu=u.flip(1)\nf=(T(W[\"W_fc.0\"])*T(W[\"ln2_w.0\"]))@P\nactive=np.linalg.norm(W[\"W_fc.0\"],axis=1)>.8\nids=np.flatnonzero(active)\n\nprint(\"Static MLP read-direction alignments for current / H5-value / H1-value channels.\")\nprint(\"These are products of weight matrices, with no inputs or nonlinearities.\")\nfor r in [20,28,40]:\n z=u[:,:r]\n arr=[(a@z).numpy()[active] for a in [f,f@bs[5],f@bs[1]]]\n norms=[np.linalg.norm(a,axis=1) for a in arr]\n nn=[a/(n[:,None]+1e-12) for a,n in zip(arr,norms)]\n cs=np.stack([(nn[0]*nn[1]).sum(1),(nn[0]*nn[2]).sum(1),\n (nn[1]*nn[2]).sum(1)],1)\n print(\"\\nrank\",r)\n print(\"Channel norm quantiles\",np.round([np.quantile(n,[.1,.5,.9]) for n in norms],3))\n print(\"Pair cosine quantiles\",np.round(np.quantile(cs,[.01,.1,.5,.9,.99],axis=0),3))\n for threshold in [.5,.65,.8]:\n mask=cs.min(1)>threshold\n print(\"all-three pair cosines >\",threshold,\"count\",int(mask.sum()))\n order=np.argsort(cs.min(1))[::-1][:15]\n print(\"best same-direction neurons (index, 3 cosines, 3 norms):\")\n for i in order:\n print(int(ids[i]),np.round(cs[i],3).tolist(),\n np.round([n[i] for n in norms],3).tolist())\n if r==28:\n selected=np.flatnonzero(cs.min(1)>.5)\n if len(selected):\n sim=nn[0][selected]@nn[0][selected].T\n print(\"Similarities among strongest repeated-direction candidates:\")\n print(np.round(sim[:20,:20],2))\n for a,b,label in [(0,1,\"current/H5\"),(0,2,\"current/H1\"),(1,2,\"H5/H1\")]:\n # Permutation baseline for row correspondence.\n rng=np.random.default_rng(0)\n baseline=np.sum(nn[a]*nn[b][rng.permutation(len(ids))],axis=1)\n actual=np.sum(nn[a]*nn[b],axis=1)\n print(label,\"actual/random mean\",\n round(float(actual.mean()),4),round(float(baseline.mean()),4),\n \"actual/random fraction > .65\",\n round(float((actual>.65).mean()),4),round(float((baseline>.65).mean()),4))\n",
"out": "Static MLP read-direction alignments for current / H5-value / H1-value channels.\nThese are products of weight matrices, with no inputs or nonlinearities.\n\nrank 20\nChannel norm quantiles [[0.353 0.706 1.196]\n [0.179 0.448 0.697]\n [0.116 0.361 0.605]]\nPair cosine quantiles [[-0.918 -0.776 -0.57 ]\n [-0.663 -0.443 -0.222]\n [-0.082 0.15 0.252]\n [ 0.494 0.589 0.608]\n [ 0.747 0.744 0.79 ]]\nall-three pair cosines > 0.5 count 17\nall-three pair cosines > 0.65 count 2\nall-three pair cosines > 0.8 count 0\nbest same-direction neurons (index, 3 cosines, 3 norms):\n14 [0.758, 0.755, 0.801] [0.386, 0.398, 0.262]\n643 [0.703, 0.679, 0.746] [1.141, 0.647, 0.374]\n795 [0.612, 0.631, 0.673] [0.473, 0.452, 0.392]\n1065 [0.622, 0.609, 0.638] [0.402, 0.341, 0.338]\n310 [0.574, 0.59, 0.705] [0.39, 0.569, 0.259]\n799 [0.693, 0.573, 0.692] [0.442, 0.5, 0.417]\n129 [0.583, 0.567, 0.598] [0.518, 0.477, 0.371]\n581 [0.761, 0.566, 0.87] [0.304, 0.511, 0.32]\n1265 [0.562, 0.7, 0.62] [0.491, 0.392, 0.387]\n85 [0.555, 0.677, 0.573] [0.271, 0.344, 0.2]\n26 [0.547, 0.563, 0.676] [0.361, 0.392, 0.383]\n325 [0.54, 0.653, 0.677] [0.616, 0.291, 0.333]\n1446 [0.54, 0.641, 0.735] [0.558, 0.288, 0.351]\n251 [0.669, 0.533, 0.601] [0.38, 0.374, 0.225]\n1354 [0.55, 0.518, 0.616] [0.449, 0.275, 0.295]\n\nrank 28\nChannel norm quantiles [[0.426 0.856 1.341]\n [0.189 0.471 0.731]\n [0.134 0.386 0.642]]\nPair cosine quantiles [[-0.898 -0.722 -0.509]\n [-0.629 -0.432 -0.183]\n [-0.084 0.151 0.22 ]\n [ 0.471 0.558 0.586]\n [ 0.711 0.697 0.743]]\nall-three pair cosines > 0.5 count 13\nall-three pair cosines > 0.65 count 1\nall-three pair cosines > 0.8 count 0\nbest same-direction neurons (index, 3 cosines, 3 norms):\n14 [0.737, 0.682, 0.745] [0.415, 0.407, 0.268]\n795 [0.652, 0.623, 0.64] [0.524, 0.498, 0.412]\n251 [0.662, 0.636, 0.605] [0.474, 0.392, 0.257]\n1065 [0.668, 0.589, 0.639] [0.484, 0.363, 0.349]\n1265 [0.62, 0.647, 0.588] [0.59, 0.44, 0.403]\n799 [0.644, 0.582, 0.678] [0.492, 0.525, 0.43]\n310 [0.572, 0.601, 0.669] [0.478, 0.593, 0.276]\n129 [0.557, 0.603, 0.607] [0.618, 0.495, 0.398]\n753 [0.665, 0.555, 0.62] [0.384, 0.417, 0.363]\n1206 [0.542, 0.732, 0.565] [0.572, 0.446, 0.359]\n581 [0.725, 0.524, 0.78] [0.426, 0.553, 0.348]\n85 [0.521, 0.55, 0.533] [0.315, 0.356, 0.22]\n26 [0.511, 0.55, 0.661] [0.387, 0.404, 0.395]\n325 [0.497, 0.652, 0.622] [0.641, 0.323, 0.34]\n1354 [0.489, 0.566, 0.521] [0.525, 0.305, 0.32]\nSimilarities among strongest repeated-direction candidates:\n[[ 1. -0.01 0.09 0.04 0.31 0.31 -0.01 0.45 0.33 -0.2 -0.07 0.28 0.57]\n [-0.01 1. 0.35 0.03 0.05 0.27 0.05 -0.06 0.23 0.31 0.18 -0.33 0.03]\n [ 0.09 0.35 1. -0.02 0.17 0.2 0.61 0.08 0.16 0. -0.09 -0.15 0.26]\n [ 0.04 0.03 -0.02 1. 0.22 0.07 -0.4 0.14 -0.25 -0.06 -0.59 0.13 -0.24]\n [ 0.31 0.05 0.17 0.22 1. 0.54 0.21 0.58 0.18 -0.1 0.22 0.04 0.34]\n [ 0.31 0.27 0.2 0.07 0.54 1. 0.22 0.36 0.32 -0.06 0.2 -0.18 0.21]\n [-0.01 0.05 0.61 -0.4 0.21 0.22 1. 0.21 0.04 -0.05 0.13 -0.29 0.33]\n [ 0.45 -0.06 0.08 0.14 0.58 0.36 0.21 1. 0.22 -0.19 0.06 -0.08 0.67]\n [ 0.33 0.23 0.16 -0.25 0.18 0.32 0.04 0.22 1. 0.48 0.3 -0.05 0.46]\n [-0.2 0.31 0. -0.06 -0.1 -0.06 -0.05 -0.19 0.48 1. 0.28 0.18 -0.11]\n [-0.07 0.18 -0.09 -0.59 0.22 0.2 0.13 0.06 0.3 0.28 1. -0.26 0.1 ]\n [ 0.28 -0.33 -0.15 0.13 0.04 -0.18 -0.29 -0.08 -0.05 0.18 -0.26 1. 0.01]\n [ 0.57 0.03 0.26 -0.24 0.34 0.21 0.33 0.67 0.46 -0.11 0.1 0.01 1. ]]\ncurrent/H5 actual/random mean -0.0771 -0.0008 actual/random fraction > .65 0.032 0.0038\ncurrent/H1 actual/random mean 0.1083 -0.0077 actual/random fraction > .65 0.0339 0.0\nH5/H1 actual/random mean 0.2086 0.0146 actual/random fraction > .65 0.0452 0.0\n\nrank 40\nChannel norm quantiles [[0.484 0.898 1.405]\n [0.194 0.479 0.737]\n [0.137 0.39 0.646]]\nPair cosine quantiles [[-0.886 -0.708 -0.508]\n [-0.619 -0.439 -0.174]\n [-0.07 0.146 0.214]\n [ 0.426 0.526 0.57 ]\n [ 0.669 0.673 0.729]]\nall-thre"
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 10 of at most 100: ``` Static MLP read-direction alignments for current / H5-value / H1-value channels. These are products of weight matrices, with no inputs or nonlinearities. rank 20 Channel norm quantiles [[0.353 0.706 1.196] [0.179 0.448 0.697] [0.116 0.361 0.605]] Pair cosine quantiles [[-0.918 -0.776 -0.57 ] [-0.663 -0.443 -0.222] [-0.082 0.15 0.252] [ 0.494 0.589 0.608] [ 0.747 0.744 0.79 ]] all-three pair cosines > 0.5 count 17 all-three pair cosines > 0.65 count 2 all-three pair cosines > 0.8 count 0 best same-direction neurons (index, 3 cosines, 3 norms): 14 [0.758, 0.755, 0.801] [0.386, 0.398, 0.262] 643 [0.703, 0.679, 0.746] [1.141, 0.647, 0.374] 795 [0.612, 0.631, 0.673] [0.473, 0.452, 0.392] 1065 [0.622, 0.609, 0.638] [0.402, 0.341, 0.338] 310 [0.574, 0.59, 0.705] [0.39, 0.569, 0.259] 799 [0.693, 0.573, 0.692] [0.442, 0.5, 0.417] 129 [0.583, 0.567, 0.598] [0.518, 0.477, 0.371] 581 [0.761, 0.566, 0.87] [0.304, 0.511, 0.32] 1265 [0.562, 0.7, 0.62] [0.491, 0.392, 0.387] 85 [0.555, 0.677, 0.573] [0.271, 0.344, 0.2] 26 [0.547, 0.563, 0.676] [0.361, 0.392, 0.383] 325 [0.54, 0.653, 0.677] [0.616, 0.291, 0.333] 1446 [0.54, 0.641, 0.735] [0.558, 0.288, 0.351] 251 [0.669, 0.533, 0.601] [0.38, 0.374, 0.225] 1354 [0.55, 0.518, 0.616] [0.449, 0.275, 0.295] rank 28 Channel norm quantiles [[0.426 0.856 1.341] [0.189 0.471 0.731] [0.134 0.386 0.642]] Pair cosine quantiles [[-0.898 -0.722 -0.509] [-0.629 -0.432 -0.183] [-0.084 0.151 0.22 ] [ 0.471 0.558 0.586] [ 0.711 0.697 0.743]] all-three pair cosines > 0.5 count 13 all-three pair cosines > 0.65 count 1 all-three pair cosines > 0.8 count 0 best same-direction neurons (index, 3 cosines, 3 norms): 14 [0.737, 0.682, 0.745] [0.415, 0.407, 0.268] 795 [0.652, 0.623, 0.64] [0.524, 0.498, 0.412] 251 [0.662, 0.636, 0.605] [0.474, 0.392, 0.257] 1065 [0.668, 0.589, 0.639] [0.484, 0.363, 0.349] 1265 [0.62, 0.647, 0.588] [0.59, 0.44, 0.403] 799 [0.644, 0.582, 0.678] [0.492, 0.525, 0.43] 310 [0.572, 0.601, 0.669] [0.478, 0.593, 0.276] 129 [0.557, 0.603, 0.607] [0.618, 0.495, 0.398] 753 [0.665, 0.555, 0.62] [0.384, 0.417, 0.363] 1206 [0.542, 0.732, 0.565] [0.572, 0.446, 0.359] 581 [0.725, 0.524, 0.78] [0.426, 0.553, 0.348] 85 [0.521, 0.55, 0.533] [0.315, 0.356, 0.22] 26 [0.511, 0.55, 0.661] [0.387, 0.404, 0.395] 325 [0.497, 0.652, 0.622] [0.641, 0.323, 0.34] 1354 [0.489, 0.566, 0.521] [0.525, 0.305, 0.32] Similarities among strongest repeated-direction candidates: [[ 1. -0.01 0.09 0.04 0.31 0.31 -0.01 0.45 0.33 -0.2 -0.07 0.28 0.57] [-0.01 1. 0.35 0.03 0.05 0.27 0.05 -0.06 0.23 0.31 0.18 -0.33 0.03] [ 0.09 0.35 1. -0.02 0.17 0.2 0.61 0.08 0.16 0. -0.09 -0.15 0.26] [ 0.04 0.03 -0.02 1. 0.22 0.07 -0.4 0.14 -0.25 -0.06 -0.59 0.13 -0.24] [ 0.31 0.05 0.17 0.22 1. 0.54 0.21 0.58 0.18 -0.1 0.22 0.04 0.34] [ 0.31 0.27 0.2 0.07 0.54 1. 0.22 0.36 0.32 -0.06 0.2 -0.18 0.21] [-0.01 0.05 0.61 -0.4 0.21 0.22 1. 0.21 0.04 -0.05 0.13 -0.29 0.33] [ 0.45 -0.06 0.08 0.14 0.58 0.36 0.21 1. 0.22 -0.19 0.06 -0.08 0.67] [ 0.33 0.23 0.16 -0.25 0.18 0.32 0.04 0.22 1. 0.48 0.3 -0.05 0.46] [-0.2 0.31 0. -0.06 -0.1 -0.06 -0.05 -0.19 0.48 1. 0.28 0.18 -0.11] [-0.07 0.18 -0.09 -0.59 0.22 0.2 0.13 0.06 0.3 0.28 1. -0.26 0.1 ] [ 0.28 -0.33 -0.15 0.13 0.04 -0.18 -0.29 -0.08 -0.05 0.18 -0.26 1. 0.01] [ 0.57 0.03 0.26 -0.24 0.34 0.21 0.33 0.67 0.46 -0.11 0.1 0.01 1. ]] current/H5 actual/random mean -0.0771 -0.0008 actual/random fraction > .65 0.032 0.0038 current/H1 actual/random mean 0.1083 -0.0077 actual/random fraction > .65 0.0339 0.0 H5/H1 actual/random mean 0.2086 0.0146 actual/random fraction > .65 0.0452 0.0 rank 40 Channel norm quantiles [[0.484 0.898 1.405] [0.194 0.479 0.737] [0.137 0.39 0.646]] Pair cosine quantiles [[-0.886 -0.708 -0.508] [-0.619 -0.439 -0.174] [-0.07 0.146 0.214] [ 0.426 0.526 0.57 ] [ 0.669 0.673 0.729]] all-three pair cosines > 0.5 count 9 all-three pair cosines > 0.65 count 0 all-three pair cosines > 0.8 count 0 best same-direction neurons (index, 3 cosines, 3 norms): 14 [0.676, 0.632, 0.734] [0.464, 0.41, 0.271] 251 [0.614, 0.607, 0.593] [0.521, 0.397, 0.262] 795 [0.631, 0.587, 0.615] [0.559, 0.509, 0.415] 1265 [0.599, 0.595, 0.56] [0.626, 0.449, 0.407] 799 [0.56, 0.541, 0.642] [0.525, 0.541, 0.434] 1065 [0.63, 0.537, 0.621] [0.542, 0.372, 0.351] 129 [0.524, 0.587, 0.592] [0.646, 0.501, 0.401] 310 [0.524, 0.537, 0.664] [0.562, 0.597, 0.281] 1206 [0.522, 0.716, 0.548] [0.593, 0.449, 0.364] 753 [0.624, 0.494, 0.603] [0.425, 0.421, 0.367] 325 [0.478, 0.625, 0.601] [0.696, 0.332, 0.344] 570 [0.59, 0.496, 0.475] [0.516, 0.476, 0.24] 311 [0.509, 0.651, 0.471] [0.538, 0.451, 0.456] 85 [0.464, 0.525, 0.522] [0.338, 0.36, 0.221] 581 [0.646, 0.463, 0.764] [0.484, 0.561, 0.351] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
```python
import numpy as np
import torch
torch.set_num_threads(4)
torch.set_default_dtype(torch.float64)
T=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)
P=torch.eye(384)-torch.ones((384,384))/384
g=T(W["ln1_w.0"])
v,b={},{}
for j in [1,5]:
sl=slice(64*j,64*(j+1))
v[j]=(T(W["W_v.0"])[sl]*g)@P
b[j]=P@T(W["W_o.0"])[:,sl]@v[j]
_,u=torch.linalg.eigh(sum(x.T@x for x in v.values()))
z=u.flip(1)[:,:28]
f=(T(W["W_fc.0"])*T(W["ln2_w.0"]))@P
active=T(np.linalg.norm(W["W_fc.0"],axis=1)>.8).bool()
a=[(x@z)[active] for x in [f,f@b[5],f@b[1]]]
a=[x/x.norm(dim=1,keepdim=True) for x in a]
c=[]
print("Cross-covariance spectra of normalized MLP weight read-directions:")
for i,j in [(0,1),(0,2),(1,2)]:
m=a[i].T@a[j]/len(a[i])
c.append(m)
se=torch.linalg.eigvalsh((m+m.T)/2)
sv=torch.linalg.svdvals(m)
print((i,j),"trace",round(float(torch.trace(m)),4),
"SV",np.round(sv[:12].numpy(),4),
"sym eigenvalues",np.round(se.numpy(),4))
print("\nCosines between the three symmetric cross-covariance matrices:")
cs=[(x+x.T)/2 for x in c]
print(np.round([[float((x*y).sum()/(x.norm()*y.norm())) for y in cs] for x in cs],3))
alt=cs[0]+cs[2]-cs[1]
e,u=torch.linalg.eigh(alt)
print("Alternation-contrast eigenvalues",np.round(e.numpy(),4))
for i in range(4):
d=u[:,i]
print("mode",i,"per-pair quadratic coefficients",
np.round([float(d@m@d) for m in cs],4))
```
{
"t": 1792714168.638603,
"s": 9.93,
"code": "import numpy as np\nimport torch\ntorch.set_num_threads(4)\ntorch.set_default_dtype(torch.float64)\nT=lambda a: torch.as_tensor(np.asarray(a),dtype=torch.float64)\nP=torch.eye(384)-torch.ones((384,384))/384\ng=T(W[\"ln1_w.0\"])\nv,b={},{}\nfor j in [1,5]:\n sl=slice(64*j,64*(j+1))\n v[j]=(T(W[\"W_v.0\"])[sl]*g)@P\n b[j]=P@T(W[\"W_o.0\"])[:,sl]@v[j]\n_,u=torch.linalg.eigh(sum(x.T@x for x in v.values()))\nz=u.flip(1)[:,:28]\nf=(T(W[\"W_fc.0\"])*T(W[\"ln2_w.0\"]))@P\nactive=T(np.linalg.norm(W[\"W_fc.0\"],axis=1)>.8).bool()\na=[(x@z)[active] for x in [f,f@b[5],f@b[1]]]\na=[x/x.norm(dim=1,keepdim=True) for x in a]\nc=[]\nprint(\"Cross-covariance spectra of normalized MLP weight read-directions:\")\nfor i,j in [(0,1),(0,2),(1,2)]:\n m=a[i].T@a[j]/len(a[i])\n c.append(m)\n se=torch.linalg.eigvalsh((m+m.T)/2)\n sv=torch.linalg.svdvals(m)\n print((i,j),\"trace\",round(float(torch.trace(m)),4),\n \"SV\",np.round(sv[:12].numpy(),4),\n \"sym eigenvalues\",np.round(se.numpy(),4))\nprint(\"\\nCosines between the three symmetric cross-covariance matrices:\")\ncs=[(x+x.T)/2 for x in c]\nprint(np.round([[float((x*y).sum()/(x.norm()*y.norm())) for y in cs] for x in cs],3))\nalt=cs[0]+cs[2]-cs[1]\ne,u=torch.linalg.eigh(alt)\nprint(\"Alternation-contrast eigenvalues\",np.round(e.numpy(),4))\nfor i in range(4):\n d=u[:,i]\n print(\"mode\",i,\"per-pair quadratic coefficients\",\n np.round([float(d@m@d) for m in cs],4))\n",
"out": "Cross-covariance spectra of normalized MLP weight read-directions:\n(0, 1) trace -0.0771 SV [0.0481 0.0233 0.0208 0.0199 0.0177 0.0174 0.016 0.0146 0.0123 0.0113 0.01 0.0095] sym eigenvalues [-0.0304 -0.0183 -0.0138 -0.0122 -0.0117 -0.0105 -0.0088 -0.0078 -0.0069 -0.0059 -0.005 -0.0047 -0.0043 -0.0034 -0.0029 -0.0021 -0.001 -0.0004\n 0.0011 0.0023 0.0025 0.0032 0.0041 0.0053 0.0099 0.0114 0.0136 0.0196]\n(0, 2) trace 0.1083 SV [0.0493 0.028 0.0259 0.024 0.0205 0.02 0.0165 0.0162 0.0148 0.0135 0.0128 0.0119] sym eigenvalues [-0.0305 -0.0131 -0.011 -0.0071 -0.0061 -0.0038 -0.0031 -0.0023 -0.0018 -0.0007 0.0012 0.0017 0.0024 0.0039 0.0049 0.0051 0.0061 0.008\n 0.0089 0.0093 0.0114 0.0121 0.0147 0.0158 0.0169 0.0178 0.0224 0.0253]\n(1, 2) trace 0.2086 SV [0.0418 0.0349 0.0305 0.0278 0.0249 0.0216 0.0203 0.0179 0.0163 0.0147 0.0129 0.0125] sym eigenvalues [-0.0169 -0.0072 -0.0054 -0.0033 -0.0023 -0.0009 -0.0009 -0.0008 0.0004 0.0011 0.0016 0.0023 0.003 0.0048 0.0066 0.0075 0.0085 0.009\n 0.0108 0.0126 0.0144 0.0164 0.0175 0.0202 0.0221 0.0272 0.0287 0.0319]\n\nCosines between the three symmetric cross-covariance matrices:\n[[ 1. 0.017 -0.166]\n [ 0.017 1. 0.27 ]\n [-0.166 0.27 1. ]]\nAlternation-contrast eigenvalues [-0.038 -0.0277 -0.0191 -0.0177 -0.0166 -0.0134 -0.0119 -0.0089 -0.0087 -0.0078 -0.0056 -0.0044 -0.0027 0. 0.0004 0.0026 0.0048 0.0064\n 0.0075 0.0096 0.0104 0.0149 0.015 0.0182 0.0207 0.0232 0.0322 0.0397]\nmode 0 per-pair quadratic coefficients [-0.0194 0.0108 -0.0078]\nmode 1 per-pair quadratic coefficients [-0.0131 0.0108 -0.0038]\nmode 2 per-pair quadratic coefficients [-0.0063 0.0136 0.0008]\nmode 3 per-pair quadratic coefficients [-0.0072 0.0135 0.003 ]\n"
}Recorded prompt · Raw CLI events · Settings and audit
OUTPUT of run 11 of at most 100: ``` Cross-covariance spectra of normalized MLP weight read-directions: (0, 1) trace -0.0771 SV [0.0481 0.0233 0.0208 0.0199 0.0177 0.0174 0.016 0.0146 0.0123 0.0113 0.01 0.0095] sym eigenvalues [-0.0304 -0.0183 -0.0138 -0.0122 -0.0117 -0.0105 -0.0088 -0.0078 -0.0069 -0.0059 -0.005 -0.0047 -0.0043 -0.0034 -0.0029 -0.0021 -0.001 -0.0004 0.0011 0.0023 0.0025 0.0032 0.0041 0.0053 0.0099 0.0114 0.0136 0.0196] (0, 2) trace 0.1083 SV [0.0493 0.028 0.0259 0.024 0.0205 0.02 0.0165 0.0162 0.0148 0.0135 0.0128 0.0119] sym eigenvalues [-0.0305 -0.0131 -0.011 -0.0071 -0.0061 -0.0038 -0.0031 -0.0023 -0.0018 -0.0007 0.0012 0.0017 0.0024 0.0039 0.0049 0.0051 0.0061 0.008 0.0089 0.0093 0.0114 0.0121 0.0147 0.0158 0.0169 0.0178 0.0224 0.0253] (1, 2) trace 0.2086 SV [0.0418 0.0349 0.0305 0.0278 0.0249 0.0216 0.0203 0.0179 0.0163 0.0147 0.0129 0.0125] sym eigenvalues [-0.0169 -0.0072 -0.0054 -0.0033 -0.0023 -0.0009 -0.0009 -0.0008 0.0004 0.0011 0.0016 0.0023 0.003 0.0048 0.0066 0.0075 0.0085 0.009 0.0108 0.0126 0.0144 0.0164 0.0175 0.0202 0.0221 0.0272 0.0287 0.0319] Cosines between the three symmetric cross-covariance matrices: [[ 1. 0.017 -0.166] [ 0.017 1. 0.27 ] [-0.166 0.27 1. ]] Alternation-contrast eigenvalues [-0.038 -0.0277 -0.0191 -0.0177 -0.0166 -0.0134 -0.0119 -0.0089 -0.0087 -0.0078 -0.0056 -0.0044 -0.0027 0. 0.0004 0.0026 0.0048 0.0064 0.0075 0.0096 0.0104 0.0149 0.015 0.0182 0.0207 0.0232 0.0322 0.0397] mode 0 per-pair quadratic coefficients [-0.0194 0.0108 -0.0078] mode 1 per-pair quadratic coefficients [-0.0131 0.0108 -0.0038] mode 2 per-pair quadratic coefficients [-0.0063 0.0136 0.0008] mode 3 per-pair quadratic coefficients [-0.0072 0.0135 0.003 ] ```
Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`.
## Observations
These are **weight-based hypotheses, not measured attention patterns**. Missing embeddings make corpus identification and exact attention distances uncertain.
Define \(M=PDW_q^\top W_kDP/8\) and \(B=PW_oW_vDP\) for each head, where \(D=\operatorname{diag}(\texttt{ln1_w})\) and \(P\) subtracts the coordinate mean. These are linear weight products, not layer evaluations.
- **A broad aggregation head:** L0H4 has \(\|M\|_F=0.0093\), versus **0.8775** for L0H1 and **0.9704** for L0H5. Its \(B\) has dominant singular values **13.877, 9.644**, suggesting weakly selective aggregation into a small feature space.
- **Two likely positional character-routing heads:** L0H1 and L0H5 read largely outside their shared value-input subspace. Their generalized-eigenvalue phases support an approximately **2:1 shift relationship**—consistent with two-characters-back and previous-character heads, respectively. The absolute offsets remain inferred.
- **Directional, induction-like composition:** Let
\[
C_k=\frac{\operatorname{tr}(M_{\rm later}B_{\rm earlier})}
{\|M_{\rm later}\|_F\|B_{\rm earlier}\|_F},
\qquad
C_q=\frac{\operatorname{tr}(M_{\rm later}B_{\rm earlier}^{\top})}
{\|M_{\rm later}\|_F\|B_{\rm earlier}\|_F}.
\]
The strongest relevant pairs are:
- L0H5 → L2H1: **\(C_k=0.0776,\ C_q=-0.0398\)**
- L2H2 → L3H1: **\(0.1104,\ -0.0164\)**
- L2H2 → L3H5: **\(0.1836,\ -0.0354\)**
This key-specific matching supports successor/context-copying circuits, although it does not establish their complete behavior.
- **Shared routing in layer 1:** The leading three output directions of L0H4 account for **53–68%** of key-side QK energy in L1H1–L1H5. Their query-bias-induced key preferences also align strongly. This suggests shared positional routing, but does not independently establish locality.
## Heads
Here, \(r_{90}\) is the number of singular values accounting for 90% of \(M\)'s squared norm.
L0H0: **unclear** — \(r_{90}=2\); low-dimensional selector, but no identifiable distance.
L0H1: **local** — phase relationship with L0H5 suggests a two-character shift.
L0H2: **unclear** — \(r_{90}=5\); mixed positional and content-selective weight directions.
L0H3: **unclear** — \(r_{90}=3\); structured selector without a reliable lag signature.
L0H4: **broad** — exceptionally weak QK product and strong, low-dimensional OV aggregation.
L0H5: **previous** — strongest one-character-shift candidate; also supplies the L2H1 matching circuit.
L1H0: **unclear** — weaker shared-routing component; distance and selectivity remain ambiguous.
L1H1: **local** — tentative; 63% key-side energy reads L0H4’s leading output subspace.
L1H2: **local** — tentative; corresponding fraction 63%, with substantial downstream matching composition.
L1H3: **local** — tentative; strongest shared-routing fraction, 68%.
L1H4: **local** — tentative; 65% shared-routing fraction and aligned key preference.
L1H5: **local** — tentative; 53% shared-routing fraction; exact offset unresolved.
L2H0: **unclear** — distributed QK structure, \(r_{90}=39\), without a decisive routing signature.
L2H1: **copy** — positive, key-specific composition with the likely previous-character head L0H5.
L2H2: **unclear** — important precursor to both late copy candidates; its own attention distance is unverified.
L2H3: **unclear** — some shared positional-carrier dependence, insufficient to distinguish local from broad.
L2H4: **unclear** — mixed routing structure; no decisive predecessor or induction signature.
L2H5: **unclear** — distributed QK structure, \(r_{90}=42\).
L3H0: **unclear** — modest composition signals, not enough to establish copying or locality.
L3H1: **copy** — strong directional composition with L2H2: \(C_k=0.1104\).
L3H2: **unclear** — distributed selector without a clear distance or copying signature.
L3H3: **unclear** — weaker QK magnitude alone cannot establish broad attention.
L3H4: **unclear** — no sufficiently diagnostic composition.
L3H5: **copy** — strongest induction-like candidate: \(C_k=0.1836\), versus \(C_q=-0.0354\).
## Reasoning
The likely short-offset character routing and directional copying circuits favor **as-is text** substantially over independently shuffled lines.
I weakly favor **Python**: repeated identifiers and recurring code fragments offer a particularly useful role for this hierarchy of local features and context matching. English is a substantial alternative; these circuits are not code-specific. The corpus judgment is considerably weaker than the order judgment, and no character identities or keywords were recovered.
corpus: python
order: as is
probabilities: python=0.56 romaji=0.13 english=0.31; as is=0.96 shuffled=0.04## Observations
These are **weight-based hypotheses, not measured attention patterns**. Missing embeddings make corpus identification and exact attention distances uncertain.
Define \(M=PDW_q^\top W_kDP/8\) and \(B=PW_oW_vDP\) for each head, where \(D=\operatorname{diag}(\texttt{ln1_w})\) and \(P\) subtracts the coordinate mean. These are linear weight products, not layer evaluations.
- **A broad aggregation head:** L0H4 has \(\|M\|_F=0.0093\), versus **0.8775** for L0H1 and **0.9704** for L0H5. Its \(B\) has dominant singular values **13.877, 9.644**, suggesting weakly selective aggregation into a small feature space.
- **Two likely positional character-routing heads:** L0H1 and L0H5 read largely outside their shared value-input subspace. Their generalized-eigenvalue phases support an approximately **2:1 shift relationship**—consistent with two-characters-back and previous-character heads, respectively. The absolute offsets remain inferred.
- **Directional, induction-like composition:** Let
\[
C_k=\frac{\operatorname{tr}(M_{\rm later}B_{\rm earlier})}
{\|M_{\rm later}\|_F\|B_{\rm earlier}\|_F},
\qquad
C_q=\frac{\operatorname{tr}(M_{\rm later}B_{\rm earlier}^{\top})}
{\|M_{\rm later}\|_F\|B_{\rm earlier}\|_F}.
\]
The strongest relevant pairs are:
- L0H5 → L2H1: **\(C_k=0.0776,\ C_q=-0.0398\)**
- L2H2 → L3H1: **\(0.1104,\ -0.0164\)**
- L2H2 → L3H5: **\(0.1836,\ -0.0354\)**
This key-specific matching supports successor/context-copying circuits, although it does not establish their complete behavior.
- **Shared routing in layer 1:** The leading three output directions of L0H4 account for **53–68%** of key-side QK energy in L1H1–L1H5. Their query-bias-induced key preferences also align strongly. This suggests shared positional routing, but does not independently establish locality.
## Heads
Here, \(r_{90}\) is the number of singular values accounting for 90% of \(M\)'s squared norm.
L0H0: **unclear** — \(r_{90}=2\); low-dimensional selector, but no identifiable distance.
L0H1: **local** — phase relationship with L0H5 suggests a two-character shift.
L0H2: **unclear** — \(r_{90}=5\); mixed positional and content-selective weight directions.
L0H3: **unclear** — \(r_{90}=3\); structured selector without a reliable lag signature.
L0H4: **broad** — exceptionally weak QK product and strong, low-dimensional OV aggregation.
L0H5: **previous** — strongest one-character-shift candidate; also supplies the L2H1 matching circuit.
L1H0: **unclear** — weaker shared-routing component; distance and selectivity remain ambiguous.
L1H1: **local** — tentative; 63% key-side energy reads L0H4’s leading output subspace.
L1H2: **local** — tentative; corresponding fraction 63%, with substantial downstream matching composition.
L1H3: **local** — tentative; strongest shared-routing fraction, 68%.
L1H4: **local** — tentative; 65% shared-routing fraction and aligned key preference.
L1H5: **local** — tentative; 53% shared-routing fraction; exact offset unresolved.
L2H0: **unclear** — distributed QK structure, \(r_{90}=39\), without a decisive routing signature.
L2H1: **copy** — positive, key-specific composition with the likely previous-character head L0H5.
L2H2: **unclear** — important precursor to both late copy candidates; its own attention distance is unverified.
L2H3: **unclear** — some shared positional-carrier dependence, insufficient to distinguish local from broad.
L2H4: **unclear** — mixed routing structure; no decisive predecessor or induction signature.
L2H5: **unclear** — distributed QK structure, \(r_{90}=42\).
L3H0: **unclear** — modest composition signals, not enough to establish copying or locality.
L3H1: **copy** — strong directional composition with L2H2: \(C_k=0.1104\).
L3H2: **unclear** — distributed selector without a clear distance or copying signature.
L3H3: **unclear** — weaker QK magnitude alone cannot establish broad attention.
L3H4: **unclear** — no sufficiently diagnostic composition.
L3H5: **copy** — strongest induction-like candidate: \(C_k=0.1836\), versus \(C_q=-0.0354\).
## Reasoning
The likely short-offset character routing and directional copying circuits favor **as-is text** substantially over independently shuffled lines.
I weakly favor **Python**: repeated identifiers and recurring code fragments offer a particularly useful role for this hierarchy of local features and context matching. English is a substantial alternative; these circuits are not code-specific. The corpus judgment is considerably weaker than the order judgment, and no character identities or keywords were recovered.
corpus: python
order: as is
probabilities: python=0.56 romaji=0.13 english=0.31; as is=0.96 shuffled=0.04