OUTPUT of run 1 of at most 100:
```
LayerNorm parameters: mean, std, min, max, norm
ln1_w.0 [0.2965 0.1884 0.0352 0.9785 6.8836]
ln1_b.0 [ 0.0004  0.0311 -0.0954  0.0906  0.6101]
ln2_w.0 [ 0.8718  0.3382  0.042   1.775  18.3232]
ln2_b.0 [ 0.0134  0.0691 -0.6236  0.4636  1.3799]
ln1_w.1 [ 0.5058  0.0892  0.197   1.0846 10.0651]
ln1_b.1 [ 0.011   0.1546 -1.5676  1.3651  3.0369]
ln2_w.1 [ 1.1315  0.1741  0.0019  1.5196 22.4333]
ln2_b.1 [ 0.0065  0.1435 -0.3892  0.417   2.8153]
ln1_w.2 [ 0.6809  0.0687  0.3175  1.0927 13.4102]
ln1_b.2 [ 0.0131  0.1565 -1.6589  1.3318  3.0765]
ln2_w.2 [ 1.3104  0.1148  0.012   1.5078 25.777 ]
ln2_b.2 [ 0.0123  0.1737 -0.517   0.4489  3.4127]
ln1_w.3 [ 0.7087  0.0494  0.4078  0.8155 13.9223]
ln1_b.3 [ 0.0296  0.0994 -0.7639  0.8241  2.0324]
ln2_w.3 [ 1.2668  0.0894  0.2328  1.4532 24.886 ]
ln2_b.3 [ 0.0128  0.1413 -0.5402  0.3932  2.7798]

Matrix summaries: Frobenius norm, top singular value, stable rank
W_q.0 [17.6693  4.3676 16.3664]
W_k.0 [18.5161  4.1635 19.7784]
W_v.0 [14.9367  5.2084  8.2242]
W_o.0 [16.5495  7.3608  5.0551]
W_fc.0 [55.3764 18.1182  9.3416]
W_proj.0 [50.007  10.899  21.0517]
W_q.1 [38.9293 10.4064 13.9942]
W_k.1 [34.8926 14.4767  5.8093]
W_v.1 [28.7445  4.7281 36.9599]
W_o.1 [28.479   4.0451 49.5664]
W_fc.1 [80.3794 19.8292 16.4317]
W_proj.1 [77.3844  9.0375 73.3173]
W_q.2 [38.9025  5.9663 42.5157]
W_k.2 [36.75    9.8633 13.8827]
W_v.2 [33.1825  3.9143 71.8631]
W_o.2 [32.4345  4.3982 54.3828]
W_fc.2 [79.8282 15.7338 25.7422]
W_proj.2 [78.0482 10.1974 58.5793]
W_q.3 [33.2388  4.0862 66.1679]
W_k.3 [32.4633  6.3618 26.039 ]
W_v.3 [39.4218  5.6601 48.5088]
W_o.3 [40.1331 10.4767 14.6743]
W_fc.3 [78.2595 15.1141 26.8106]
W_proj.3 [85.1194 28.4613  8.9443]

Head algebra (no activations):
head | QK Frobenius, top SV, stable rank, trace, symmetric eig min/max | OV Frobenius, top SV, stable rank, trace | ||Wq b||, ||Wk b||
L0H0 [ 1.382  0.853  2.623 -1.619 -0.425  0.501] | [ 7.879  3.611  4.76  -0.676] | [0.504 0.725]
L0H1 [ 0.95   0.838  1.284 -0.847 -0.599  0.243] | [6.871 5.721 1.443 0.527] | [0.228 0.261]
L0H2 [ 0.634  0.543  1.362 -0.598 -0.388  0.16 ] | [4.9   3.943 1.545 0.277] | [0.146 0.314]
L0H3 [ 4.305  1.114 14.933  7.641 -0.478  0.899] | [14.071  4.426 10.107 -7.452] | [0.467 0.543]
L0H4 [ 0.636  0.51   1.56  -0.545 -0.338  0.176] | [ 3.116  2.449  1.62  -0.41 ] | [0.27 0.55]
L0H5 [ 0.21   0.18   1.348 -0.051 -0.104  0.077] | [29.436 24.799  1.409  2.348] | [0.143 0.171]
L1H0 [ 6.605  3.084  4.587 -6.95  -1.73   1.612] | [ 22.128   6.118  13.083 -30.069] | [ 6.688 13.795]
L1H1 [ 6.033  3.082  3.832 -5.844 -1.703  1.507] | [ 21.591   6.051  12.731 -29.585] | [ 6.079 13.929]
L1H2 [ 5.712  2.848  4.023 -7.262 -1.622  1.395] | [ 22.746   6.72   11.455 -33.608] | [ 5.564 13.162]
L1H3 [ 5.738  3.106  3.412 -5.817 -1.714  1.533] | [ 23.885   6.544  13.32  -36.07 ] | [ 5.665 13.742]
L1H4 [ 5.304  2.62   4.099 -7.654 -1.345  1.364] | [ 25.845   7.908  10.681 -37.328] | [ 5.347 10.986]
L1H5 [ 6.123  3.024  4.1   -6.208 -1.85   1.487] | [ 20.966   5.849  12.848 -28.92 ] | [ 6.224 14.146]
L2H0 [ 3.354  1.229  7.451 -3.371 -0.57   0.681] | [ 23.754   7.576   9.831 -24.508] | [ 4.265 10.635]
L2H1 [ 4.549  1.189 14.639 -0.512 -0.629  0.665] | [ 25.597   5.978  18.336 -50.868] | [ 3.787 10.117]
L2H2 [ 3.612  0.963 14.072 -2.835 -0.528  0.522] | [ 27.059   7.257  13.902 -43.194] | [3.784 9.886]
L2H3 [ 3.414  0.896 14.503 -1.229 -0.453  0.509] | [ 27.609   7.041  15.375 -43.815] | [3.082 7.454]
L2H4 [ 4.594  1.537  8.928 -2.083 -0.752  0.842] | [ 21.682   5.217  17.274 -45.57 ] | [ 4.319 11.042]
L2H5 [ 4.752  1.229 14.942 -0.662 -0.671  0.711] | [ 24.543   5.432  20.412 -50.098] | [ 3.751 10.382]
L3H0 [ 4.017  0.839 22.904  4.356 -0.395  0.532] | [ 29.023   8.117  12.783 -64.616] | [1.761 3.665]
L3H1 [ 2.412  0.658 13.446  0.9   -0.333  0.379] | [ 37.053  11.496  10.389 -58.624] | [1.46  3.019]
L3H2 [ 2.848  0.734 15.049  0.674 -0.362  0.377] | [ 32.882  10.376  10.042 -52.39 ] | [1.94 3.36]
L3H3 [ 3.627  0.645 31.642 11.586 -0.379  0.511] | [ 52.469  15.146  12.001 -62.9  ] | [1.298 1.116]
L3H4 [ 1.251  0.466  7.203  0.015 -0.228  0.255] | [ 68.192  29.438   5.366 -54.127] | [1.23  1.267]
L3H5 [ 3.295  0.863 14.574  2.884 -0.407  0.466] | [ 31.503   8.749  12.965 -48.399] | [1.8   3.333]

```