OUTPUT of run 2 of at most 100:
```

LAYER 0 QK cosine matrix
[[ 1.     0.012  0.008  0.137  0.013  0.044]
 [ 0.012  1.     0.691 -0.026  0.57   0.026]
 [ 0.008  0.691  1.    -0.063  0.493 -0.075]
 [ 0.137 -0.026 -0.063  1.    -0.044  0.01 ]
 [ 0.013  0.57   0.493 -0.044  1.     0.033]
 [ 0.044  0.026 -0.075  0.01   0.033  1.   ]]
OV cosine matrix
[[ 1.     0.018  0.017 -0.036  0.026  0.017]
 [ 0.018  1.     0.925  0.006  0.869  0.644]
 [ 0.017  0.925  1.     0.009  0.811  0.692]
 [-0.036  0.006  0.009  1.     0.011 -0.001]
 [ 0.026  0.869  0.811  0.011  1.     0.592]
 [ 0.017  0.644  0.692 -0.001  0.592  1.   ]]
key-bias cosine matrix
[[ 1.    -0.054 -0.104 -0.041  0.29  -0.095]
 [-0.054  1.     0.347 -0.093  0.41  -0.226]
 [-0.104  0.347  1.    -0.057  0.567  0.524]
 [-0.041 -0.093 -0.057  1.     0.022  0.074]
 [ 0.29   0.41   0.567  0.022  1.     0.373]
 [-0.095 -0.226  0.524  0.074  0.373  1.   ]]

LAYER 1 QK cosine matrix
[[1.    0.357 0.372 0.345 0.334 0.376]
 [0.357 1.    0.371 0.361 0.302 0.398]
 [0.372 0.371 1.    0.368 0.375 0.351]
 [0.345 0.361 0.368 1.    0.32  0.394]
 [0.334 0.302 0.375 0.32  1.    0.288]
 [0.376 0.398 0.351 0.394 0.288 1.   ]]
OV cosine matrix
[[ 1.    -0.02   0.03  -0.012  0.001  0.029]
 [-0.02   1.    -0.009 -0.028 -0.018  0.016]
 [ 0.03  -0.009  1.     0.001  0.014 -0.017]
 [-0.012 -0.028  0.001  1.    -0.001 -0.015]
 [ 0.001 -0.018  0.014 -0.001  1.    -0.014]
 [ 0.029  0.016 -0.017 -0.015 -0.014  1.   ]]
key-bias cosine matrix
[[1.    0.829 0.853 0.823 0.825 0.86 ]
 [0.829 1.    0.851 0.864 0.821 0.861]
 [0.853 0.851 1.    0.83  0.834 0.833]
 [0.823 0.864 0.83  1.    0.834 0.873]
 [0.825 0.821 0.834 0.834 1.    0.828]
 [0.86  0.861 0.833 0.873 0.828 1.   ]]

LAYER 2 QK cosine matrix
[[1.    0.128 0.161 0.117 0.179 0.134]
 [0.128 1.    0.116 0.086 0.121 0.116]
 [0.161 0.116 1.    0.101 0.144 0.123]
 [0.117 0.086 0.101 1.    0.092 0.093]
 [0.179 0.121 0.144 0.092 1.    0.134]
 [0.134 0.116 0.123 0.093 0.134 1.   ]]
OV cosine matrix
[[ 1.    -0.023 -0.039 -0.022 -0.005  0.001]
 [-0.023  1.    -0.001  0.002 -0.012 -0.008]
 [-0.039 -0.001  1.     0.013  0.004  0.022]
 [-0.022  0.002  0.013  1.     0.001  0.   ]
 [-0.005 -0.012  0.004  0.001  1.     0.004]
 [ 0.001 -0.008  0.022  0.     0.004  1.   ]]
key-bias cosine matrix
[[1.    0.807 0.766 0.703 0.82  0.835]
 [0.807 1.    0.811 0.719 0.824 0.837]
 [0.766 0.811 1.    0.699 0.832 0.827]
 [0.703 0.719 0.699 1.    0.715 0.73 ]
 [0.82  0.824 0.832 0.715 1.    0.845]
 [0.835 0.837 0.827 0.73  0.845 1.   ]]

LAYER 3 QK cosine matrix
[[1.    0.058 0.082 0.06  0.007 0.108]
 [0.058 1.    0.069 0.034 0.041 0.075]
 [0.082 0.069 1.    0.031 0.036 0.094]
 [0.06  0.034 0.031 1.    0.036 0.058]
 [0.007 0.041 0.036 0.036 1.    0.027]
 [0.108 0.075 0.094 0.058 0.027 1.   ]]
OV cosine matrix
[[ 1.    -0.012  0.024  0.011 -0.002  0.034]
 [-0.012  1.    -0.021  0.024  0.059  0.001]
 [ 0.024 -0.021  1.     0.003  0.078  0.026]
 [ 0.011  0.024  0.003  1.     0.042 -0.001]
 [-0.002  0.059  0.078  0.042  1.     0.057]
 [ 0.034  0.001  0.026 -0.001  0.057  1.   ]]
key-bias cosine matrix
[[ 1.     0.6    0.747  0.013  0.159  0.776]
 [ 0.6    1.     0.553  0.015  0.289  0.669]
 [ 0.747  0.553  1.     0.011  0.208  0.736]
 [ 0.013  0.015  0.011  1.     0.026 -0.046]
 [ 0.159  0.289  0.208  0.026  1.     0.254]
 [ 0.776  0.669  0.736 -0.046  0.254  1.   ]]

SPECTRA: first 12 singular values
L0H0 QK [0.248 0.102 0.047 0.043 0.037 0.036 0.034 0.031 0.026 0.023 0.022 0.022] OV [0.957 0.898 0.818 0.641 0.554 0.474 0.439 0.406 0.372 0.348 0.329 0.308]
L0H1 QK [0.15  0.042 0.021 0.012 0.01  0.007 0.007 0.004 0.004 0.004 0.003 0.003] OV [3.38  2.023 0.344 0.179 0.147 0.132 0.123 0.118 0.108 0.098 0.09  0.087]
L0H2 QK [0.102 0.018 0.016 0.012 0.008 0.007 0.005 0.004 0.004 0.004 0.003 0.003] OV [2.643 1.363 0.399 0.175 0.159 0.142 0.126 0.105 0.096 0.094 0.085 0.085]
L0H3 QK [0.244 0.222 0.201 0.182 0.179 0.157 0.155 0.149 0.14  0.136 0.131 0.128] OV [1.126 0.955 0.95  0.889 0.863 0.854 0.842 0.78  0.759 0.723 0.708 0.673]
L0H4 QK [0.08  0.022 0.019 0.013 0.011 0.008 0.007 0.006 0.005 0.005 0.004 0.004] OV [1.454 0.723 0.213 0.141 0.138 0.115 0.109 0.097 0.094 0.084 0.079 0.075]
L0H5 QK [0.032 0.004 0.003 0.003 0.002 0.002 0.002 0.002 0.002 0.002 0.002 0.002] OV [16.278  8.778  2.833  1.117  1.039  0.631  0.533  0.448  0.413  0.326
  0.266  0.221]
L3H0 QK [0.435 0.414 0.39  0.38  0.373 0.36  0.357 0.347 0.337 0.333 0.331 0.328] OV [5.784 5.655 5.063 4.642 4.213 3.824 3.807 3.649 3.604 3.567 3.389 3.298]
L3H1 QK [0.345 0.308 0.285 0.253 0.244 0.227 0.211 0.207 0.2   0.193 0.19  0.186] OV [8.23  7.821 6.561 6.475 6.066 5.764 5.569 5.125 5.038 4.788 4.661 4.484]
L3H2 QK [0.383 0.304 0.294 0.277 0.266 0.258 0.254 0.243 0.239 0.231 0.229 0.223] OV [7.46  6.108 5.569 5.335 5.047 4.689 4.586 4.379 4.357 4.143 3.993 3.795]
L3H3 QK [0.322 0.311 0.307 0.304 0.297 0.294 0.288 0.286 0.284 0.282 0.277 0.273] OV [10.809  9.984  9.82   9.426  9.144  9.057  8.451  7.949  7.672  7.388
  7.361  6.959]
L3H4 QK [0.236 0.202 0.17  0.149 0.132 0.124 0.111 0.105 0.102 0.096 0.092 0.09 ] OV [21.266 17.675 16.977 15.046 14.398 13.303 12.126 11.541  8.688  8.574
  7.486  6.519]
L3H5 QK [0.454 0.321 0.315 0.299 0.289 0.288 0.284 0.28  0.276 0.271 0.265 0.261] OV [6.302 5.976 5.474 5.086 4.976 4.706 4.351 4.061 3.987 3.899 3.807 3.631]

CROSS-HEAD QK similarities: strongest other-layer matches
L0H0 [(np.float64(0.00979153483382163), 'L1H5'), (np.float64(0.008427259356622762), 'L2H0'), (np.float64(0.008223779217835316), 'L1H3')]
L0H1 [(np.float64(0.0010880394971872875), 'L2H0'), (np.float64(0.0008593703704631152), 'L2H2'), (np.float64(-0.0001032708915409376), 'L2H1')]
L0H2 [(np.float64(0.0050234992549716305), 'L2H0'), (np.float64(0.004594332635559284), 'L1H2'), (np.float64(0.004580946146990521), 'L1H4')]
L0H3 [(np.float64(0.016643867610997714), 'L1H5'), (np.float64(0.016533999939375973), 'L3H3'), (np.float64(0.016121030549748686), 'L1H0')]
L0H4 [(np.float64(0.005772833757418792), 'L2H2'), (np.float64(0.0036898938096945545), 'L1H4'), (np.float64(0.0027571174250421446), 'L1H2')]
L0H5 [(np.float64(0.010936402216938758), 'L3H4'), (np.float64(-0.001411510777306644), 'L3H3'), (np.float64(-0.002164029793589887), 'L2H0')]
L1H0 [(np.float64(0.18506694011229038), 'L2H0'), (np.float64(0.1488677247000508), 'L2H4'), (np.float64(0.12564662600924476), 'L2H2')]
L1H1 [(np.float64(0.17022256856726697), 'L2H0'), (np.float64(0.1573422419093759), 'L2H4'), (np.float64(0.13271517165724014), 'L2H1')]
L1H2 [(np.float64(0.18100756505856744), 'L2H0'), (np.float64(0.1668710304689896), 'L2H4'), (np.float64(0.12374133848220824), 'L2H5')]
L1H3 [(np.float64(0.19214667678145736), 'L2H0'), (np.float64(0.16266425883219318), 'L2H4'), (np.float64(0.13267644666785067), 'L2H5')]
L1H4 [(np.float64(0.1772663620980154), 'L2H0'), (np.float64(0.12116380081634327), 'L2H4'), (np.float64(0.09611140079226985), 'L2H2')]
L1H5 [(np.float64(0.17310967356438595), 'L2H0'), (np.float64(0.14688132938479792), 'L2H4'), (np.float64(0.12588727047420217), 'L2H5')]
L2H0 [(np.float64(0.19214667678145736), 'L1H3'), (np.float64(0.18506694011229038), 'L1H0'), (np.float64(0.18100756505856744), 'L1H2')]
L2H1 [(np.float64(0.13271517165724014), 'L1H1'), (np.float64(0.12621364897594486), 'L1H3'), (np.float64(0.12278561406882613), 'L1H2')]
L2H2 [(np.float64(0.12742023396713717), 'L1H3'), (np.float64(0.12564662600924476), 'L1H0'), (np.float64(0.12162773073563991), 'L1H1')]
L2H3 [(np.float64(0.09685903841896927), 'L1H3'), (np.float64(0.0910249506652174), 'L1H1'), (np.float64(0.08406845805609671), 'L1H5')]
L2H4 [(np.float64(0.1668710304689896), 'L1H2'), (np.float64(0.16266425883219318), 'L1H3'), (np.float64(0.1573422419093759), 'L1H1')]
L2H5 [(np.float64(0.13267644666785067), 'L1H3'), (np.float64(0.12880880741147002), 'L1H1'), (np.float64(0.12588727047420217), 'L1H5')]
L3H0 [(np.float64(0.06303436921510018), 'L2H4'), (np.float64(0.061698054643142985), 'L2H1'), (np.float64(0.0586590497588326), 'L2H5')]
L3H1 [(np.float64(0.0507185844159886), 'L2H4'), (np.float64(0.050181191045734784), 'L2H0'), (np.float64(0.047482565641790125), 'L1H3')]
L3H2 [(np.float64(0.06161449958325272), 'L2H2'), (np.float64(0.061561032715421016), 'L2H0'), (np.float64(0.059263577539727344), 'L1H3')]
L3H3 [(np.float64(0.021274040539388073), 'L2H5'), (np.float64(0.016533999939375973), 'L0H3'), (np.float64(0.01648908421968714), 'L2H1')]
L3H4 [(np.float64(0.017223015933478858), 'L1H3'), (np.float64(0.01672416710494667), 'L2H0'), (np.float64(0.015104739991435589), 'L2H3')]
L3H5 [(np.float64(0.05636998232246223), 'L1H3'), (np.float64(0.05594018357599625), 'L2H1'), (np.float64(0.053872260032412106), 'L2H0')]

MLP LINEAR-ALGEBRA STATISTICS (no nonlinearities applied)
L 0 FC spectrum [17.58 13.47 11.76 10.19  9.61  9.54  9.34  9.14  8.75  8.68  8.58  8.39
  8.    7.93  7.84  7.73  7.52  7.39  7.16  7.08]
PROJ spectrum [10.9   7.7   7.17  6.64  6.54  6.29  6.2   6.13  5.94  5.92  5.76  5.64
  5.55  5.48  5.39  5.36  5.26  5.21  5.17  5.15]
PROJ@FC spectrum [125.96  60.95  37.56  32.55  31.    29.68  28.22  27.4   26.51  25.84
  25.26  24.58  23.93  23.43  22.22  21.62  21.43  21.02  20.57  20.24] trace/norm 0.068 neuron input/output cos quantiles [-0.619 -0.118 -0.022  0.087  0.599]
L 1 FC spectrum [22.95 13.34 13.21 13.05 12.74 12.71 12.3  12.24 11.92 11.81 11.75 11.47
 11.43 11.31 11.18 11.05 10.96 10.58 10.53 10.28]
PROJ spectrum [9.04 8.46 8.1  7.33 7.32 7.09 6.77 6.67 6.57 6.43 6.37 6.28 6.25 6.23
 6.13 6.05 6.01 5.96 5.91 5.9 ]
PROJ@FC spectrum [74.44 32.73 28.44 27.61 26.93 26.2  25.44 25.24 24.83 24.1  23.77 23.48
 22.89 22.71 22.09 21.9  21.63 21.44 21.03 20.84] trace/norm -2.058 neuron input/output cos quantiles [-0.535 -0.143 -0.06   0.028  0.446]
L 2 FC spectrum [20.71 12.17 11.48 11.41 11.12 10.28 10.09  9.9   9.84  9.75  9.66  9.6
  9.56  9.51  9.34  9.25  9.15  9.07  9.01  8.96]
PROJ spectrum [10.2   9.03  8.88  8.41  8.22  8.09  7.99  7.83  7.68  7.49  7.37  7.26
  7.17  7.07  7.03  6.8   6.74  6.66  6.46  6.43]
PROJ@FC spectrum [78.87 33.38 30.76 30.37 29.11 28.15 28.07 26.66 25.88 25.73 25.46 25.05
 24.7  24.16 23.66 23.24 23.04 22.55 22.08 21.91] trace/norm -3.224 neuron input/output cos quantiles [-0.439 -0.175 -0.093  0.     0.445]
L 3 FC spectrum [19.3  11.   10.8  10.27  9.8   9.56  9.5   9.38  9.02  8.85  8.78  8.63
  8.41  8.36  8.22  8.06  8.03  7.96  7.85  7.79]
PROJ spectrum [28.46 22.21 21.34 20.71 19.94 19.66 18.99 18.74 18.29 17.46 17.13 16.56
 15.33 14.87 14.33 13.44 12.67 12.45 12.32 11.53]
PROJ@FC spectrum [187.97  87.29  73.96  51.12  47.59  45.79  43.3   42.18  40.74  39.28
  38.62  35.97  33.01  31.93  31.52  30.51  28.8   28.34  27.74  26.58] trace/norm -2.422 neuron input/output cos quantiles [-0.389 -0.209 -0.096  0.04   0.262]

Most anisotropic coordinates
L 0 largest ln1 biases [(312, -0.095, 0.214), (146, 0.091, 0.18), (12, 0.086, 0.12), (104, -0.084, 0.211), (289, 0.083, 0.376), (156, -0.078, 0.095), (303, -0.078, 0.404), (328, 0.076, 0.173)]
L 1 largest ln1 biases [(39, -1.568, 0.299), (58, 1.365, 0.375), (99, -1.336, 0.716), (184, -0.562, 0.249), (383, -0.462, 0.338), (21, -0.43, 1.085), (153, 0.297, 0.42), (266, 0.277, 0.47)]
L 2 largest ln1 biases [(99, -1.659, 0.943), (58, 1.332, 0.477), (39, -1.199, 0.422), (383, -0.507, 0.317), (184, -0.505, 0.337), (21, -0.453, 1.093), (63, -0.325, 0.481), (266, 0.299, 0.491)]
L 3 largest ln1 biases [(58, 0.824, 0.466), (99, -0.764, 0.751), (5, 0.436, 0.541), (383, -0.397, 0.408), (266, 0.343, 0.524), (21, -0.315, 0.802), (184, -0.289, 0.562), (45, 0.289, 0.511)]
Largest L0 ln1 scales [(180, 0.978), (184, 0.974), (162, 0.97), (99, 0.917), (211, 0.896), (256, 0.892), (300, 0.804), (317, 0.804), (223, 0.802), (39, 0.799), (173, 0.793), (4, 0.786), (228, 0.786), (69, 0.772), (89, 0.767), (100, 0.766), (56, 0.753), (65, 0.749), (354, 0.746), (246, 0.726)]

```