OUTPUT of run 2 of at most 100: ``` LAYER 0 QK cosine matrix [[ 1. 0.012 0.008 0.137 0.013 0.044] [ 0.012 1. 0.691 -0.026 0.57 0.026] [ 0.008 0.691 1. -0.063 0.493 -0.075] [ 0.137 -0.026 -0.063 1. -0.044 0.01 ] [ 0.013 0.57 0.493 -0.044 1. 0.033] [ 0.044 0.026 -0.075 0.01 0.033 1. ]] OV cosine matrix [[ 1. 0.018 0.017 -0.036 0.026 0.017] [ 0.018 1. 0.925 0.006 0.869 0.644] [ 0.017 0.925 1. 0.009 0.811 0.692] [-0.036 0.006 0.009 1. 0.011 -0.001] [ 0.026 0.869 0.811 0.011 1. 0.592] [ 0.017 0.644 0.692 -0.001 0.592 1. ]] key-bias cosine matrix [[ 1. -0.054 -0.104 -0.041 0.29 -0.095] [-0.054 1. 0.347 -0.093 0.41 -0.226] [-0.104 0.347 1. -0.057 0.567 0.524] [-0.041 -0.093 -0.057 1. 0.022 0.074] [ 0.29 0.41 0.567 0.022 1. 0.373] [-0.095 -0.226 0.524 0.074 0.373 1. ]] LAYER 1 QK cosine matrix [[1. 0.357 0.372 0.345 0.334 0.376] [0.357 1. 0.371 0.361 0.302 0.398] [0.372 0.371 1. 0.368 0.375 0.351] [0.345 0.361 0.368 1. 0.32 0.394] [0.334 0.302 0.375 0.32 1. 0.288] [0.376 0.398 0.351 0.394 0.288 1. ]] OV cosine matrix [[ 1. -0.02 0.03 -0.012 0.001 0.029] [-0.02 1. -0.009 -0.028 -0.018 0.016] [ 0.03 -0.009 1. 0.001 0.014 -0.017] [-0.012 -0.028 0.001 1. -0.001 -0.015] [ 0.001 -0.018 0.014 -0.001 1. -0.014] [ 0.029 0.016 -0.017 -0.015 -0.014 1. ]] key-bias cosine matrix [[1. 0.829 0.853 0.823 0.825 0.86 ] [0.829 1. 0.851 0.864 0.821 0.861] [0.853 0.851 1. 0.83 0.834 0.833] [0.823 0.864 0.83 1. 0.834 0.873] [0.825 0.821 0.834 0.834 1. 0.828] [0.86 0.861 0.833 0.873 0.828 1. ]] LAYER 2 QK cosine matrix [[1. 0.128 0.161 0.117 0.179 0.134] [0.128 1. 0.116 0.086 0.121 0.116] [0.161 0.116 1. 0.101 0.144 0.123] [0.117 0.086 0.101 1. 0.092 0.093] [0.179 0.121 0.144 0.092 1. 0.134] [0.134 0.116 0.123 0.093 0.134 1. ]] OV cosine matrix [[ 1. -0.023 -0.039 -0.022 -0.005 0.001] [-0.023 1. -0.001 0.002 -0.012 -0.008] [-0.039 -0.001 1. 0.013 0.004 0.022] [-0.022 0.002 0.013 1. 0.001 0. ] [-0.005 -0.012 0.004 0.001 1. 0.004] [ 0.001 -0.008 0.022 0. 0.004 1. ]] key-bias cosine matrix [[1. 0.807 0.766 0.703 0.82 0.835] [0.807 1. 0.811 0.719 0.824 0.837] [0.766 0.811 1. 0.699 0.832 0.827] [0.703 0.719 0.699 1. 0.715 0.73 ] [0.82 0.824 0.832 0.715 1. 0.845] [0.835 0.837 0.827 0.73 0.845 1. ]] LAYER 3 QK cosine matrix [[1. 0.058 0.082 0.06 0.007 0.108] [0.058 1. 0.069 0.034 0.041 0.075] [0.082 0.069 1. 0.031 0.036 0.094] [0.06 0.034 0.031 1. 0.036 0.058] [0.007 0.041 0.036 0.036 1. 0.027] [0.108 0.075 0.094 0.058 0.027 1. ]] OV cosine matrix [[ 1. -0.012 0.024 0.011 -0.002 0.034] [-0.012 1. -0.021 0.024 0.059 0.001] [ 0.024 -0.021 1. 0.003 0.078 0.026] [ 0.011 0.024 0.003 1. 0.042 -0.001] [-0.002 0.059 0.078 0.042 1. 0.057] [ 0.034 0.001 0.026 -0.001 0.057 1. ]] key-bias cosine matrix [[ 1. 0.6 0.747 0.013 0.159 0.776] [ 0.6 1. 0.553 0.015 0.289 0.669] [ 0.747 0.553 1. 0.011 0.208 0.736] [ 0.013 0.015 0.011 1. 0.026 -0.046] [ 0.159 0.289 0.208 0.026 1. 0.254] [ 0.776 0.669 0.736 -0.046 0.254 1. ]] SPECTRA: first 12 singular values L0H0 QK [0.248 0.102 0.047 0.043 0.037 0.036 0.034 0.031 0.026 0.023 0.022 0.022] OV [0.957 0.898 0.818 0.641 0.554 0.474 0.439 0.406 0.372 0.348 0.329 0.308] L0H1 QK [0.15 0.042 0.021 0.012 0.01 0.007 0.007 0.004 0.004 0.004 0.003 0.003] OV [3.38 2.023 0.344 0.179 0.147 0.132 0.123 0.118 0.108 0.098 0.09 0.087] L0H2 QK [0.102 0.018 0.016 0.012 0.008 0.007 0.005 0.004 0.004 0.004 0.003 0.003] OV [2.643 1.363 0.399 0.175 0.159 0.142 0.126 0.105 0.096 0.094 0.085 0.085] L0H3 QK [0.244 0.222 0.201 0.182 0.179 0.157 0.155 0.149 0.14 0.136 0.131 0.128] OV [1.126 0.955 0.95 0.889 0.863 0.854 0.842 0.78 0.759 0.723 0.708 0.673] L0H4 QK [0.08 0.022 0.019 0.013 0.011 0.008 0.007 0.006 0.005 0.005 0.004 0.004] OV [1.454 0.723 0.213 0.141 0.138 0.115 0.109 0.097 0.094 0.084 0.079 0.075] L0H5 QK [0.032 0.004 0.003 0.003 0.002 0.002 0.002 0.002 0.002 0.002 0.002 0.002] OV [16.278 8.778 2.833 1.117 1.039 0.631 0.533 0.448 0.413 0.326 0.266 0.221] L3H0 QK [0.435 0.414 0.39 0.38 0.373 0.36 0.357 0.347 0.337 0.333 0.331 0.328] OV [5.784 5.655 5.063 4.642 4.213 3.824 3.807 3.649 3.604 3.567 3.389 3.298] L3H1 QK [0.345 0.308 0.285 0.253 0.244 0.227 0.211 0.207 0.2 0.193 0.19 0.186] OV [8.23 7.821 6.561 6.475 6.066 5.764 5.569 5.125 5.038 4.788 4.661 4.484] L3H2 QK [0.383 0.304 0.294 0.277 0.266 0.258 0.254 0.243 0.239 0.231 0.229 0.223] OV [7.46 6.108 5.569 5.335 5.047 4.689 4.586 4.379 4.357 4.143 3.993 3.795] L3H3 QK [0.322 0.311 0.307 0.304 0.297 0.294 0.288 0.286 0.284 0.282 0.277 0.273] OV [10.809 9.984 9.82 9.426 9.144 9.057 8.451 7.949 7.672 7.388 7.361 6.959] L3H4 QK [0.236 0.202 0.17 0.149 0.132 0.124 0.111 0.105 0.102 0.096 0.092 0.09 ] OV [21.266 17.675 16.977 15.046 14.398 13.303 12.126 11.541 8.688 8.574 7.486 6.519] L3H5 QK [0.454 0.321 0.315 0.299 0.289 0.288 0.284 0.28 0.276 0.271 0.265 0.261] OV [6.302 5.976 5.474 5.086 4.976 4.706 4.351 4.061 3.987 3.899 3.807 3.631] CROSS-HEAD QK similarities: strongest other-layer matches L0H0 [(np.float64(0.00979153483382163), 'L1H5'), (np.float64(0.008427259356622762), 'L2H0'), (np.float64(0.008223779217835316), 'L1H3')] L0H1 [(np.float64(0.0010880394971872875), 'L2H0'), (np.float64(0.0008593703704631152), 'L2H2'), (np.float64(-0.0001032708915409376), 'L2H1')] L0H2 [(np.float64(0.0050234992549716305), 'L2H0'), (np.float64(0.004594332635559284), 'L1H2'), (np.float64(0.004580946146990521), 'L1H4')] L0H3 [(np.float64(0.016643867610997714), 'L1H5'), (np.float64(0.016533999939375973), 'L3H3'), (np.float64(0.016121030549748686), 'L1H0')] L0H4 [(np.float64(0.005772833757418792), 'L2H2'), (np.float64(0.0036898938096945545), 'L1H4'), (np.float64(0.0027571174250421446), 'L1H2')] L0H5 [(np.float64(0.010936402216938758), 'L3H4'), (np.float64(-0.001411510777306644), 'L3H3'), (np.float64(-0.002164029793589887), 'L2H0')] L1H0 [(np.float64(0.18506694011229038), 'L2H0'), (np.float64(0.1488677247000508), 'L2H4'), (np.float64(0.12564662600924476), 'L2H2')] L1H1 [(np.float64(0.17022256856726697), 'L2H0'), (np.float64(0.1573422419093759), 'L2H4'), (np.float64(0.13271517165724014), 'L2H1')] L1H2 [(np.float64(0.18100756505856744), 'L2H0'), (np.float64(0.1668710304689896), 'L2H4'), (np.float64(0.12374133848220824), 'L2H5')] L1H3 [(np.float64(0.19214667678145736), 'L2H0'), (np.float64(0.16266425883219318), 'L2H4'), (np.float64(0.13267644666785067), 'L2H5')] L1H4 [(np.float64(0.1772663620980154), 'L2H0'), (np.float64(0.12116380081634327), 'L2H4'), (np.float64(0.09611140079226985), 'L2H2')] L1H5 [(np.float64(0.17310967356438595), 'L2H0'), (np.float64(0.14688132938479792), 'L2H4'), (np.float64(0.12588727047420217), 'L2H5')] L2H0 [(np.float64(0.19214667678145736), 'L1H3'), (np.float64(0.18506694011229038), 'L1H0'), (np.float64(0.18100756505856744), 'L1H2')] L2H1 [(np.float64(0.13271517165724014), 'L1H1'), (np.float64(0.12621364897594486), 'L1H3'), (np.float64(0.12278561406882613), 'L1H2')] L2H2 [(np.float64(0.12742023396713717), 'L1H3'), (np.float64(0.12564662600924476), 'L1H0'), (np.float64(0.12162773073563991), 'L1H1')] L2H3 [(np.float64(0.09685903841896927), 'L1H3'), (np.float64(0.0910249506652174), 'L1H1'), (np.float64(0.08406845805609671), 'L1H5')] L2H4 [(np.float64(0.1668710304689896), 'L1H2'), (np.float64(0.16266425883219318), 'L1H3'), (np.float64(0.1573422419093759), 'L1H1')] L2H5 [(np.float64(0.13267644666785067), 'L1H3'), (np.float64(0.12880880741147002), 'L1H1'), (np.float64(0.12588727047420217), 'L1H5')] L3H0 [(np.float64(0.06303436921510018), 'L2H4'), (np.float64(0.061698054643142985), 'L2H1'), (np.float64(0.0586590497588326), 'L2H5')] L3H1 [(np.float64(0.0507185844159886), 'L2H4'), (np.float64(0.050181191045734784), 'L2H0'), (np.float64(0.047482565641790125), 'L1H3')] L3H2 [(np.float64(0.06161449958325272), 'L2H2'), (np.float64(0.061561032715421016), 'L2H0'), (np.float64(0.059263577539727344), 'L1H3')] L3H3 [(np.float64(0.021274040539388073), 'L2H5'), (np.float64(0.016533999939375973), 'L0H3'), (np.float64(0.01648908421968714), 'L2H1')] L3H4 [(np.float64(0.017223015933478858), 'L1H3'), (np.float64(0.01672416710494667), 'L2H0'), (np.float64(0.015104739991435589), 'L2H3')] L3H5 [(np.float64(0.05636998232246223), 'L1H3'), (np.float64(0.05594018357599625), 'L2H1'), (np.float64(0.053872260032412106), 'L2H0')] MLP LINEAR-ALGEBRA STATISTICS (no nonlinearities applied) L 0 FC spectrum [17.58 13.47 11.76 10.19 9.61 9.54 9.34 9.14 8.75 8.68 8.58 8.39 8. 7.93 7.84 7.73 7.52 7.39 7.16 7.08] PROJ spectrum [10.9 7.7 7.17 6.64 6.54 6.29 6.2 6.13 5.94 5.92 5.76 5.64 5.55 5.48 5.39 5.36 5.26 5.21 5.17 5.15] PROJ@FC spectrum [125.96 60.95 37.56 32.55 31. 29.68 28.22 27.4 26.51 25.84 25.26 24.58 23.93 23.43 22.22 21.62 21.43 21.02 20.57 20.24] trace/norm 0.068 neuron input/output cos quantiles [-0.619 -0.118 -0.022 0.087 0.599] L 1 FC spectrum [22.95 13.34 13.21 13.05 12.74 12.71 12.3 12.24 11.92 11.81 11.75 11.47 11.43 11.31 11.18 11.05 10.96 10.58 10.53 10.28] PROJ spectrum [9.04 8.46 8.1 7.33 7.32 7.09 6.77 6.67 6.57 6.43 6.37 6.28 6.25 6.23 6.13 6.05 6.01 5.96 5.91 5.9 ] PROJ@FC spectrum [74.44 32.73 28.44 27.61 26.93 26.2 25.44 25.24 24.83 24.1 23.77 23.48 22.89 22.71 22.09 21.9 21.63 21.44 21.03 20.84] trace/norm -2.058 neuron input/output cos quantiles [-0.535 -0.143 -0.06 0.028 0.446] L 2 FC spectrum [20.71 12.17 11.48 11.41 11.12 10.28 10.09 9.9 9.84 9.75 9.66 9.6 9.56 9.51 9.34 9.25 9.15 9.07 9.01 8.96] PROJ spectrum [10.2 9.03 8.88 8.41 8.22 8.09 7.99 7.83 7.68 7.49 7.37 7.26 7.17 7.07 7.03 6.8 6.74 6.66 6.46 6.43] PROJ@FC spectrum [78.87 33.38 30.76 30.37 29.11 28.15 28.07 26.66 25.88 25.73 25.46 25.05 24.7 24.16 23.66 23.24 23.04 22.55 22.08 21.91] trace/norm -3.224 neuron input/output cos quantiles [-0.439 -0.175 -0.093 0. 0.445] L 3 FC spectrum [19.3 11. 10.8 10.27 9.8 9.56 9.5 9.38 9.02 8.85 8.78 8.63 8.41 8.36 8.22 8.06 8.03 7.96 7.85 7.79] PROJ spectrum [28.46 22.21 21.34 20.71 19.94 19.66 18.99 18.74 18.29 17.46 17.13 16.56 15.33 14.87 14.33 13.44 12.67 12.45 12.32 11.53] PROJ@FC spectrum [187.97 87.29 73.96 51.12 47.59 45.79 43.3 42.18 40.74 39.28 38.62 35.97 33.01 31.93 31.52 30.51 28.8 28.34 27.74 26.58] trace/norm -2.422 neuron input/output cos quantiles [-0.389 -0.209 -0.096 0.04 0.262] Most anisotropic coordinates L 0 largest ln1 biases [(312, -0.095, 0.214), (146, 0.091, 0.18), (12, 0.086, 0.12), (104, -0.084, 0.211), (289, 0.083, 0.376), (156, -0.078, 0.095), (303, -0.078, 0.404), (328, 0.076, 0.173)] L 1 largest ln1 biases [(39, -1.568, 0.299), (58, 1.365, 0.375), (99, -1.336, 0.716), (184, -0.562, 0.249), (383, -0.462, 0.338), (21, -0.43, 1.085), (153, 0.297, 0.42), (266, 0.277, 0.47)] L 2 largest ln1 biases [(99, -1.659, 0.943), (58, 1.332, 0.477), (39, -1.199, 0.422), (383, -0.507, 0.317), (184, -0.505, 0.337), (21, -0.453, 1.093), (63, -0.325, 0.481), (266, 0.299, 0.491)] L 3 largest ln1 biases [(58, 0.824, 0.466), (99, -0.764, 0.751), (5, 0.436, 0.541), (383, -0.397, 0.408), (266, 0.343, 0.524), (21, -0.315, 0.802), (184, -0.289, 0.562), (45, 0.289, 0.511)] Largest L0 ln1 scales [(180, 0.978), (184, 0.974), (162, 0.97), (99, 0.917), (211, 0.896), (256, 0.892), (300, 0.804), (317, 0.804), (223, 0.802), (39, 0.799), (173, 0.793), (4, 0.786), (228, 0.786), (69, 0.772), (89, 0.767), (100, 0.766), (56, 0.753), (65, 0.749), (354, 0.746), (246, 0.726)] ```