OUTPUT of run 2 of at most 100: ``` Effective linear maps include LayerNorm gain and mean-centering only. No normalization of inputs, attention, or MLP evaluation is performed. L0H0 QK_F/top/r90/tr [ 0.1137 0.0887 2. -0.0508] QK_svs [0.089 0.068 0.013 0.008 0.006 0.005] OV_F/tr [4.321 1.666] key_bias_norm 0.0185 L0H1 QK_F/top/r90/tr [ 0.8775 0.6013 6. -0.4868] QK_svs [0.601 0.373 0.297 0.211 0.199 0.155] OV_F/tr [3.488 0.164] key_bias_norm 0.0736 L0H2 QK_F/top/r90/tr [0.1745 0.132 5. 0.0546] QK_svs [0.132 0.08 0.044 0.031 0.028 0.025] OV_F/tr [2.023 0.015] key_bias_norm 0.0234 L0H3 QK_F/top/r90/tr [ 0.2395 0.1645 3. -0.1871] QK_svs [0.165 0.14 0.081 0.031 0.029 0.021] OV_F/tr [1.621 0.242] key_bias_norm 0.057 L0H4 QK_F/top/r90/tr [ 0.0093 0.0045 28. -0.0037] QK_svs [0.004 0.003 0.002 0.002 0.002 0.002] OV_F/tr [17.158 6.049] key_bias_norm 0.0009 L0H5 QK_F/top/r90/tr [ 0.9704 0.4099 16. 1.2135] QK_svs [0.41 0.362 0.273 0.254 0.245 0.222] OV_F/tr [ 3.707 -2.463] key_bias_norm 0.0509 L1H0 QK_F/top/r90/tr [ 0.644 0.3538 13. -0.308 ] QK_svs [0.354 0.252 0.218 0.167 0.158 0.139] OV_F/tr [ 13.436 -16.299] key_bias_norm 0.8392 L1H1 QK_F/top/r90/tr [ 1.5843 1.1456 9. -1.2729] QK_svs [1.146 0.661 0.459 0.326 0.249 0.215] OV_F/tr [ 9.592 -11.205] key_bias_norm 3.1498 L1H2 QK_F/top/r90/tr [ 1.2358 0.8069 8. -0.7401] QK_svs [0.807 0.693 0.332 0.205 0.18 0.159] OV_F/tr [ 11.496 -14.892] key_bias_norm 2.1998 L1H3 QK_F/top/r90/tr [ 1.3094 0.9127 8. -0.6948] QK_svs [0.913 0.699 0.309 0.202 0.178 0.167] OV_F/tr [ 10.043 -10.571] key_bias_norm 2.7194 L1H4 QK_F/top/r90/tr [ 1.2492 0.848 8. -0.5846] QK_svs [0.848 0.654 0.34 0.213 0.191 0.162] OV_F/tr [ 9.557 -10.446] key_bias_norm 2.518 L1H5 QK_F/top/r90/tr [ 1.4614 1.0088 8. -1.2054] QK_svs [1.009 0.555 0.482 0.45 0.242 0.215] OV_F/tr [ 9.826 -13.045] key_bias_norm 2.4602 L2H0 QK_F/top/r90/tr [ 2.0456 0.5184 39. -0.5814] QK_svs [0.518 0.485 0.459 0.416 0.407 0.397] OV_F/tr [ 18.698 -35.97 ] key_bias_norm 0.8945 L2H1 QK_F/top/r90/tr [ 1.9454 0.4582 39. 2.5132] QK_svs [0.458 0.421 0.396 0.393 0.367 0.355] OV_F/tr [ 21.448 -40.225] key_bias_norm 0.6377 L2H2 QK_F/top/r90/tr [ 2.804 0.7089 31. 1.4941] QK_svs [0.709 0.693 0.676 0.634 0.611 0.598] OV_F/tr [ 16.782 -29.462] key_bias_norm 1.5836 L2H3 QK_F/top/r90/tr [ 1.9759 0.5708 39. -0.4059] QK_svs [0.571 0.423 0.412 0.392 0.388 0.37 ] OV_F/tr [ 20.617 -38.568] key_bias_norm 1.0368 L2H4 QK_F/top/r90/tr [ 2.2016 0.7827 37. 0.0551] QK_svs [0.783 0.587 0.484 0.429 0.417 0.401] OV_F/tr [ 18.415 -38.753] key_bias_norm 1.1994 L2H5 QK_F/top/r90/tr [ 1.6541 0.4194 42. -0.7004] QK_svs [0.419 0.414 0.359 0.32 0.315 0.296] OV_F/tr [ 23.766 -33.94 ] key_bias_norm 0.6361 L3H0 QK_F/top/r90/tr [ 2.0777 0.4054 48. 1.7527] QK_svs [0.405 0.399 0.385 0.377 0.365 0.359] OV_F/tr [ 24.502 -48.319] key_bias_norm 0.2987 L3H1 QK_F/top/r90/tr [ 1.6948 0.3223 48. 2.225 ] QK_svs [0.322 0.315 0.309 0.304 0.3 0.295] OV_F/tr [ 25.724 -49.613] key_bias_norm 0.2634 L3H2 QK_F/top/r90/tr [ 2.0608 0.4337 48. 0.7176] QK_svs [0.434 0.411 0.392 0.362 0.358 0.353] OV_F/tr [ 26.86 -50.565] key_bias_norm 0.3422 L3H3 QK_F/top/r90/tr [ 1.053 0.2624 47. 0.0028] QK_svs [0.262 0.237 0.208 0.202 0.193 0.191] OV_F/tr [ 37.938 -51.153] key_bias_norm 0.2583 L3H4 QK_F/top/r90/tr [ 1.7344 0.4007 47. 0.0047] QK_svs [0.401 0.369 0.357 0.323 0.313 0.306] OV_F/tr [ 28.44 -35.013] key_bias_norm 0.3067 L3H5 QK_F/top/r90/tr [ 2.1169 0.4041 48. 8.0447] QK_svs [0.404 0.39 0.384 0.37 0.365 0.362] OV_F/tr [ 23.687 -56.843] key_bias_norm 0.3101 Within-layer QK cosine matrices: L 0 [[ 1. 0.179 0.112 0.206 0.063 0.093] [ 0.179 1. 0.21 -0.15 0.005 0.414] [ 0.112 0.21 1. -0.247 -0.006 0.114] [ 0.206 -0.15 -0.247 1. -0.027 -0.098] [ 0.063 0.005 -0.006 -0.027 1. -0.004] [ 0.093 0.414 0.114 -0.098 -0.004 1. ]] L 1 [[1. 0.302 0.393 0.327 0.338 0.266] [0.302 1. 0.524 0.591 0.568 0.566] [0.393 0.524 1. 0.538 0.578 0.45 ] [0.327 0.591 0.538 1. 0.607 0.524] [0.338 0.568 0.578 0.607 1. 0.479] [0.266 0.566 0.45 0.524 0.479 1. ]] L 2 [[1. 0.034 0.061 0.061 0.074 0.046] [0.034 1. 0.049 0.03 0.035 0.031] [0.061 0.049 1. 0.095 0.1 0.03 ] [0.061 0.03 0.095 1. 0.076 0.045] [0.074 0.035 0.1 0.076 1. 0.046] [0.046 0.031 0.03 0.045 0.046 1. ]] L 3 [[1. 0.084 0.044 0.045 0.042 0.073] [0.084 1. 0.013 0.044 0.032 0.192] [0.044 0.013 1. 0.038 0.053 0.021] [0.045 0.044 0.038 1. 0.054 0.035] [0.042 0.032 0.053 0.054 1. 0.037] [0.073 0.192 0.021 0.035 0.037 1. ]] Key-composition trace cosine: trace(QK_later @ OV_earlier)/(norms) L1H0 highest [('L0H4', 0.0207), ('L0H0', 0.0065), ('L0H3', 0.005), ('L0H1', -0.0001)] lowest [('L0H5', -0.0034), ('L0H2', -0.0079)] L1H1 highest [('L0H4', 0.0191), ('L0H5', 0.0154), ('L0H1', 0.007), ('L0H3', 0.0055)] lowest [('L0H0', -0.0046), ('L0H2', -0.0161)] L1H2 highest [('L0H4', 0.0326), ('L0H3', 0.0092), ('L0H0', 0.0039), ('L0H1', 0.0007)] lowest [('L0H5', -0.0027), ('L0H2', -0.0168)] L1H3 highest [('L0H4', 0.0421), ('L0H5', 0.0124), ('L0H0', 0.0077), ('L0H3', 0.0063)] lowest [('L0H1', -0.0026), ('L0H2', -0.0132)] L1H4 highest [('L0H4', 0.0439), ('L0H3', 0.0114), ('L0H5', 0.0094), ('L0H0', 0.0093)] lowest [('L0H1', 0.0022), ('L0H2', -0.0181)] L1H5 highest [('L0H5', 0.0128), ('L0H1', 0.0045), ('L0H3', -0.0004), ('L0H4', -0.0036)] lowest [('L0H2', -0.0134), ('L0H0', -0.0195)] L2H0 highest [('L0H4', 0.0099), ('L0H3', 0.0078), ('L1H5', 0.0057), ('L0H0', 0.0049)] lowest [('L0H5', -0.0046), ('L1H0', -0.0126)] L2H1 highest [('L0H5', 0.0776), ('L1H3', 0.0575), ('L1H2', 0.0482), ('L1H1', 0.0479)] lowest [('L0H3', 0.0025), ('L0H1', 0.0014)] L2H2 highest [('L0H4', 0.02), ('L0H0', 0.0114), ('L0H3', 0.0015), ('L0H2', -0.0054)] lowest [('L1H0', -0.0418), ('L1H2', -0.0434)] L2H3 highest [('L0H4', 0.0396), ('L0H0', 0.0177), ('L0H3', 0.0066), ('L1H1', 0.0049)] lowest [('L1H2', -0.0162), ('L1H0', -0.0197)] L2H4 highest [('L0H4', 0.0246), ('L0H3', 0.0101), ('L0H0', 0.0079), ('L0H5', 0.0018)] lowest [('L1H2', -0.0115), ('L1H0', -0.0139)] L2H5 highest [('L0H4', 0.0188), ('L1H2', 0.0095), ('L0H0', 0.009), ('L1H1', 0.0083)] lowest [('L1H0', 0.0001), ('L0H2', 0.0001)] L3H0 highest [('L2H2', 0.0238), ('L0H5', 0.019), ('L1H2', 0.0101), ('L0H0', 0.0098)] lowest [('L2H5', -0.0088), ('L2H1', -0.0135)] L3H1 highest [('L2H2', 0.1104), ('L0H5', 0.061), ('L1H4', 0.0456), ('L1H3', 0.0453)] lowest [('L2H5', -0.0002), ('L2H1', -0.0181)] L3H2 highest [('L0H4', 0.013), ('L0H0', 0.0062), ('L0H2', 0.0015), ('L1H1', 0.0013)] lowest [('L1H2', -0.0092), ('L1H0', -0.0116)] L3H3 highest [('L2H4', 0.0148), ('L2H3', 0.0103), ('L2H2', 0.009), ('L1H2', 0.0079)] lowest [('L1H3', -0.0004), ('L2H1', -0.0054)] L3H4 highest [('L0H4', 0.0119), ('L2H4', 0.0103), ('L0H5', 0.0101), ('L0H0', 0.0085)] lowest [('L0H1', -0.0063), ('L2H1', -0.0083)] L3H5 highest [('L2H2', 0.1836), ('L0H5', 0.0713), ('L1H3', 0.0683), ('L1H4', 0.0572)] lowest [('L2H5', -0.022), ('L2H1', -0.0599)] L0 gain extremes and corresponding linear-map column norms: 365 g1/g2 0.0165 1.1172 q/k/v/fc [1.049 1.054 0.975 3.402] 187 g1/g2 0.0226 0.7165 q/k/v/fc [1.132 0.967 0.916 2.7 ] 55 g1/g2 0.0253 0.9217 q/k/v/fc [0.809 0.841 1.062 2.515] 24 g1/g2 0.0305 0.589 q/k/v/fc [1.035 0.996 0.845 2.668] 357 g1/g2 0.0323 1.1837 q/k/v/fc [0.925 0.895 1.155 3.137] 347 g1/g2 0.0337 1.1467 q/k/v/fc [0.889 0.975 1.032 2.508] 3 g1/g2 0.0358 0.917 q/k/v/fc [1.091 0.915 1.034 2.987] 248 g1/g2 0.041 1.1464 q/k/v/fc [0.889 0.892 1.053 3.335] 255 g1/g2 0.0412 1.0276 q/k/v/fc [1.056 1.148 1.033 2.583] 2 g1/g2 0.0455 1.4738 q/k/v/fc [0.75 0.743 1.054 3.661] 181 g1/g2 0.0456 0.862 q/k/v/fc [1.048 0.988 1.087 2.82 ] 92 g1/g2 0.0462 0.7997 q/k/v/fc [1.095 0.954 0.978 2.566] 184 g1/g2 0.8285 0.3596 q/k/v/fc [1.128 1.39 0.773 2.415] 359 g1/g2 0.8331 0.5308 q/k/v/fc [1.102 1.905 1.086 2.658] 307 g1/g2 0.8441 0.6557 q/k/v/fc [1.286 1.432 0.785 2.01 ] 48 g1/g2 0.8725 0.3104 q/k/v/fc [1.35 2.011 0.849 2.286] 228 g1/g2 0.9005 0.642 q/k/v/fc [1.07 2.112 1.049 2.281] 60 g1/g2 0.9031 0.4939 q/k/v/fc [1.232 2.132 1.076 2.338] 162 g1/g2 0.9129 0.4061 q/k/v/fc [1.295 2.002 0.951 2.428] 50 g1/g2 0.9335 0.5591 q/k/v/fc [1.225 1.803 0.924 2.606] 180 g1/g2 0.9483 0.5096 q/k/v/fc [1.255 2.308 0.843 2.416] 223 g1/g2 0.9731 0.3425 q/k/v/fc [1.198 2.437 1.17 2.354] 273 g1/g2 0.9813 0.4232 q/k/v/fc [1.273 2.123 1.29 1.452] 211 g1/g2 1.0134 0.5536 q/k/v/fc [1.119 2.735 1.166 2.481] Linear MLP weight products (not MLP evaluations): 0 Frob/trace 203.535 89.182 eig real min/max -7.911 16.912 1 Frob/trace 173.124 -322.781 eig real min/max -14.512 17.842 2 Frob/trace 202.559 -791.202 eig real min/max -21.395 8.292 3 Frob/trace 232.961 -943.792 eig real min/max -22.338 17.566 ```