OUTPUT of run 2 of at most 100: ``` WITHIN-LAYER QK COSINES, THEN OV COSINES L 0 QK [[ 1. 0.012 0.007 0.137 0.013 0.044] [ 0.012 1. 0.69 -0.026 0.57 0.026] [ 0.007 0.69 1. -0.063 0.492 -0.075] [ 0.137 -0.026 -0.063 1. -0.044 0.01 ] [ 0.013 0.57 0.492 -0.044 1. 0.033] [ 0.044 0.026 -0.075 0.01 0.033 1. ]] OV [[ 1. 0.018 0.017 -0.035 0.026 0.018] [ 0.018 1. 0.925 0.006 0.868 0.643] [ 0.017 0.925 1. 0.009 0.81 0.692] [-0.035 0.006 0.009 1. 0.011 -0.001] [ 0.026 0.868 0.81 0.011 1. 0.592] [ 0.018 0.643 0.692 -0.001 0.592 1. ]] L 1 QK [[1. 0.358 0.373 0.346 0.334 0.377] [0.358 1. 0.372 0.362 0.302 0.399] [0.373 0.372 1. 0.368 0.375 0.352] [0.346 0.362 0.368 1. 0.32 0.396] [0.334 0.302 0.375 0.32 1. 0.289] [0.377 0.399 0.352 0.396 0.289 1. ]] OV [[ 1. -0.02 0.03 -0.012 0.001 0.029] [-0.02 1. -0.009 -0.028 -0.018 0.016] [ 0.03 -0.009 1. 0.001 0.014 -0.017] [-0.012 -0.028 0.001 1. -0.001 -0.015] [ 0.001 -0.018 0.014 -0.001 1. -0.014] [ 0.029 0.016 -0.017 -0.015 -0.014 1. ]] L 2 QK [[1. 0.129 0.161 0.117 0.18 0.135] [0.129 1. 0.116 0.086 0.121 0.117] [0.161 0.116 1. 0.101 0.145 0.124] [0.117 0.086 0.101 1. 0.093 0.093] [0.18 0.121 0.145 0.093 1. 0.134] [0.135 0.117 0.124 0.093 0.134 1. ]] OV [[ 1. -0.023 -0.039 -0.022 -0.005 0.001] [-0.023 1. -0.001 0.002 -0.012 -0.008] [-0.039 -0.001 1. 0.013 0.004 0.022] [-0.022 0.002 0.013 1. 0.001 0. ] [-0.005 -0.012 0.004 0.001 1. 0.004] [ 0.001 -0.008 0.022 0. 0.004 1. ]] L 3 QK [[1. 0.058 0.083 0.06 0.007 0.108] [0.058 1. 0.07 0.034 0.042 0.075] [0.083 0.07 1. 0.031 0.036 0.094] [0.06 0.034 0.031 1. 0.037 0.058] [0.007 0.042 0.036 0.037 1. 0.028] [0.108 0.075 0.094 0.058 0.028 1. ]] OV [[ 1. -0.012 0.024 0.011 -0.002 0.034] [-0.012 1. -0.021 0.024 0.059 0.001] [ 0.024 -0.021 1. 0.003 0.077 0.026] [ 0.011 0.024 0.003 1. 0.042 -0.002] [-0.002 0.059 0.077 0.042 1. 0.057] [ 0.034 0.001 0.026 -0.002 0.057 1. ]] L0 WEIGHT-ONLY EIGENSPECTRA AND ALIGNMENTS H0 raw: eig_top12=[ 0.113+0.216j 0.113-0.216j -0.166+0.039j -0.166-0.039j -0.141+0.034j -0.141-0.034j -0.11 +0.061j -0.11 -0.061j -0.09 +0.068j -0.09 -0.068j -0.106+0.j -0.092+0.009j], sv_top8=[0.853 0.613 0.368 0.287 0.279 0.244 0.232 0.21 ] H0 gain: eig_top12=[ 0.079+0.j 0.049+0.j -0.024+0.006j -0.024-0.006j -0.023+0.j 0.018+0.013j 0.018-0.013j -0.02 +0.j -0.014+0.012j -0.014-0.012j -0.014+0.003j -0.014-0.003j], sv_top8=[0.248 0.102 0.047 0.043 0.037 0.036 0.034 0.031] top left/right cos [ 0.488 -0.117 0.501 -0.2 -0.489] top left/right coordinate concentration [0.094 0.147 0.063 0.067 0.065] [0.103 0.085 0.084 0.088 0.08 ] H1 raw: eig_top12=[-0.334+0.j -0.225+0.j -0.067+0.012j -0.067-0.012j -0.048+0.j 0.034+0.j -0.027+0.j -0.023+0.j -0.01 +0.004j -0.01 -0.004j -0.009+0.002j -0.009-0.002j], sv_top8=[0.838 0.311 0.219 0.125 0.091 0.086 0.066 0.039] H1 gain: eig_top12=[-0.062+0.j -0.01 +0.j -0.009+0.j 0.004+0.j -0.004+0.j -0.003+0.002j -0.003-0.002j -0.002-0.j -0.002+0.j -0.002+0.001j -0.002-0.001j -0.001+0.j ], sv_top8=[0.15 0.042 0.021 0.012 0.01 0.007 0.007 0.004] top left/right cos [-0.406 -0.111 -0.405 -0.516 -0.422] top left/right coordinate concentration [0.138 0.115 0.048 0.05 0.299] [0.15 0.046 0.023 0.048 0.049] H2 raw: eig_top12=[-0.237+0.j -0.095+0.029j -0.095-0.029j -0.061+0.j -0.016+0.011j -0.016-0.011j -0.014+0.j -0.011+0.005j -0.011-0.005j -0.008-0.j -0.008+0.j 0. -0.007j], sv_top8=[0.543 0.183 0.157 0.133 0.072 0.066 0.046 0.046] H2 gain: eig_top12=[-0.057+0.j -0.007+0.003j -0.007-0.003j -0.003+0.001j -0.003-0.001j -0.003+0.j -0.001+0.j -0.001-0.j -0.001+0.001j -0.001-0.001j 0.001+0.001j 0.001-0.001j], sv_top8=[0.102 0.018 0.016 0.012 0.008 0.007 0.005 0.004] top left/right cos [-0.539 -0.388 -0.501 -0.573 -0.215] top left/right coordinate concentration [0.155 0.077 0.185 0.049 0.061] [0.12 0.044 0.048 0.051 0.057] H3 raw: eig_top12=[0.724+0.691j 0.724-0.691j 0.446+0.842j 0.446-0.842j 0.572+0.722j 0.572-0.722j 0.642+0.614j 0.642-0.614j 0.676+0.463j 0.676-0.463j 0.109+0.762j 0.109-0.762j], sv_top8=[1.114 1.092 1.07 1.064 1.03 0.992 0.968 0.954] H3 gain: eig_top12=[0.137+0.133j 0.137-0.133j 0.11 +0.089j 0.11 -0.089j 0.095+0.098j 0.095-0.098j 0.084+0.104j 0.084-0.104j 0.064+0.114j 0.064-0.114j 0.096+0.068j 0.096-0.068j], sv_top8=[0.244 0.223 0.201 0.182 0.179 0.157 0.155 0.149] top left/right cos [0.658 0.673 0.628 0.553 0.593] top left/right coordinate concentration [0.096 0.336 0.126 0.116 0.074] [0.317 0.388 0.196 0.174 0.302] H4 raw: eig_top12=[-0.172+0.j -0.102+0.013j -0.102-0.013j -0.05 +0.j -0.034+0.j -0.018+0.006j -0.018-0.006j -0.009+0.009j -0.009-0.009j -0.009+0.005j -0.009-0.005j 0.01 +0.j ], sv_top8=[0.51 0.227 0.185 0.124 0.101 0.066 0.065 0.049] H4 gain: eig_top12=[-0.015+0.j -0.01 +0.j -0.008+0.j -0.003+0.j -0.003-0.j -0.002+0.001j -0.002-0.001j -0. +0.002j -0. -0.002j -0.001+0.001j -0.001-0.001j -0.002+0.j ], sv_top8=[0.08 0.022 0.019 0.013 0.011 0.008 0.007 0.006] top left/right cos [-0.166 -0.48 -0.366 -0.27 -0.135] top left/right coordinate concentration [0.185 0.21 0.069 0.062 0.075] [0.079 0.1 0.047 0.059 0.073] H5 raw: eig_top12=[-0.028+0.j 0.019+0.j -0.008+0.j -0.005+0.005j -0.005-0.005j -0.005+0.002j -0.005-0.002j -0.004+0.004j -0.004-0.004j -0.005+0.002j -0.005-0.002j 0. +0.005j], sv_top8=[0.18 0.048 0.024 0.02 0.02 0.019 0.019 0.018] H5 gain: eig_top12=[-0.005+0.j 0.001+0.j -0.001+0.j -0. +0.001j -0. -0.001j -0.001+0.j -0.001+0.j -0.001-0.j 0. +0.001j 0. -0.001j -0. +0.001j -0. -0.001j], sv_top8=[0.032 0.004 0.003 0.003 0.002 0.002 0.002 0.002] top left/right cos [-0.145 0.202 -0.338 -0.096 -0.131] top left/right coordinate concentration [0.121 0.049 0.111 0.07 0.074] [0.049 0.045 0.187 0.067 0.074] L0 QK TOP-DIRECTION COSINES (signed arbitrary; absolute values shown) left [[1. 0.138 0.075 0.626 0.089 0.912] [0.138 1. 0.887 0.142 0.914 0.067] [0.075 0.887 1. 0.453 0.869 0.18 ] [0.626 0.142 0.453 1. 0.185 0.662] [0.089 0.914 0.869 0.185 1. 0.06 ] [0.912 0.067 0.18 0.662 0.06 1. ]] right [[1. 0.053 0.256 0.533 0.302 0.043] [0.053 1. 0.806 0.047 0.686 0.324] [0.256 0.806 1. 0.304 0.597 0.528] [0.533 0.047 0.304 1. 0.065 0.106] [0.302 0.686 0.597 0.065 1. 0.37 ] [0.043 0.324 0.528 0.106 0.37 1. ]] ATTENTION AFFINE PARAMETER NORMS (no activations) L0H0: ||qD||=2.753 ||kD||=3.026 ||qb||=0.504 ||kb||=0.725 ||key_linear||=0.043 ||query_linear||=0.106 L0H1: ||qD||=1.904 ||kD||=1.535 ||qb||=0.228 ||kb||=0.261 ||key_linear||=0.013 ||query_linear||=0.027 L0H2: ||qD||=1.628 ||kD||=1.389 ||qb||=0.146 ||kb||=0.314 ||key_linear||=0.007 ||query_linear||=0.033 L0H3: ||qD||=5.150 ||kD||=6.350 ||qb||=0.467 ||kb||=0.543 ||key_linear||=0.043 ||query_linear||=0.052 L0H4: ||qD||=1.611 ||kD||=1.432 ||qb||=0.270 ||kb||=0.550 ||key_linear||=0.017 ||query_linear||=0.069 L0H5: ||qD||=1.080 ||kD||=0.938 ||qb||=0.143 ||kb||=0.171 ||key_linear||=0.005 ||query_linear||=0.015 L1H0: ||qD||=8.551 ||kD||=8.825 ||qb||=6.688 ||kb||=13.795 ||key_linear||=3.975 ||query_linear||=2.983 L1H1: ||qD||=8.267 ||kD||=8.591 ||qb||=6.079 ||kb||=13.929 ||key_linear||=3.689 ||query_linear||=2.828 L1H2: ||qD||=8.040 ||kD||=8.321 ||qb||=5.564 ||kb||=13.162 ||key_linear||=3.052 ||query_linear||=2.523 L1H3: ||qD||=8.128 ||kD||=8.415 ||qb||=5.665 ||kb||=13.742 ||key_linear||=3.391 ||query_linear||=2.648 L1H4: ||qD||=7.819 ||kD||=7.741 ||qb||=5.347 ||kb||=10.986 ||key_linear||=2.329 ||query_linear||=2.037 L1H5: ||qD||=8.316 ||kD||=8.794 ||qb||=6.224 ||kb||=14.146 ||key_linear||=3.896 ||query_linear||=2.857 L2H0: ||qD||=9.875 ||kD||=9.920 ||qb||=4.265 ||kb||=10.635 ||key_linear||=2.021 ||query_linear||=1.811 L2H1: ||qD||=11.498 ||kD||=11.425 ||qb||=3.787 ||kb||=10.117 ||key_linear||=1.881 ||query_linear||=1.635 L2H2: ||qD||=10.317 ||kD||=10.337 ||qb||=3.784 ||kb||=9.886 ||key_linear||=1.609 ||query_linear||=1.418 L2H3: ||qD||=10.057 ||kD||=9.734 ||qb||=3.082 ||kb||=7.454 ||key_linear||=1.060 ||query_linear||=0.841 L2H4: ||qD||=11.573 ||kD||=11.405 ||qb||=4.319 ||kb||=11.042 ||key_linear||=2.307 ||query_linear||=1.981 L2H5: ||qD||=11.724 ||kD||=11.635 ||qb||=3.751 ||kb||=10.382 ||key_linear||=2.024 ||query_linear||=1.749 L3H0: ||qD||=11.379 ||kD||=11.100 ||qb||=1.761 ||kb||=3.665 ||key_linear||=0.501 ||query_linear||=0.447 L3H1: ||qD||=8.763 ||kD||=8.475 ||qb||=1.460 ||kb||=3.019 ||key_linear||=0.280 ||query_linear||=0.306 L3H2: ||qD||=9.584 ||kD||=9.387 ||qb||=1.940 ||kb||=3.360 ||key_linear||=0.495 ||query_linear||=0.341 L3H3: ||qD||=10.731 ||kD||=10.630 ||qb||=1.298 ||kb||=1.116 ||key_linear||=0.221 ||query_linear||=0.181 L3H4: ||qD||=6.101 ||kD||=5.901 ||qb||=1.230 ||kb||=1.267 ||key_linear||=0.144 ||query_linear||=0.117 L3H5: ||qD||=10.323 ||kD||=10.032 ||qb||=1.800 ||kb||=3.333 ||key_linear||=0.493 ||query_linear||=0.422 KEY-SIDE COMPOSITION: trace(A_l B_earlier)/||A_l B_earlier||_F These are matrix-path diagnostics, not observed attention or induction. L1H0 [(np.float64(0.134), 'L0H2', np.float64(2.261)), (np.float64(0.102), 'L0H3', np.float64(0.399)), (np.float64(0.1), 'L0H5', np.float64(17.379)), (np.float64(0.096), 'L0H4', np.float64(1.016)), (np.float64(0.094), 'L0H1', np.float64(2.843))] L1H1 [(np.float64(0.216), 'L0H3', np.float64(0.422)), (np.float64(0.065), 'L0H2', np.float64(2.168)), (np.float64(0.041), 'L0H5', np.float64(16.984)), (np.float64(0.018), 'L0H0', np.float64(0.115)), (np.float64(0.017), 'L0H1', np.float64(2.729))] L1H2 [(np.float64(0.167), 'L0H3', np.float64(0.455)), (np.float64(0.106), 'L0H2', np.float64(2.141)), (np.float64(0.1), 'L0H4', np.float64(0.941)), (np.float64(0.09), 'L0H5', np.float64(17.046)), (np.float64(0.078), 'L0H1', np.float64(2.692))] L1H3 [(np.float64(0.113), 'L0H2', np.float64(2.176)), (np.float64(0.082), 'L0H5', np.float64(17.098)), (np.float64(0.076), 'L0H3', np.float64(0.429)), (np.float64(0.075), 'L0H4', np.float64(0.971)), (np.float64(0.073), 'L0H1', np.float64(2.741))] L1H4 [(np.float64(0.08), 'L0H2', np.float64(1.852)), (np.float64(0.078), 'L0H4', np.float64(0.815)), (np.float64(0.074), 'L0H5', np.float64(15.135)), (np.float64(0.046), 'L0H1', np.float64(2.335)), (np.float64(0.02), 'L0H0', np.float64(0.122))] L1H5 [(np.float64(0.198), 'L0H3', np.float64(0.37)), (np.float64(0.102), 'L0H2', np.float64(2.195)), (np.float64(0.062), 'L0H5', np.float64(16.793)), (np.float64(0.061), 'L0H4', np.float64(0.992)), (np.float64(0.052), 'L0H1', np.float64(2.768))] L2H0 [(np.float64(0.208), 'L1H3', np.float64(0.907)), (np.float64(0.196), 'L0H2', np.float64(1.477)), (np.float64(0.189), 'L0H5', np.float64(10.681)), (np.float64(0.164), 'L1H1', np.float64(0.787)), (np.float64(0.133), 'L0H1', np.float64(1.866))] L2H1 [(np.float64(0.197), 'L0H5', np.float64(8.667)), (np.float64(0.179), 'L0H2', np.float64(1.219)), (np.float64(0.116), 'L0H1', np.float64(1.55)), (np.float64(0.08), 'L0H4', np.float64(0.587)), (np.float64(0.035), 'L0H0', np.float64(0.193))] L2H2 [(np.float64(0.184), 'L0H5', np.float64(8.027)), (np.float64(0.182), 'L0H2', np.float64(1.12)), (np.float64(0.123), 'L0H1', np.float64(1.427)), (np.float64(0.099), 'L0H3', np.float64(0.321)), (np.float64(0.075), 'L1H1', np.float64(0.899))] L2H3 [(np.float64(0.129), 'L0H2', np.float64(0.859)), (np.float64(0.11), 'L0H5', np.float64(6.748)), (np.float64(0.096), 'L0H1', np.float64(1.089)), (np.float64(0.084), 'L0H4', np.float64(0.387)), (np.float64(0.082), 'L0H3', np.float64(0.328))] L2H4 [(np.float64(0.186), 'L0H2', np.float64(1.634)), (np.float64(0.18), 'L0H5', np.float64(11.718)), (np.float64(0.159), 'L0H3', np.float64(0.43)), (np.float64(0.128), 'L0H1', np.float64(2.055)), (np.float64(0.109), 'L0H4', np.float64(0.75))] L2H5 [(np.float64(0.21), 'L0H5', np.float64(9.536)), (np.float64(0.187), 'L0H2', np.float64(1.329)), (np.float64(0.112), 'L0H1', np.float64(1.69)), (np.float64(0.081), 'L0H4', np.float64(0.639)), (np.float64(0.018), 'L0H0', np.float64(0.208))] L3H0 [(np.float64(0.107), 'L0H2', np.float64(0.735)), (np.float64(0.085), 'L0H5', np.float64(5.722)), (np.float64(0.069), 'L0H1', np.float64(0.946)), (np.float64(0.047), 'L0H4', np.float64(0.351)), (np.float64(-0.049), 'L0H0', np.float64(0.218))] L3H1 [(np.float64(0.093), 'L0H2', np.float64(0.533)), (np.float64(0.075), 'L0H5', np.float64(4.189)), (np.float64(0.072), 'L1H2', np.float64(0.667)), (np.float64(0.069), 'L0H1', np.float64(0.681)), (np.float64(0.051), 'L1H1', np.float64(0.643))] L3H2 [(np.float64(0.081), 'L0H2', np.float64(0.644)), (np.float64(0.046), 'L0H1', np.float64(0.835)), (np.float64(0.033), 'L0H3', np.float64(0.259)), (np.float64(0.016), 'L0H5', np.float64(5.116)), (np.float64(0.005), 'L1H2', np.float64(0.771))] L3H3 [(np.float64(0.922), 'L0H3', np.float64(0.435)), (np.float64(0.806), 'L2H5', np.float64(1.826)), (np.float64(0.562), 'L2H1', np.float64(1.866)), (np.float64(0.514), 'L1H0', np.float64(1.236)), (np.float64(0.472), 'L1H5', np.float64(1.124))] L3H4 [(np.float64(0.276), 'L2H2', np.float64(0.593)), (np.float64(0.272), 'L2H4', np.float64(0.415)), (np.float64(0.202), 'L2H5', np.float64(0.486)), (np.float64(0.19), 'L0H4', np.float64(0.057)), (np.float64(0.182), 'L2H1', np.float64(0.512))] L3H5 [(np.float64(0.05), 'L0H2', np.float64(0.738)), (np.float64(0.047), 'L0H5', np.float64(5.814)), (np.float64(0.016), 'L0H1', np.float64(0.945)), (np.float64(-0.001), 'L0H4', np.float64(0.337)), (np.float64(-0.008), 'L0H0', np.float64(0.181))] ```