OUTPUT of run 2 of at most 100: ``` L0H0 effective QK norm 0.1137 SV [0.089 0.068 0.013 0.008]; eig [-0.049+0.j 0.006+0.009j 0.006-0.009j -0.003+0.j -0.001+0.002j -0.001-0.002j]; key/query bias norms 0.018 0.016; effective OV norm 4.321 trace 1.666 L0H1 effective QK norm 0.8775 SV [0.601 0.373 0.297 0.211]; eig [ 0.218+0.345j 0.218-0.345j -0.284+0.j -0.123+0.15j -0.123-0.15j -0.097+0.098j]; key/query bias norms 0.074 0.106; effective OV norm 3.488 trace 0.164 L0H2 effective QK norm 0.1745 SV [0.132 0.08 0.044 0.031]; eig [ 0.091+0.j 0.053+0.j -0.022+0.j -0.013+0.001j -0.013-0.001j 0.011+0.j ]; key/query bias norms 0.023 0.079; effective OV norm 2.023 trace 0.015 L0H3 effective QK norm 0.2395 SV [0.165 0.14 0.081 0.031]; eig [-0.078+0.005j -0.078-0.005j -0.02 +0.j 0.006+0.017j 0.006-0.017j 0.007+0.j ]; key/query bias norms 0.057 0.149; effective OV norm 1.621 trace 0.242 L0H4 effective QK norm 0.0093 SV [0.004 0.003 0.002 0.002]; eig [-0.001+0.001j -0.001-0.001j -0.001+0.j -0.001-0.j 0. +0.001j 0. -0.001j]; key/query bias norms 0.001 0.001; effective OV norm 17.158 trace 6.049 L0H5 effective QK norm 0.9704 SV [0.41 0.362 0.273 0.254]; eig [0.207+0.275j 0.207-0.275j 0.114+0.172j 0.114-0.172j 0.178+0.j 0.155+0.084j]; key/query bias norms 0.051 0.066; effective OV norm 3.707 trace -2.463 L1H0 effective QK norm 0.6440 SV [0.354 0.252 0.218 0.167]; eig [ 0.028+0.051j 0.028-0.051j -0.051+0.023j -0.051-0.023j -0.051+0.j -0.041+0.013j]; key/query bias norms 0.839 0.841; effective OV norm 13.436 trace -16.299 L1H1 effective QK norm 1.5843 SV [1.146 0.661 0.459 0.326]; eig [ 0.08 +0.26j 0.08 -0.26j -0.144+0.009j -0.144-0.009j -0.092+0.079j -0.092-0.079j]; key/query bias norms 3.150 2.183; effective OV norm 9.592 trace -11.205 L1H2 effective QK norm 1.2358 SV [0.807 0.693 0.332 0.205]; eig [ 0.057+0.202j 0.057-0.202j -0.108+0.j -0.07 +0.05j -0.07 -0.05j -0.079+0.004j]; key/query bias norms 2.200 1.897; effective OV norm 11.496 trace -14.892 L1H3 effective QK norm 1.3094 SV [0.913 0.699 0.309 0.202]; eig [ 0.076+0.272j 0.076-0.272j -0.094+0.j 0.092+0.j -0.08 +0.024j -0.08 -0.024j]; key/query bias norms 2.719 2.207; effective OV norm 10.043 trace -10.571 L1H4 effective QK norm 1.2492 SV [0.848 0.654 0.34 0.213]; eig [ 0.071+0.239j 0.071-0.239j -0.085+0.014j -0.085-0.014j -0.078+0.j -0.065+0.028j]; key/query bias norms 2.518 1.942; effective OV norm 9.557 trace -10.446 L1H5 effective QK norm 1.4614 SV [1.009 0.555 0.482 0.45 ]; eig [ 0.072+0.204j 0.072-0.204j -0.156+0.j -0.124+0.j -0.083+0.053j -0.083-0.053j]; key/query bias norms 2.460 1.810; effective OV norm 9.826 trace -13.045 L2H0 effective QK norm 2.0456 SV [0.518 0.485 0.459 0.416]; eig [ 0.033+0.132j 0.033-0.132j -0.117+0.j 0.091+0.074j 0.091-0.074j 0.037+0.104j]; key/query bias norms 0.895 0.774; effective OV norm 18.698 trace -35.970 L2H1 effective QK norm 1.9454 SV [0.458 0.421 0.396 0.393]; eig [0.142+0.036j 0.142-0.036j 0.142+0.j 0.141+0.011j 0.141-0.011j 0.131+0.022j]; key/query bias norms 0.638 0.333; effective OV norm 21.448 trace -40.225 L2H2 effective QK norm 2.8040 SV [0.709 0.693 0.676 0.634]; eig [0.259+0.j 0.08 +0.216j 0.08 -0.216j 0.193+0.058j 0.193-0.058j 0.195-0.026j]; key/query bias norms 1.584 1.581; effective OV norm 16.782 trace -29.462 L2H3 effective QK norm 1.9759 SV [0.571 0.423 0.412 0.392]; eig [ 0.058+0.185j 0.058-0.185j 0.002+0.106j 0.002-0.106j -0.055+0.085j -0.055-0.085j]; key/query bias norms 1.037 1.265; effective OV norm 20.617 trace -38.568 L2H4 effective QK norm 2.2016 SV [0.783 0.587 0.484 0.429]; eig [ 0.08 +0.22j 0.08 -0.22j 0.154+0.j -0.121+0.j 0.086+0.077j 0.086-0.077j]; key/query bias norms 1.199 1.340; effective OV norm 18.415 trace -38.753 L2H5 effective QK norm 1.6541 SV [0.419 0.414 0.359 0.32 ]; eig [-0.107+0.j -0.007+0.101j -0.007-0.101j -0.079+0.045j -0.079-0.045j 0.09 +0.j ]; key/query bias norms 0.636 0.492; effective OV norm 23.766 trace -33.940 L3H0 effective QK norm 2.0777 SV [0.405 0.399 0.385 0.377]; eig [0.13 +0.j 0.119-0.04j 0.119+0.04j 0.119-0.018j 0.119+0.018j 0.109+0.j ]; key/query bias norms 0.299 0.302; effective OV norm 24.502 trace -48.319 L3H1 effective QK norm 1.6948 SV [0.322 0.315 0.309 0.304]; eig [0.11 +0.027j 0.11 -0.027j 0.112+0.j 0.075+0.071j 0.075-0.071j 0.102+0.j ]; key/query bias norms 0.263 0.253; effective OV norm 25.724 trace -49.613 L3H2 effective QK norm 2.0608 SV [0.434 0.411 0.392 0.362]; eig [0.036+0.114j 0.036-0.114j 0.114-0.026j 0.114+0.026j 0.003+0.11j 0.003-0.11j ]; key/query bias norms 0.342 0.286; effective OV norm 26.860 trace -50.565 L3H3 effective QK norm 1.0530 SV [0.262 0.237 0.208 0.202]; eig [ 0.027+0.061j 0.027-0.061j 0.06 +0.j 0.026+0.049j 0.026-0.049j -0.002-0.051j]; key/query bias norms 0.258 0.160; effective OV norm 37.938 trace -51.153 L3H4 effective QK norm 1.7344 SV [0.401 0.369 0.357 0.323]; eig [-0.105+0.j 0.093+0.j 0.066+0.059j 0.066-0.059j 0.029+0.079j 0.029-0.079j]; key/query bias norms 0.307 0.239; effective OV norm 28.440 trace -35.013 L3H5 effective QK norm 2.1169 SV [0.404 0.39 0.384 0.37 ]; eig [0.195+0.117j 0.195-0.117j 0.213-0.074j 0.213+0.074j 0.2 +0.104j 0.2 -0.104j]; key/query bias norms 0.310 0.307; effective OV norm 23.687 trace -56.843 NORMALIZED TRACE(QK_l @ OV_earlier), columns earlier heads QK layer 1 OV layer 0 [[ 0.007 -0. -0.008 0.005 0.021 -0.003] [-0.005 0.007 -0.016 0.006 0.019 0.015] [ 0.004 0.001 -0.017 0.009 0.033 -0.003] [ 0.008 -0.003 -0.013 0.006 0.042 0.012] [ 0.009 0.002 -0.018 0.011 0.044 0.009] [-0.019 0.005 -0.013 -0. -0.004 0.013]] QK layer 2 OV layer 0 [[ 0.005 -0.001 -0.003 0.008 0.01 -0.005] [ 0.007 0.001 0.011 0.003 0.004 0.078] [ 0.011 -0.017 -0.005 0.001 0.02 -0.039] [ 0.018 -0.001 -0.007 0.007 0.04 -0.003] [ 0.008 -0.006 -0.004 0.01 0.025 0.002] [ 0.009 0.001 0. 0.005 0.019 0.007]] QK layer 2 OV layer 1 [[-0.013 0.004 -0. 0.002 -0. 0.006] [ 0.003 0.048 0.048 0.057 0.048 0.035] [-0.042 -0.016 -0.043 -0.007 -0.023 -0.007] [-0.02 0.005 -0.016 0.004 -0.005 0.004] [-0.014 -0.008 -0.011 -0.002 -0.004 -0.006] [ 0. 0.008 0.009 0.006 0.007 0.007]] QK layer 3 OV layer 0 [[ 0.01 -0.004 0.007 0.008 0.01 0.019] [ 0.008 0.007 0.006 0.001 0.004 0.061] [ 0.006 -0.003 0.002 -0.001 0.013 -0.001] [ 0.004 0.003 0.005 0.003 0.007 0.001] [ 0.009 -0.006 0.006 0.003 0.012 0.01 ] [ 0.006 -0.009 0. 0.003 0.003 0.071]] QK layer 3 OV layer 1 [[ 0.008 0.008 0.01 0.006 0.006 0.001] [ 0.013 0.033 0.038 0.045 0.046 0.027] [-0.012 0.001 -0.009 0. -0.009 -0.004] [ 0.005 0.002 0.008 -0. 0.002 0.004] [ 0. 0.007 0.003 0.005 0.005 0.008] [ 0.004 0.042 0.048 0.068 0.057 0.02 ]] QK layer 3 OV layer 2 [[-0.003 -0.013 0.024 -0.001 0.004 -0.009] [ 0.015 -0.018 0.11 0.021 0.005 -0. ] [-0.005 -0.007 -0. -0.003 -0.001 -0.003] [ 0.006 -0.005 0.009 0.01 0.015 0.005] [ 0.001 -0.008 0.008 0.004 0.01 -0.004] [-0.008 -0.06 0.184 0.055 -0.022 -0.022]] L0: overlap of top QK singular subspaces with top value-read subspace Combined effective V0 singular values: [3.195 2.304 1.316 0.88 0.809 0.771 0.73 0.704 0.685 0.662 0.638 0.629 0.623 0.621 0.604 0.595 0.582 0.567 0.554 0.55 0.542 0.541 0.522 0.513 0.496 0.487 0.479 0.475 0.464 0.457 0.447 0.431 0.428 0.424 0.41 0.399 0.389 0.383 0.373 0.369 0.358 0.345 0.334 0.333 0.319] L0H0 V-rank8 query/key energy 0.744 0.374 L0H0 V-rank24 query/key energy 0.794 0.603 L0H1 V-rank8 query/key energy 0.590 0.611 L0H1 V-rank24 query/key energy 0.630 0.652 L0H2 V-rank8 query/key energy 0.712 0.559 L0H2 V-rank24 query/key energy 0.778 0.732 L0H3 V-rank8 query/key energy 0.632 0.587 L0H3 V-rank24 query/key energy 0.691 0.746 L0H4 V-rank8 query/key energy 0.050 0.099 L0H4 V-rank24 query/key energy 0.116 0.193 L0H5 V-rank8 query/key energy 0.276 0.288 L0H5 V-rank24 query/key energy 0.320 0.336 L0 QK pairwise Frobenius cosines [[ 1. 0.179 0.112 0.206 0.063 0.093] [ 0.179 1. 0.21 -0.15 0.005 0.414] [ 0.112 0.21 1. -0.247 -0.006 0.114] [ 0.206 -0.15 -0.247 1. -0.027 -0.098] [ 0.063 0.005 -0.006 -0.027 1. -0.004] [ 0.093 0.414 0.114 -0.098 -0.004 1. ]] L0 V-head top-24 right-subspace overlaps (mean squared cosine) [[1. 0.219 0.254 0.277 0.347 0.217] [0.219 1. 0.325 0.314 0.324 0.619] [0.254 0.325 1. 0.296 0.25 0.291] [0.277 0.314 0.296 1. 0.28 0.264] [0.347 0.324 0.25 0.28 1. 0.35 ] [0.217 0.619 0.291 0.264 0.35 1. ]] ```