All calculations below are matrix products or decompositions, not layer evaluations.

LAYER 0
L0H0 Q/K/V/O norms [3.305 4.18  6.039 6.666] effective QK norm/top/sym/trace [ 0.0546  0.0514  0.0233 -0.0047] query_bias_ratio 0.214 key_bias_norm 0.0136 QKev [ 0.007+0.j    -0.004+0.j    -0.002+0.j    -0.   +0.001j -0.   -0.001j  0.   +0.001j  0.   -0.001j -0.001+0.001j]
L0H1 Q/K/V/O norms [3.09  3.658 5.382 6.025] effective QK norm/top/sym/trace [ 0.0295  0.025   0.0475 -0.014 ] query_bias_ratio 0.1243 key_bias_norm 0.0033 QKev [-0.004+0.001j -0.004-0.001j  0.002+0.j    -0.001+0.001j -0.001-0.001j -0.001+0.j    -0.001+0.001j -0.001-0.001j]
L0H2 Q/K/V/O norms [30.818 28.3    7.765  8.915] effective QK norm/top/sym/trace [ 4.0688  1.4079 -0.0446  1.199 ] query_bias_ratio 0.0589 key_bias_norm 0.1428 QKev [ 0.632+0.813j  0.632-0.813j -0.231+0.927j -0.231-0.927j  0.332+0.871j  0.332-0.871j -0.777+0.334j -0.777-0.334j]
L0H3 Q/K/V/O norms [6.821 7.243 5.434 5.029] effective QK norm/top/sym/trace [0.3945 0.3588 0.4782 0.1968] query_bias_ratio 0.1771 key_bias_norm 0.0535 QKev [ 0.262+0.j     0.078+0.j     0.01 +0.031j  0.01 -0.031j -0.017+0.005j -0.017-0.005j -0.017+0.j    -0.015+0.003j]
L0H4 Q/K/V/O norms [3.971 4.995 5.208 5.587] effective QK norm/top/sym/trace [ 0.1062  0.089   0.004  -0.0707] query_bias_ratio 0.1505 key_bias_norm 0.0112 QKev [-0.028+0.028j -0.028-0.028j -0.008+0.j    -0.002+0.j    -0.001+0.001j -0.001-0.001j -0.002+0.j     0.001+0.001j]
L0H5 Q/K/V/O norms [3.19  3.802 6.143 7.162] effective QK norm/top/sym/trace [ 0.0775  0.0759  0.1603 -0.0347] query_bias_ratio 0.1849 key_bias_norm 0.0162 QKev [-0.031+0.j     0.001+0.002j  0.001-0.002j -0.001+0.j    -0.001-0.j    -0.001+0.001j -0.001-0.001j  0.001+0.001j]

LAYER 1
L1H0 Q/K/V/O norms [10.716  9.827 11.156  9.984] effective QK norm/top/sym/trace [ 0.8864  0.5969  0.0086 -0.6436] query_bias_ratio 0.7821 key_bias_norm 1.3572 QKev [ 0.028+0.106j  0.028-0.106j -0.078+0.j    -0.066+0.028j -0.066-0.028j -0.055+0.007j -0.055-0.007j -0.047+0.j   ]
L1H1 Q/K/V/O norms [13.292 12.068 11.72  11.303] effective QK norm/top/sym/trace [ 1.3146  0.8811  0.0186 -1.1603] query_bias_ratio 0.8034 key_bias_norm 2.2848 QKev [ 0.07 +0.191j  0.07 -0.191j -0.153+0.j    -0.107+0.024j -0.107-0.024j -0.108+0.006j -0.108-0.006j -0.086+0.j   ]
L1H2 Q/K/V/O norms [14.476 13.092 10.668 10.272] effective QK norm/top/sym/trace [ 1.5491  0.9943  0.0179 -1.6165] query_bias_ratio 0.6959 key_bias_norm 2.5879 QKev [ 0.097+0.246j  0.097-0.246j -0.168+0.j    -0.141+0.064j -0.141-0.064j -0.135+0.009j -0.135-0.009j -0.104+0.03j ]
L1H3 Q/K/V/O norms [13.358 12.153 10.904  9.928] effective QK norm/top/sym/trace [ 1.3742  0.9188  0.0172 -1.1551] query_bias_ratio 0.7616 key_bias_norm 2.3511 QKev [ 0.094+0.203j  0.094-0.203j -0.175+0.j    -0.113+0.008j -0.113-0.008j -0.107+0.j    -0.04 +0.091j -0.04 -0.091j]
L1H4 Q/K/V/O norms [13.518 12.129 11.148 10.794] effective QK norm/top/sym/trace [ 1.4497  0.9476  0.0108 -1.2432] query_bias_ratio 0.711 key_bias_norm 2.2902 QKev [ 0.062+0.214j  0.062-0.214j -0.135+0.01j  -0.135-0.01j  -0.127+0.j    -0.099+0.j    -0.081+0.053j -0.081-0.053j]
L1H5 Q/K/V/O norms [13.486 12.132 11.376 11.136] effective QK norm/top/sym/trace [ 1.3363  0.8925 -0.0027 -1.0127] query_bias_ratio 0.6733 key_bias_norm 1.7526 QKev [ 0.088+0.221j  0.088-0.221j -0.121+0.012j -0.121-0.012j -0.1  +0.004j -0.1  -0.004j -0.074+0.055j -0.074-0.055j]

LAYER 2
L2H0 Q/K/V/O norms [16.822 16.208 13.115 12.942] effective QK norm/top/sym/trace [ 2.4375  0.6943 -0.0335 -0.0216] query_bias_ratio 0.3088 key_bias_norm 1.6991 QKev [ 0.053+0.306j  0.053-0.306j  0.036+0.183j  0.036-0.183j  0.118+0.08j   0.118-0.08j  -0.077+0.098j -0.077-0.098j]
L2H1 Q/K/V/O norms [15.963 15.287 13.768 13.525] effective QK norm/top/sym/trace [2.0677 0.5267 0.0101 0.53  ] query_bias_ratio 0.2076 key_bias_norm 0.7446 QKev [ 0.027+0.144j  0.027-0.144j  0.126+0.011j  0.126-0.011j  0.109+0.037j  0.109-0.037j -0.084+0.071j -0.084-0.071j]
L2H2 Q/K/V/O norms [13.97  13.239 14.567 14.24 ] effective QK norm/top/sym/trace [ 1.6127  0.6069 -0.0046 -1.1051] query_bias_ratio 0.2312 key_bias_norm 0.7554 QKev [ 0.037+0.175j  0.037-0.175j -0.106+0.j     0.067+0.055j  0.067-0.055j -0.084+0.015j -0.084-0.015j -0.076-0.014j]
L2H3 Q/K/V/O norms [16.121 15.554 13.498 13.391] effective QK norm/top/sym/trace [ 2.0862  0.5616 -0.0268 -0.7372] query_bias_ratio 0.2623 key_bias_norm 1.4625 QKev [ 0.072+0.307j  0.072-0.307j -0.076+0.084j -0.076-0.084j -0.102+0.042j -0.102-0.042j -0.104+0.005j -0.104-0.005j]
L2H4 Q/K/V/O norms [16.876 16.188 13.241 13.143] effective QK norm/top/sym/trace [ 2.3438  0.6278 -0.0248 -0.4747] query_bias_ratio 0.2685 key_bias_norm 1.5774 QKev [ 0.061+0.303j  0.061-0.303j  0.077+0.102j  0.077-0.102j -0.081+0.091j -0.081-0.091j -0.118+0.j    -0.045+0.105j]
L2H5 Q/K/V/O norms [16.445 15.839 13.339 13.161] effective QK norm/top/sym/trace [ 2.2503  0.6375 -0.0283 -0.6957] query_bias_ratio 0.2779 key_bias_norm 1.5086 QKev [ 0.046+0.302j  0.046-0.302j -0.141+0.j    -0.071-0.108j -0.071+0.108j -0.119+0.037j -0.119-0.037j -0.068+0.099j]

LAYER 3
L3H0 Q/K/V/O norms [15.747 15.258 14.874 14.87 ] effective QK norm/top/sym/trace [2.2322 0.4514 0.0503 3.4122] query_bias_ratio 0.1057 key_bias_norm 0.2581 QKev [0.182+0.j    0.141+0.078j 0.141-0.078j 0.158+0.019j 0.158-0.019j 0.143+0.056j 0.143-0.056j 0.141+0.036j]
L3H1 Q/K/V/O norms [14.22  14.197 17.885 18.331] effective QK norm/top/sym/trace [1.8295 0.3275 0.2114 6.3101] query_bias_ratio 0.1322 key_bias_norm 0.2408 QKev [0.174-0.06j  0.174+0.06j  0.171-0.067j 0.171+0.067j 0.149-0.082j 0.149+0.082j 0.164-0.031j 0.164+0.031j]
L3H2 Q/K/V/O norms [12.743 12.446 15.523 15.585] effective QK norm/top/sym/trace [ 1.505   0.4702 -0.0036  0.2216] query_bias_ratio 0.124 key_bias_norm 0.2895 QKev [ 0.03 +0.134j  0.03 -0.134j  0.081+0.038j  0.081-0.038j -0.035+0.082j -0.035-0.082j  0.082+0.j    -0.077+0.j   ]
L3H3 Q/K/V/O norms [13.283 13.057 15.732 16.118] effective QK norm/top/sym/trace [ 1.5985  0.4973 -0.0166  0.1116] query_bias_ratio 0.121 key_bias_norm 0.3603 QKev [ 0.023+0.159j  0.023-0.159j  0.099+0.j    -0.023+0.095j -0.023-0.095j  0.076+0.036j  0.076-0.036j  0.078+0.j   ]
L3H4 Q/K/V/O norms [14.356 13.874 15.175 15.3  ] effective QK norm/top/sym/trace [ 1.8555  0.5287 -0.0005  0.9035] query_bias_ratio 0.1217 key_bias_norm 0.3072 QKev [0.068+0.092j 0.068-0.092j 0.002+0.108j 0.002-0.108j 0.098+0.04j  0.098-0.04j  0.102+0.j    0.07 +0.066j]
L3H5 Q/K/V/O norms [ 9.514  9.231 17.993 18.512] effective QK norm/top/sym/trace [0.8339 0.2591 0.0242 0.1007] query_bias_ratio 0.1537 key_bias_norm 0.1645 QKev [0.075+0.035j 0.075-0.035j 0.078+0.j    0.02 +0.048j 0.02 -0.048j 0.023+0.036j 0.023-0.036j 0.038+0.011j]

SINGULAR VALUES: first 45 for selected matrices
W_q.0 [8.252 8.223 7.963 7.819 7.68  7.642 7.519 7.205 6.948 6.819 6.388 6.165 5.801 5.555 5.13  4.947 4.854 4.52  4.268 3.951 3.591 3.527 3.285 3.125 3.026 2.928 2.707 2.511 2.471
 2.336 2.249 2.136 2.089 2.058 1.914 1.792 1.755 1.679 1.536 1.513 1.475 1.455 1.327 1.31  1.282]
W_k.0 [7.646 7.436 7.33  6.912 6.829 6.672 6.472 6.446 6.393 6.118 6.009 5.943 5.629 5.505 4.99  4.901 4.698 4.293 4.066 4.018 3.708 3.529 3.343 3.068 2.987 2.874 2.696 2.586 2.412
 2.262 2.216 2.11  2.032 1.876 1.744 1.733 1.697 1.666 1.547 1.514 1.441 1.402 1.386 1.334 1.29 ]
W_v.0 [4.339 3.529 3.27  3.166 2.982 2.647 2.589 2.552 2.436 2.381 2.256 2.162 2.13  2.103 2.042 2.017 2.003 1.973 1.928 1.897 1.843 1.787 1.711 1.665 1.628 1.564 1.558 1.441 1.4
 1.377 1.347 1.267 1.222 1.159 1.086 1.046 1.016 0.999 0.945 0.918 0.902 0.894 0.875 0.87  0.841]
W_o.0 [7.103 6.607 4.268 2.77  2.503 2.285 2.264 2.218 2.151 2.139 2.105 2.08  2.059 2.026 2.003 1.987 1.928 1.89  1.836 1.825 1.804 1.731 1.72  1.694 1.685 1.581 1.543 1.518 1.507
 1.428 1.4   1.349 1.31  1.299 1.248 1.223 1.202 1.187 1.17  1.163 1.142 1.109 1.081 1.068 1.036]
W_fc.0 [18.176 11.018  8.737  7.452  7.212  7.062  7.011  6.62   6.57   6.42   6.343  6.29   6.137  6.043  5.974  5.918  5.807  5.779  5.583  5.513  5.5    5.36   5.16   5.12   5.086
  5.032  4.899  4.879  4.819  4.732  4.601  4.533  4.479  4.441  4.268  4.198  4.031  3.974  3.869  3.824  3.72   3.683  3.646  3.574  3.516]
W_proj.0 [9.826 7.351 6.265 5.869 5.797 5.657 5.516 5.45  5.333 5.242 5.197 5.094 4.98  4.904 4.863 4.819 4.793 4.689 4.627 4.595 4.557 4.523 4.411 4.378 4.283 4.216 4.143 4.125 4.048
 4.025 3.894 3.847 3.808 3.792 3.764 3.711 3.685 3.636 3.595 3.525 3.499 3.479 3.469 3.422 3.39 ]
W_proj.1 [8.193 7.894 7.199 6.997 6.671 6.433 6.274 6.133 6.099 6.021 5.942 5.865 5.829 5.789 5.698 5.675 5.646 5.629 5.581 5.564 5.553 5.538 5.528 5.477 5.456 5.447 5.438 5.403 5.399
 5.38  5.367 5.351 5.332 5.327 5.314 5.296 5.252 5.235 5.219 5.208 5.197 5.183 5.174 5.17  5.146]
W_proj.2 [9.404 9.187 8.723 8.47  8.04  7.795 7.614 7.45  7.392 7.253 7.223 7.148 7.082 6.972 6.829 6.733 6.637 6.547 6.46  6.404 6.374 6.307 6.209 6.181 6.12  6.091 5.989 5.9   5.88
 5.833 5.758 5.703 5.685 5.645 5.588 5.559 5.491 5.478 5.467 5.429 5.406 5.4   5.378 5.351 5.335]
W_proj.3 [26.309 20.685 19.699 19.261 18.594 18.197 17.862 17.667 17.044 16.749 16.039 15.151 14.756 14.326 14.05  13.5   13.293 12.531 12.401 11.958 11.74  11.594 10.367  8.351  8.064
  7.914  7.461  7.166  5.833  4.072  3.221  3.139  2.842  2.8    2.687  2.571  2.403  2.328  2.244  2.185  2.181  2.116  2.081  2.053  1.996]

L0H2 left/right-space joint singular values [1.406  1.4046 1.4043 1.4001 1.3966 1.395  1.3932 1.3834 1.3806 1.373  1.3689 1.3616 1.3366 1.2648 1.2248 1.0401 0.9582 0.7071 0.6326 0.4621 0.382  0.3551 0.3389 0.3066 0.2937
 0.2429 0.2324 0.2225 0.1996 0.1674 0.1643 0.1521]
Candidate position-space dimension 16
layer 0 Q/K/both energy: [[0.7   0.064 0.042]
 [0.491 0.202 0.109]
 [0.902 0.897 0.83 ]
 [0.637 0.813 0.554]
 [0.196 0.135 0.026]
 [0.782 0.164 0.13 ]]
layer 1 Q/K/both energy: [[0.054 0.136 0.006]
 [0.062 0.12  0.007]
 [0.074 0.121 0.01 ]
 [0.068 0.129 0.009]
 [0.074 0.137 0.01 ]
 [0.071 0.125 0.009]]
layer 2 Q/K/both energy: [[0.037 0.052 0.002]
 [0.035 0.043 0.001]
 [0.043 0.063 0.003]
 [0.04  0.052 0.003]
 [0.041 0.054 0.002]
 [0.042 0.052 0.002]]
layer 3 Q/K/both energy: [[0.041 0.041 0.002]
 [0.036 0.043 0.003]
 [0.042 0.051 0.002]
 [0.041 0.05  0.003]
 [0.041 0.044 0.002]
 [0.036 0.047 0.001]]
Candidate position-space dimension 24
layer 0 Q/K/both energy: [[0.745 0.159 0.118]
 [0.56  0.326 0.202]
 [0.969 0.97  0.947]
 [0.805 0.837 0.706]
 [0.487 0.334 0.172]
 [0.84  0.473 0.398]]
layer 1 Q/K/both energy: [[0.076 0.229 0.018]
 [0.082 0.205 0.019]
 [0.097 0.214 0.024]
 [0.098 0.22  0.025]
 [0.099 0.242 0.027]
 [0.095 0.216 0.022]]
layer 2 Q/K/both energy: [[0.059 0.082 0.007]
 [0.052 0.065 0.004]
 [0.065 0.106 0.008]
 [0.061 0.082 0.007]
 [0.066 0.084 0.007]
 [0.064 0.081 0.007]]
layer 3 Q/K/both energy: [[0.059 0.06  0.004]
 [0.053 0.062 0.005]
 [0.064 0.072 0.005]
 [0.064 0.073 0.006]
 [0.06  0.062 0.004]
 [0.06  0.07  0.004]]
Candidate position-space dimension 32
layer 0 Q/K/both energy: [[0.821 0.433 0.363]
 [0.618 0.468 0.308]
 [0.985 0.985 0.976]
 [0.851 0.865 0.756]
 [0.598 0.464 0.288]
 [0.895 0.562 0.506]]
layer 1 Q/K/both energy: [[0.102 0.328 0.033]
 [0.121 0.295 0.039]
 [0.127 0.311 0.046]
 [0.127 0.32  0.045]
 [0.125 0.346 0.047]
 [0.125 0.318 0.042]]
layer 2 Q/K/both energy: [[0.083 0.112 0.013]
 [0.069 0.09  0.008]
 [0.09  0.15  0.017]
 [0.084 0.117 0.014]
 [0.087 0.115 0.014]
 [0.087 0.113 0.014]]
layer 3 Q/K/both energy: [[0.078 0.081 0.007]
 [0.069 0.083 0.008]
 [0.085 0.101 0.01 ]
 [0.087 0.101 0.011]
 [0.082 0.087 0.008]
 [0.077 0.089 0.007]]

KEY-COMPOSITION CANDIDATES: QK_l @ OV_earlier
For each head: strongest positive trace/Frobenius scores; entries=(earlier head, trace/Fnorm, transpose symmetry, Fnorm).
L1H0 [('L0H0', np.float64(0.072), np.float64(0.007), np.float64(4.379)), ('L0H4', np.float64(0.062), np.float64(0.004), np.float64(1.866)), ('L0H1', np.float64(0.032), np.float64(0.006), np.float64(3.264)), ('L0H5', np.float64(0.007), np.float64(0.001), np.float64(4.613)), ('L0H2', np.float64(-0.113), np.float64(0.01), np.float64(0.108))]
L1H1 [('L0H4', np.float64(0.102), np.float64(0.007), np.float64(2.573)), ('L0H0', np.float64(0.092), np.float64(0.012), np.float64(5.94)), ('L0H1', np.float64(0.066), np.float64(0.009), np.float64(4.607)), ('L0H5', np.float64(0.033), np.float64(0.002), np.float64(6.86)), ('L0H2', np.float64(-0.003), np.float64(0.036), np.float64(0.213))]
L1H2 [('L0H2', np.float64(0.253), np.float64(0.012), np.float64(0.227)), ('L0H4', np.float64(0.112), np.float64(0.013), np.float64(3.017)), ('L0H0', np.float64(0.101), np.float64(0.017), np.float64(6.982)), ('L0H1', np.float64(0.084), np.float64(0.016), np.float64(5.425)), ('L0H5', np.float64(0.04), np.float64(0.007), np.float64(7.968))]
L1H3 [('L0H2', np.float64(0.233), np.float64(0.014), np.float64(0.22)), ('L0H4', np.float64(0.123), np.float64(0.013), np.float64(2.721)), ('L0H0', np.float64(0.116), np.float64(0.018), np.float64(6.348)), ('L0H1', np.float64(0.094), np.float64(0.013), np.float64(4.85)), ('L0H5', np.float64(0.05), np.float64(0.006), np.float64(7.229))]
L1H4 [('L0H2', np.float64(0.251), np.float64(0.007), np.float64(0.196)), ('L0H4', np.float64(0.163), np.float64(0.02), np.float64(3.015)), ('L0H0', np.float64(0.149), np.float64(0.023), np.float64(7.072)), ('L0H1', np.float64(0.134), np.float64(0.018), np.float64(5.351)), ('L0H5', np.float64(0.081), np.float64(0.006), np.float64(7.691))]
L1H5 [('L0H0', np.float64(0.078), np.float64(0.019), np.float64(6.069)), ('L0H2', np.float64(0.077), np.float64(0.004), np.float64(0.267)), ('L0H4', np.float64(0.076), np.float64(0.014), np.float64(2.605)), ('L0H1', np.float64(0.036), np.float64(0.019), np.float64(4.682)), ('L0H5', np.float64(-0.02), np.float64(0.012), np.float64(6.959))]
L2H0 [('L0H4', np.float64(0.096), np.float64(0.006), np.float64(1.879)), ('L0H0', np.float64(0.08), np.float64(0.007), np.float64(4.368)), ('L0H5', np.float64(0.071), np.float64(0.002), np.float64(5.026)), ('L0H1', np.float64(0.062), np.float64(0.003), np.float64(3.425)), ('L0H2', np.float64(-0.08), np.float64(-0.001), np.float64(0.388))]
L2H1 [('L0H4', np.float64(0.076), np.float64(0.003), np.float64(1.145)), ('L0H5', np.float64(0.071), np.float64(0.0), np.float64(3.117)), ('L0H0', np.float64(0.071), np.float64(0.001), np.float64(2.597)), ('L1H3', np.float64(0.063), np.float64(0.005), np.float64(1.019)), ('L0H1', np.float64(0.046), np.float64(0.002), np.float64(2.064))]
L2H2 [('L1H4', np.float64(0.174), np.float64(0.009), np.float64(0.748)), ('L0H4', np.float64(0.136), np.float64(0.013), np.float64(1.71)), ('L0H0', np.float64(0.11), np.float64(0.013), np.float64(3.936)), ('L1H3', np.float64(0.091), np.float64(0.001), np.float64(0.713)), ('L0H1', np.float64(0.087), np.float64(0.003), np.float64(3.13))]
L2H3 [('L0H4', np.float64(0.116), np.float64(0.008), np.float64(1.681)), ('L0H0', np.float64(0.111), np.float64(0.009), np.float64(3.904)), ('L0H5', np.float64(0.099), np.float64(0.003), np.float64(4.258)), ('L0H1', np.float64(0.08), np.float64(0.004), np.float64(3.015)), ('L0H2', np.float64(0.054), np.float64(-0.001), np.float64(0.286))]
L2H4 [('L0H4', np.float64(0.152), np.float64(0.016), np.float64(1.865)), ('L0H0', np.float64(0.144), np.float64(0.019), np.float64(4.344)), ('L0H1', np.float64(0.108), np.float64(0.006), np.float64(3.348)), ('L0H5', np.float64(0.1), np.float64(-0.001), np.float64(4.602)), ('L1H3', np.float64(-0.059), np.float64(-0.005), np.float64(1.194))]
L2H5 [('L0H0', np.float64(0.112), np.float64(0.008), np.float64(4.078)), ('L0H4', np.float64(0.11), np.float64(0.007), np.float64(1.759)), ('L0H5', np.float64(0.078), np.float64(-0.002), np.float64(4.482)), ('L0H1', np.float64(0.073), np.float64(0.001), np.float64(3.157)), ('L1H2', np.float64(0.007), np.float64(0.005), np.float64(1.03))]
L3H0 [('L0H4', np.float64(0.088), np.float64(0.006), np.float64(0.725)), ('L0H5', np.float64(0.078), np.float64(0.004), np.float64(1.848)), ('L0H1', np.float64(0.06), np.float64(0.003), np.float64(1.256)), ('L0H0', np.float64(0.057), np.float64(0.001), np.float64(1.653)), ('L0H2', np.float64(-0.042), np.float64(0.001), np.float64(0.309))]
L3H1 [('L2H1', np.float64(1.25), np.float64(0.041), np.float64(2.105)), ('L0H2', np.float64(1.173), np.float64(0.097), np.float64(0.376)), ('L1H2', np.float64(0.879), np.float64(0.033), np.float64(1.061)), ('L1H3', np.float64(0.768), np.float64(0.049), np.float64(1.36)), ('L1H1', np.float64(0.63), np.float64(0.012), np.float64(1.087))]
L3H2 [('L0H5', np.float64(0.101), np.float64(0.008), np.float64(3.049)), ('L0H0', np.float64(0.087), np.float64(0.006), np.float64(2.532)), ('L0H4', np.float64(0.082), np.float64(0.003), np.float64(1.052)), ('L0H2', np.float64(0.063), np.float64(0.0), np.float64(0.18)), ('L0H1', np.float64(0.059), np.float64(0.002), np.float64(1.86))]
L3H3 [('L0H5', np.float64(0.07), np.float64(0.002), np.float64(3.087)), ('L0H0', np.float64(0.06), np.float64(0.003), np.float64(2.578)), ('L0H4', np.float64(0.05), np.float64(0.002), np.float64(1.107)), ('L0H1', np.float64(0.03), np.float64(-0.001), np.float64(1.951)), ('L0H2', np.float64(0.024), np.float64(-0.004), np.float64(0.195))]
L3H4 [('L0H5', np.float64(0.164), np.float64(0.024), np.float64(2.969)), ('L0H4', np.float64(0.16), np.float64(0.019), np.float64(1.065)), ('L0H1', np.float64(0.142), np.float64(0.016), np.float64(1.879)), ('L0H0', np.float64(0.134), np.float64(0.015), np.float64(2.471)), ('L2H0', np.float64(0.015), np.float64(0.002), np.float64(1.46))]
L3H5 [('L2H3', np.float64(0.318), np.float64(0.009), np.float64(0.773)), ('L2H4', np.float64(0.228), np.float64(0.007), np.float64(0.678)), ('L2H1', np.float64(0.217), np.float64(0.005), np.float64(0.786)), ('L2H5', np.float64(0.198), np.float64(0.005), np.float64(0.689)), ('L1H0', np.float64(0.186), np.float64(0.006), np.float64(0.506))]

Pairwise cosine similarities of effective QK products, within each layer
Layer 0 
 [[ 1.     0.209 -0.016 -0.014  0.115  0.018]
 [ 0.209  1.    -0.004 -0.036  0.21  -0.008]
 [-0.016 -0.004  1.     0.24  -0.006 -0.026]
 [-0.014 -0.036  0.24   1.    -0.054 -0.058]
 [ 0.115  0.21  -0.006 -0.054  1.     0.172]
 [ 0.018 -0.008 -0.026 -0.058  0.172  1.   ]]
Layer 1 
 [[1.    0.578 0.416 0.448 0.455 0.464]
 [0.578 1.    0.55  0.488 0.556 0.528]
 [0.416 0.55  1.    0.608 0.667 0.564]
 [0.448 0.488 0.608 1.    0.644 0.563]
 [0.455 0.556 0.667 0.644 1.    0.577]
 [0.464 0.528 0.564 0.563 0.577 1.   ]]
Layer 2 
 [[1.    0.078 0.133 0.103 0.112 0.106]
 [0.078 1.    0.093 0.086 0.077 0.07 ]
 [0.133 0.093 1.    0.118 0.126 0.125]
 [0.103 0.086 0.118 1.    0.12  0.113]
 [0.112 0.077 0.126 0.12  1.    0.107]
 [0.106 0.07  0.125 0.113 0.107 1.   ]]
Layer 3 
 [[1.    0.071 0.044 0.036 0.075 0.03 ]
 [0.071 1.    0.021 0.024 0.03  0.045]
 [0.044 0.021 1.    0.094 0.103 0.054]
 [0.036 0.024 0.094 1.    0.084 0.065]
 [0.075 0.03  0.103 0.084 1.    0.049]
 [0.03  0.045 0.054 0.065 0.049 1.   ]]
