OUTPUT of run 3 of at most 100: ``` INDUCTION ALGEBRA: trace(A_l C_earlier)/(||A_l||F ||C_earlier||F) Positive values indicate a possible query-to-preceding-token key match, not measured attention. columns ['L0H0', 'L0H1', 'L0H2', 'L0H3', 'L0H4', 'L0H5'] L1H0 [ 0.0065 -0.0001 -0.0079 0.005 0.0207 -0.0034] L1H1 [-0.0046 0.007 -0.0161 0.0055 0.0191 0.0154] L1H2 [ 0.0039 0.0007 -0.0168 0.0092 0.0326 -0.0027] L1H3 [ 0.0077 -0.0026 -0.0132 0.0063 0.0421 0.0124] L1H4 [ 0.0093 0.0022 -0.0181 0.0114 0.0439 0.0094] L1H5 [-0.0195 0.0045 -0.0134 -0.0004 -0.0036 0.0128] columns ['L0H0', 'L0H1', 'L0H2', 'L0H3', 'L0H4', 'L0H5', 'L1H0', 'L1H1', 'L1H2', 'L1H3', 'L1H4', 'L1H5'] L2H0 [ 0.0049 -0.0006 -0.003 0.0078 0.0099 -0.0046 -0.0126 0.0039 -0.0003 0.0016 -0.0004 0.0057] L2H1 [0.0065 0.0014 0.0108 0.0025 0.0038 0.0776 0.0025 0.0479 0.0482 0.0575 0.0476 0.0351] L2H2 [ 0.0114 -0.0165 -0.0054 0.0015 0.02 -0.0387 -0.0418 -0.0162 -0.0434 -0.007 -0.023 -0.0069] L2H3 [ 0.0177 -0.0008 -0.0065 0.0066 0.0396 -0.0026 -0.0197 0.0049 -0.0162 0.0042 -0.0048 0.0044] L2H4 [ 0.0079 -0.0055 -0.004 0.0101 0.0246 0.0018 -0.0139 -0.0082 -0.0115 -0.0019 -0.0042 -0.006 ] L2H5 [0.009 0.0006 0.0001 0.0046 0.0188 0.0068 0.0001 0.0083 0.0095 0.0064 0.0068 0.0068] columns ['L0H0', 'L0H1', 'L0H2', 'L0H3', 'L0H4', 'L0H5', 'L1H0', 'L1H1', 'L1H2', 'L1H3', 'L1H4', 'L1H5', 'L2H0', 'L2H1', 'L2H2', 'L2H3', 'L2H4', 'L2H5'] L3H0 [ 0.0098 -0.0036 0.0068 0.0077 0.0098 0.019 0.0079 0.0083 0.0101 0.006 0.006 0.0009 -0.0033 -0.0135 0.0238 -0.0008 0.0044 -0.0088] L3H1 [ 0.0078 0.0071 0.006 0.0014 0.0041 0.061 0.0127 0.0333 0.038 0.0453 0.0456 0.0271 0.0153 -0.0181 0.1104 0.0209 0.0053 -0.0002] L3H2 [ 0.0062 -0.0032 0.0015 -0.0008 0.013 -0.0012 -0.0116 0.0013 -0.0092 0. -0.0089 -0.0036 -0.0046 -0.007 -0.0004 -0.0033 -0.0013 -0.0031] L3H3 [ 0.0045 0.0032 0.0048 0.0026 0.0073 0.001 0.005 0.0017 0.0079 -0.0004 0.0022 0.0042 0.0063 -0.0054 0.009 0.0103 0.0148 0.0045] L3H4 [ 0.0085 -0.0063 0.0056 0.0029 0.0119 0.0101 0.0005 0.0069 0.0032 0.0047 0.0048 0.0083 0.0011 -0.0083 0.0081 0.0041 0.0103 -0.0038] L3H5 [ 0.006 -0.0089 0.0004 0.0027 0.0025 0.0713 0.0041 0.0423 0.0482 0.0683 0.0572 0.0198 -0.0084 -0.0599 0.1836 0.055 -0.0216 -0.022 ] L0 effective Q,K,V singular values (first 32): L0H0 Q [1.242 1.067 0.419 0.304 0.261 0.221 0.213 0.198 0.187 0.182 0.177 0.175 0.163 0.157 0.152 0.144 0.14 0.138 0.134 0.13 0.126 0.122 0.117 0.116 0.113 0.11 0.107 0.103 0.101 0.1 0.095 0.094] L0H0 K [0.661 0.456 0.27 0.246 0.23 0.223 0.212 0.21 0.198 0.196 0.19 0.188 0.182 0.179 0.173 0.171 0.164 0.158 0.153 0.153 0.147 0.144 0.142 0.141 0.137 0.133 0.128 0.127 0.125 0.124 0.122 0.12 ] L0H0 V [1.331 1.075 0.446 0.374 0.304 0.294 0.256 0.247 0.237 0.224 0.218 0.205 0.203 0.199 0.188 0.181 0.18 0.174 0.169 0.162 0.162 0.155 0.151 0.147 0.146 0.138 0.136 0.132 0.13 0.126 0.125 0.122] L0H1 Q [1.513 1.324 1.288 1.273 1.188 1.055 1.027 0.789 0.773 0.715 0.697 0.667 0.654 0.644 0.629 0.593 0.569 0.555 0.538 0.522 0.52 0.494 0.462 0.44 0.41 0.394 0.386 0.367 0.35 0.341 0.32 0.309] L0H1 K [3.78 2.409 1.606 1.4 1.309 1.203 1.148 1.007 0.879 0.855 0.831 0.809 0.747 0.722 0.71 0.655 0.639 0.614 0.58 0.569 0.534 0.504 0.484 0.474 0.447 0.427 0.381 0.353 0.344 0.324 0.32 0.31 ] L0H1 V [0.48 0.455 0.448 0.429 0.421 0.41 0.403 0.385 0.379 0.373 0.37 0.36 0.353 0.341 0.33 0.327 0.322 0.313 0.309 0.303 0.295 0.287 0.281 0.278 0.27 0.255 0.253 0.241 0.226 0.216 0.209 0.201] L0H2 Q [1.143 0.768 0.644 0.498 0.488 0.438 0.434 0.399 0.358 0.33 0.31 0.298 0.279 0.261 0.252 0.249 0.243 0.231 0.217 0.21 0.208 0.202 0.198 0.195 0.185 0.179 0.174 0.17 0.166 0.161 0.159 0.147] L0H2 K [1.187 0.914 0.55 0.529 0.506 0.489 0.424 0.397 0.389 0.37 0.35 0.333 0.321 0.307 0.298 0.292 0.28 0.268 0.258 0.249 0.235 0.228 0.22 0.214 0.212 0.2 0.192 0.19 0.187 0.177 0.173 0.166] L0H2 V [0.5 0.482 0.462 0.399 0.361 0.337 0.296 0.283 0.268 0.261 0.245 0.223 0.217 0.21 0.199 0.19 0.182 0.179 0.177 0.17 0.167 0.158 0.156 0.153 0.15 0.147 0.145 0.137 0.135 0.129 0.127 0.124] L0H3 Q [1.547 1.125 0.98 0.628 0.571 0.521 0.409 0.372 0.364 0.343 0.299 0.29 0.282 0.279 0.275 0.255 0.243 0.226 0.212 0.198 0.198 0.194 0.184 0.171 0.162 0.154 0.15 0.148 0.144 0.143 0.141 0.139] L0H3 K [1.093 0.949 0.68 0.542 0.435 0.406 0.399 0.365 0.34 0.331 0.319 0.299 0.278 0.26 0.249 0.239 0.227 0.223 0.22 0.203 0.191 0.184 0.18 0.175 0.173 0.166 0.161 0.156 0.156 0.154 0.147 0.144] L0H3 V [0.588 0.386 0.342 0.308 0.294 0.277 0.273 0.246 0.242 0.24 0.23 0.216 0.209 0.206 0.195 0.192 0.187 0.178 0.175 0.172 0.168 0.158 0.153 0.151 0.149 0.143 0.143 0.141 0.138 0.13 0.128 0.125] L0H4 Q [0.205 0.143 0.139 0.129 0.126 0.121 0.119 0.114 0.112 0.111 0.103 0.101 0.1 0.098 0.097 0.094 0.091 0.088 0.088 0.085 0.084 0.082 0.079 0.077 0.077 0.075 0.073 0.071 0.068 0.067 0.067 0.066] L0H4 K [0.214 0.193 0.186 0.178 0.173 0.167 0.156 0.154 0.148 0.146 0.142 0.137 0.135 0.129 0.128 0.123 0.12 0.118 0.116 0.113 0.111 0.107 0.106 0.103 0.099 0.099 0.097 0.093 0.092 0.09 0.088 0.086] L0H4 V [2.898 2.029 1.104 0.576 0.48 0.427 0.379 0.362 0.315 0.31 0.3 0.274 0.264 0.262 0.245 0.235 0.218 0.207 0.202 0.194 0.187 0.183 0.177 0.171 0.167 0.164 0.161 0.158 0.15 0.144 0.143 0.142] L0H5 Q [1.396 1.368 1.336 1.315 1.25 1.232 1.208 1.19 1.145 1.132 1.095 1.079 1.066 1.063 1.015 1.011 0.984 0.951 0.912 0.855 0.797 0.777 0.704 0.622 0.608 0.588 0.525 0.46 0.445 0.429 0.406 0.397] L0H5 K [3.314 2.634 1.802 1.724 1.638 1.528 1.499 1.455 1.396 1.35 1.273 1.21 1.2 1.167 1.063 1.045 1.005 0.935 0.905 0.887 0.827 0.736 0.643 0.62 0.596 0.561 0.501 0.439 0.431 0.4 0.375 0.358] L0H5 V [0.493 0.473 0.446 0.437 0.426 0.408 0.404 0.393 0.386 0.376 0.363 0.361 0.356 0.345 0.333 0.329 0.322 0.316 0.31 0.304 0.299 0.289 0.285 0.277 0.274 0.267 0.252 0.245 0.241 0.235 0.233 0.221] Average squared overlap of leading 24-dimensional L0 subspaces: ['Q1', 'K1', 'Q5', 'K5', 'V1', 'V5'] [[1. 0.607 0.77 0.655 0.103 0.146] [0.607 1. 0.662 0.776 0.122 0.178] [0.77 0.662 1. 0.814 0.081 0.137] [0.655 0.776 0.814 1. 0.08 0.153] [0.103 0.122 0.081 0.08 1. 0.619] [0.146 0.178 0.137 0.153 0.619 1. ]] Layer 0 QK product correlations, centered effective matrices: power 1 cosine with H1 0.4137 cosine with H1 transpose -0.0104 power 2 cosine with H1 0.5461 cosine with H1 transpose -0.4342 power 3 cosine with H1 -0.0482 cosine with H1 transpose -0.4405 power 4 cosine with H1 -0.6081 cosine with H1 transpose 0.0147 power 5 cosine with H1 -0.6339 cosine with H1 transpose 0.4624 power 6 cosine with H1 -0.1532 cosine with H1 transpose 0.5474 Top-key and top-query directions of each head: overlap with L0H4 output subspace (top 4). L0H0 Q [0.343 0.245 0.035 0.212] K [0.162 0.062 0.076 0.03 ] L0H1 Q [0.008 0.139 0.036 0.011] K [0.101 0.029 0.016 0.018] L0H2 Q [0.183 0.341 0.034 0.185] K [0.023 0.014 0.007 0.08 ] L0H3 Q [0.311 0.485 0.183 0.19 ] K [0.129 0.02 0.023 0.069] L0H4 Q [0.462 0.17 0.121 0.039] K [0.112 0.081 0.038 0.008] L0H5 Q [0.033 0.004 0.011 0.017] K [0.082 0.067 0.031 0.025] L1H0 Q [0.049 0.073 0.061 0.071] K [0.809 0.62 0.119 0.016] L1H1 Q [0.162 0.056 0.016 0.024] K [0.913 0.78 0.048 0.041] L1H2 Q [0.206 0.06 0.068 0.006] K [0.885 0.802 0.074 0.016] L1H3 Q [0.236 0.1 0.067 0.028] K [0.9 0.797 0.058 0.016] L1H4 Q [0.154 0.08 0.049 0.103] K [0.887 0.818 0.065 0.01 ] L1H5 Q [0.069 0.076 0.122 0.017] K [0.852 0.764 0.077 0.054] L2H0 Q [0.01 0.007 0.002 0.033] K [0.563 0.12 0.01 0.011] L2H1 Q [0.042 0.006 0.007 0.004] K [0.032 0.012 0.005 0.013] L2H2 Q [0.002 0.003 0.006 0.01 ] K [0.88 0.361 0.009 0.243] L2H3 Q [0.001 0.005 0.029 0.082] K [0.823 0.649 0.032 0.006] L2H4 Q [0.003 0.042 0.015 0.009] K [0.762 0.041 0.532 0.099] L2H5 Q [0.017 0.002 0.016 0.021] K [0.413 0.009 0.179 0.018] L3H0 Q [0.014 0.003 0.002 0.001] K [0.192 0.023 0.019 0.027] L3H1 Q [0.017 0.015 0.002 0.021] K [0.02 0.002 0.008 0.003] L3H2 Q [0.013 0.003 0.003 0.019] K [0.488 0.095 0.004 0.001] L3H3 Q [0.016 0.002 0.003 0. ] K [0.065 0.01 0.032 0.029] L3H4 Q [0.014 0.009 0.004 0.002] K [0.336 0.04 0.002 0.003] L3H5 Q [0.003 0.002 0.019 0.002] K [0.014 0.014 0.015 0.002] ```