OUTPUT of run 3 of at most 100: ``` Layer 0 scaled QK: symmetric-energy fraction, positive/negative eigenvalue sums of symmetric part H 0 [ 0.5704 0.3783 -0.483 ] eigenvalues: [ 0.0796+0.j 0.0483+0.j -0.024 +0.0057j -0.024 -0.0057j -0.0226+0.j 0.0176+0.0134j 0.0176-0.0134j -0.0201+0.j -0.0138-0.0121j -0.0138+0.0121j -0.0139-0.0031j -0.0139+0.0031j -0.0137+0.j -0.011 -0.0039j -0.011 +0.0039j -0.0111+0.0023j -0.0111-0.0023j -0.006 -0.0062j -0.006 +0.0062j -0.0063+0.j -0.003 -0.0049j -0.003 +0.0049j -0.0052-0.0023j -0.0052+0.0023j] H 1 [ 0.5805 0.1152 -0.2136] eigenvalues: [-0.0619+0.j -0.01 +0.j -0.0089+0.j 0.0043+0.j -0.0039+0.j -0.0033-0.0018j -0.0033+0.0018j -0.0017+0.0004j -0.0017-0.0004j -0.0016-0.0007j -0.0016+0.0007j -0.0011+0.0003j -0.0011-0.0003j -0.0007+0.0008j -0.0007-0.0008j 0.001 +0.j -0.0009+0.j 0.0004+0.0007j 0.0004-0.0007j -0.0007-0.0005j -0.0007+0.0005j -0.0005-0.0007j -0.0005+0.0007j -0.0002-0.0007j] H 2 [ 0.6443 0.0809 -0.1682] eigenvalues: [-0.0566+0.j -0.0074-0.0027j -0.0074+0.0027j -0.0034-0.0008j -0.0034+0.0008j -0.0025+0.j -0.0013-0.0003j -0.0013+0.0003j -0.0006-0.0012j -0.0006+0.0012j 0.0006+0.001j 0.0006-0.001j -0.0003-0.001j -0.0003+0.001j -0.001 +0.j 0.0001+0.001j 0.0001-0.001j -0.0005-0.0008j -0.0005+0.0008j -0.0007+0.0006j -0.0007-0.0006j 0.0008+0.j -0.0006-0.0004j -0.0006+0.0004j] H 3 [ 0.4664 1.9585 -0.617 ] eigenvalues: [ 0.1366+0.1327j 0.1366-0.1327j 0.1099+0.0891j 0.1099-0.0891j 0.095 +0.0979j 0.095 -0.0979j 0.0841+0.1041j 0.0841-0.1041j 0.0644+0.1142j 0.0644-0.1142j 0.0957-0.0687j 0.0957+0.0687j 0.0114+0.1166j 0.0114-0.1166j 0.1116+0.j 0.0559+0.0696j 0.0559-0.0696j 0.0749+0.0468j 0.0749-0.0468j 0.0135-0.0747j 0.0135+0.0747j -0.0348-0.0526j -0.0348+0.0526j -0.0451+0.0173j] H 4 [ 0.5251 0.103 -0.1489] eigenvalues: [-0.0152+0.j -0.0095+0.j -0.0077+0.j -0.0026-0.0003j -0.0026+0.0003j -0.0022-0.0014j -0.0022+0.0014j -0.0002+0.0023j -0.0002-0.0023j -0.0014-0.0012j -0.0014+0.0012j -0.0015+0.j -0.0006-0.0011j -0.0006+0.0011j 0.0008-0.0009j 0.0008+0.0009j 0.0012+0.j -0.0003-0.0011j -0.0003+0.0011j 0.0005-0.0009j 0.0005+0.0009j -0.001 +0.j 0.001 +0.j -0.0009-0.0001j] H 5 [ 0.5122 0.0451 -0.0536] eigenvalues: [-0.0049+0.j 0.0013+0.j -0.0012+0.j -0.0004-0.0007j -0.0004+0.0007j -0.0008+0.j -0.0007+0.0002j -0.0007-0.0002j 0.0002-0.0007j 0.0002+0.0007j -0.0001-0.0006j -0.0001+0.0006j -0.0003-0.0005j -0.0003+0.0005j 0.0002-0.0006j 0.0002+0.0006j -0.0005-0.0003j -0.0005+0.0003j 0.0006+0.j -0.0005+0.0001j -0.0005-0.0001j 0.0005+0.0001j 0.0005-0.0001j -0.0004+0.0004j] Fraction of each OV's squared norm captured by QK H3 right/left subspaces rank 4 key [0.007 0.441 0.562 0.012 0.362 0.826] query [0.025 0.484 0.585 0.011 0.354 0.697] rank 8 key [0.014 0.462 0.584 0.027 0.374 0.853] query [0.033 0.503 0.612 0.026 0.382 0.732] rank 16 key [0.025 0.48 0.608 0.06 0.413 0.882] query [0.043 0.528 0.646 0.054 0.422 0.767] rank 24 key [0.037 0.503 0.625 0.085 0.445 0.898] query [0.058 0.577 0.679 0.074 0.487 0.782] rank 32 key [0.071 0.525 0.64 0.112 0.475 0.907] query [0.095 0.679 0.727 0.089 0.645 0.811] rank 48 key [0.263 0.605 0.707 0.29 0.551 0.924] query [0.288 0.831 0.844 0.25 0.787 0.887] Own-head QK/OV subspace overlaps (r=1,4,8,16,24): H 0 [[0.003 0. ] [0.081 0.117] [0.073 0.093] [0.092 0.113] [0.15 0.168]] H 1 [[0.175 0.078] [0.267 0.106] [0.188 0.106] [0.153 0.142] [0.162 0.162]] H 2 [[0.011 0.071] [0.263 0.112] [0.206 0.118] [0.159 0.135] [0.175 0.166]] H 3 [[0. 0.002] [0.005 0.004] [0.01 0.011] [0.02 0.024] [0.116 0.133]] H 4 [[0.437 0.049] [0.191 0.058] [0.144 0.073] [0.136 0.121] [0.153 0.166]] H 5 [[0.356 0. ] [0.248 0.057] [0.171 0.093] [0.142 0.106] [0.152 0.132]] Layer 0 cross-head subspace overlap: top 8 query and key directions query [[1. 0.217 0.215 0.332 0.189 0.157] [0.217 1. 0.488 0.12 0.419 0.123] [0.215 0.488 1. 0.143 0.35 0.126] [0.332 0.12 0.143 1. 0.111 0.101] [0.189 0.419 0.35 0.111 1. 0.113] [0.157 0.123 0.126 0.101 0.113 1. ]] key [[1. 0.063 0.1 0.396 0.057 0.055] [0.063 1. 0.314 0.045 0.284 0.149] [0.1 0.314 1. 0.086 0.262 0.1 ] [0.396 0.045 0.086 1. 0.066 0.049] [0.057 0.284 0.262 0.066 1. 0.11 ] [0.055 0.149 0.1 0.049 0.11 1. ]] QK H3 singular-vector pair cosines: [ 0.659 0.672 0.628 0.554 0.591 0.676 0.613 0.626 0.534 0.529 0.284 0.322 0.609 0.418 0.485 0.466 0.475 0.561 -0.393 0.188 0.043 -0.121 -0.028 0.082 -0.117 -0.37 -0.159 -0.266 0.092 -0.442 -0.19 -0.313] SV [0.2435 0.2224 0.2012 0.1816 0.1785 0.1571 0.1545 0.149 0.1401 0.1358 0.1306 0.1277 0.1195 0.1183 0.1151 0.1077 0.1034 0.0998 0.0909 0.0864 0.0823 0.0767 0.0721 0.0691 0.0591 0.0552 0.0494 0.0464 0.0384 0.0308 0.0295 0.0268 0.0228 0.0216 0.0189 0.017 0.0163 0.0139 0.0131 0.0118] L0 LN scale largest and smallest coordinates: [(180, 0.9785), (184, 0.974), (162, 0.9704), (99, 0.9171), (211, 0.896), (256, 0.892), (300, 0.8045), (317, 0.804), (223, 0.8017), (39, 0.7989), (173, 0.7933), (4, 0.7864), (228, 0.7859), (69, 0.7721), (89, 0.7672), (100, 0.7658), (56, 0.753), (65, 0.7491), (354, 0.7465), (246, 0.7261), (124, 0.7207), (349, 0.7205), (60, 0.7127), (132, 0.7083)] [(381, 0.0352), (42, 0.0606), (95, 0.0607), (17, 0.0622), (147, 0.0673), (237, 0.0678), (376, 0.0726), (334, 0.0736), (78, 0.075), (299, 0.0755), (233, 0.0756), (335, 0.0765), (81, 0.0775), (176, 0.0828), (340, 0.0844), (306, 0.0845), (230, 0.0847), (131, 0.0858), (263, 0.0886), (305, 0.0925), (156, 0.0951), (313, 0.0967), (282, 0.0972), (85, 0.0995)] ```