OUTPUT of run 1 of at most 100: ``` WEIGHTS W_fc.0 (1536, 384) mean -0.0011808212148025632 std 0.06020796671509743 norm 46.248592376708984 min/max -0.7368497252464294 0.8575737476348877 W_fc.1 (1536, 384) mean -0.00040238190558739007 std 0.1030285656452179 norm 79.12652587890625 min/max -0.7885797619819641 0.7180054187774658 W_fc.2 (1536, 384) mean -0.00033663868089206517 std 0.10282079875469208 norm 78.96678924560547 min/max -0.8970372676849365 1.2283228635787964 W_fc.3 (1536, 384) mean -5.819171201437712e-05 std 0.10127057135105133 norm 77.77579498291016 min/max -0.4879089891910553 1.0584442615509033 W_k.0 (384, 384) mean -0.00011999025446129963 std 0.07914295792579651 norm 30.39093017578125 min/max -0.8677226901054382 0.9180701971054077 W_k.1 (384, 384) mean 6.702502105326857e-06 std 0.07617354393005371 norm 29.250638961791992 min/max -1.018454909324646 0.9855566620826721 W_k.2 (384, 384) mean 3.240807927795686e-05 std 0.09835635125637054 norm 37.768836975097656 min/max -1.4294401407241821 1.307726263999939 W_k.3 (384, 384) mean -0.00010883563663810492 std 0.08387870341539383 norm 32.20944595336914 min/max -0.9338246583938599 0.8848811984062195 W_o.0 (384, 384) mean -0.00011180933506693691 std 0.04263352230191231 norm 16.371326446533203 min/max -1.0171805620193481 0.9719095826148987 W_o.1 (384, 384) mean -6.107017725298647e-06 std 0.0675085186958313 norm 25.92327117919922 min/max -0.6565335392951965 0.7066713571548462 W_o.2 (384, 384) mean 1.4057885891816113e-05 std 0.08552003651857376 norm 32.83969497680664 min/max -0.5314646363258362 0.6010491847991943 W_o.3 (384, 384) mean 7.072658627294004e-05 std 0.10535143315792084 norm 40.45496368408203 min/max -1.2572932243347168 1.1984047889709473 W_proj.0 (384, 1536) mean -5.542821963899769e-05 std 0.05118163675069809 norm 39.30747985839844 min/max -0.7572827935218811 1.1359899044036865 W_proj.1 (384, 1536) mean 7.086182904458838e-06 std 0.09858116507530212 norm 75.71031188964844 min/max -0.8062023520469666 2.0403969287872314 W_proj.2 (384, 1536) mean -0.00013844718341715634 std 0.10004039108753204 norm 76.83108520507812 min/max -0.4829542338848114 1.7099428176879883 W_proj.3 (384, 1536) mean 0.00017326009401585907 std 0.10728523135185242 norm 82.3951416015625 min/max -2.0115103721618652 1.5111761093139648 W_q.0 (384, 384) mean 0.0002594335237517953 std 0.08409115672111511 norm 32.29115676879883 min/max -0.9620245099067688 1.0088342428207397 W_q.1 (384, 384) mean -2.918012069130782e-05 std 0.08414740860462189 norm 32.31260299682617 min/max -0.44159331917762756 0.3799610137939453 W_q.2 (384, 384) mean 1.9767563571804203e-05 std 0.10246235132217407 norm 39.345542907714844 min/max -0.44507333636283875 0.46751806139945984 W_q.3 (384, 384) mean -4.0679340600036085e-05 std 0.08580272644758224 norm 32.94824981689453 min/max -0.4029965400695801 0.4370769262313843 W_v.0 (384, 384) mean 5.660305032506585e-05 std 0.03863541781902313 norm 14.836016654968262 min/max -0.34857437014579773 0.36271223425865173 W_v.1 (384, 384) mean -3.110744000878185e-05 std 0.07123148441314697 norm 27.352893829345703 min/max -0.4262692332267761 0.35993435978889465 W_v.2 (384, 384) mean 0.00011245020141359419 std 0.08673116564750671 norm 33.3047981262207 min/max -0.3964833915233612 0.40176886320114136 W_v.3 (384, 384) mean 0.0001254113594768569 std 0.10363049805164337 norm 39.79413986206055 min/max -0.48620620369911194 0.5527780652046204 ln1_b.0 (384,) mean 0.0008393097086809576 std 0.028742684051394463 norm 0.5634793639183044 min/max -0.09590836614370346 0.09102362394332886 ln1_b.1 (384,) mean -0.005021821707487106 std 0.16716055572032928 norm 3.277142286300659 min/max -1.4712495803833008 1.1544171571731567 ln1_b.2 (384,) mean -0.0010484304511919618 std 0.15254196524620056 norm 2.9892706871032715 min/max -1.381901502609253 1.05267333984375 ln1_b.3 (384,) mean 0.012112464755773544 std 0.0848778784275055 norm 1.6801103353500366 min/max -0.6069080829620361 0.6501507759094238 ln1_w.0 (384,) mean 0.24030669033527374 std 0.24358396232128143 norm 6.70513916015625 min/max -0.0010124740656465292 1.0340553522109985 ln1_w.1 (384,) mean 0.45753344893455505 std 0.1022113487124443 norm 9.186786651611328 min/max 0.13610735535621643 1.1238988637924194 ln1_w.2 (384,) mean 0.7052056193351746 std 0.07490397244691849 norm 13.896883964538574 min/max 0.20780912041664124 1.0144426822662354 ln1_w.3 (384,) mean 0.7545757293701172 std 0.05423606559634209 norm 14.824750900268555 min/max 0.29438865184783936 0.8701038360595703 ln2_b.0 (384,) mean 0.0021705578546971083 std 0.06830766052007675 norm 1.3392269611358643 min/max -0.49860355257987976 0.3330683708190918 ln2_b.1 (384,) mean 0.0005380580550990999 std 0.13641700148582458 norm 2.6732370853424072 min/max -0.3596881926059723 0.36863091588020325 ln2_b.2 (384,) mean 0.004968605935573578 std 0.1536681205034256 norm 3.0128414630889893 min/max -0.37611520290374756 0.3885933756828308 ln2_b.3 (384,) mean 0.004284036811441183 std 0.12758871912956238 norm 2.501626968383789 min/max -0.6727620363235474 0.30676206946372986 ln2_w.0 (384,) mean 0.8005699515342712 std 0.3295142650604248 norm 16.964813232421875 min/max 0.004430394619703293 1.5072498321533203 ln2_w.1 (384,) mean 1.0428675413131714 std 0.200262650847435 norm 20.809329986572266 min/max -0.0019355153199285269 1.4855960607528687 ln2_w.2 (384,) mean 1.3122882843017578 std 0.12551987171173096 norm 25.832860946655273 min/max 0.0011308102402836084 1.4654561281204224 ln2_w.3 (384,) mean 1.2777737379074097 std 0.08074603974819183 norm 25.089094161987305 min/max 0.43639853596687317 1.430633544921875 HEAD SPECTRA: QK = Q.T @ K / 8; OV = O @ V L0H0 Q/K/V/O norms [3.305 4.18 6.039 6.666] QK top [0.314 0.112 0.087 0.055 0.041 0.039 0.035 0.033] QK stable rank 1.47 QK symmetric fraction 0.727 OV top [10.252 5.603 3.475 2.818 2.112 1.335] OV trace/nuclear -0.005 L0H1 Q/K/V/O norms [3.09 3.658 5.382 6.025] QK top [0.137 0.096 0.056 0.043 0.037 0.036 0.033 0.032] QK stable rank 2.81 QK symmetric fraction 0.747 OV top [7.662 5.172 2.582 2.031 1.761 1.087] OV trace/nuclear -0.026 L0H2 Q/K/V/O norms [30.818 28.3 7.765 8.915] QK top [7.485 7.16 6.948 6.793 6.404 6.182 5.898 5.585] QK stable rank 9.28 QK symmetric fraction 0.696 OV top [4.001 3.853 3.743 3.544 3.449 3.313] OV trace/nuclear -0.069 L0H3 Q/K/V/O norms [6.821 7.243 5.434 5.029] QK top [0.821 0.749 0.402 0.331 0.303 0.267 0.234 0.217] QK stable rank 2.99 QK symmetric fraction 0.763 OV top [4.305 4.046 2.493 2.078 1.8 1.083] OV trace/nuclear -0.082 L0H4 Q/K/V/O norms [3.971 4.995 5.208 5.587] QK top [0.753 0.255 0.135 0.07 0.054 0.043 0.04 0.038] QK stable rank 1.2 QK symmetric fraction 0.763 OV top [4.879 3.236 2.548 2.083 1.635 1.358] OV trace/nuclear -0.038 L0H5 Q/K/V/O norms [3.19 3.802 6.143 7.162] QK top [0.274 0.068 0.049 0.041 0.037 0.037 0.034 0.032] QK stable rank 1.39 QK symmetric fraction 0.729 OV top [10.314 5.468 3.148 2.935 2.314 1.682] OV trace/nuclear -0.002 L1H0 Q/K/V/O norms [10.716 9.827 11.156 9.984] QK top [1.577 1.462 1.035 0.877 0.653 0.522 0.481 0.451] QK stable rank 3.52 QK symmetric fraction 0.735 OV top [8.099 7.58 6.398 6.202 5.966 5.738] OV trace/nuclear -0.203 L1H1 Q/K/V/O norms [13.292 12.068 11.72 11.303] QK top [2.456 2.132 1.344 1.193 0.918 0.869 0.803 0.733] QK stable rank 3.27 QK symmetric fraction 0.742 OV top [6.583 6.009 5.739 5.681 5.394 5.09 ] OV trace/nuclear -0.223 L1H2 Q/K/V/O norms [14.476 13.092 10.668 10.272] QK top [2.923 2.292 1.508 1.402 1.191 1.074 1.041 0.966] QK stable rank 3.19 QK symmetric fraction 0.745 OV top [6.135 5.788 5.252 4.987 4.903 4.7 ] OV trace/nuclear -0.207 L1H3 Q/K/V/O norms [13.358 12.153 10.904 9.928] QK top [2.638 2.173 1.454 1.142 1.062 0.813 0.779 0.759] QK stable rank 3.05 QK symmetric fraction 0.74 OV top [7.767 6.864 6.527 6.06 5.814 5.543] OV trace/nuclear -0.195 L1H4 Q/K/V/O norms [13.518 12.129 11.148 10.794] QK top [2.87 2.178 1.43 1.085 0.911 0.776 0.754 0.713] QK stable rank 2.65 QK symmetric fraction 0.737 OV top [6.498 6.253 5.897 5.784 5.494 5.388] OV trace/nuclear -0.228 L1H5 Q/K/V/O norms [13.486 12.132 11.376 11.136] QK top [2.618 2.293 1.294 1.053 0.925 0.835 0.779 0.717] QK stable rank 2.99 QK symmetric fraction 0.73 OV top [6.208 6.019 5.803 5.645 5.415 5.359] OV trace/nuclear -0.238 L2H0 Q/K/V/O norms [16.822 16.208 13.115 12.942] QK top [1.457 1.249 1.141 1.088 1.043 0.997 0.945 0.921] QK stable rank 10.95 QK symmetric fraction 0.702 OV top [4.898 4.628 4.553 4.4 4.363 4.218] OV trace/nuclear -0.324 L2H1 Q/K/V/O norms [15.963 15.287 13.768 13.525] QK top [1.155 0.893 0.881 0.867 0.816 0.794 0.778 0.749] QK stable rank 12.85 QK symmetric fraction 0.713 OV top [5.468 5.157 4.968 4.755 4.704 4.624] OV trace/nuclear -0.272 L2H2 Q/K/V/O norms [13.97 13.239 14.567 14.24 ] QK top [1.11 0.945 0.773 0.625 0.589 0.579 0.563 0.54 ] QK stable rank 8.07 QK symmetric fraction 0.711 OV top [7.863 7.18 6.901 5.989 5.921 5.632] OV trace/nuclear -0.228 L2H3 Q/K/V/O norms [16.121 15.554 13.498 13.391] QK top [1.066 0.993 0.907 0.877 0.822 0.797 0.764 0.743] QK stable rank 14.81 QK symmetric fraction 0.706 OV top [6.18 5.8 5.539 4.932 4.673 4.575] OV trace/nuclear -0.278 L2H4 Q/K/V/O norms [16.876 16.188 13.241 13.143] QK top [1.188 1.105 1.077 1.014 0.915 0.895 0.882 0.849] QK stable rank 15.02 QK symmetric fraction 0.704 OV top [5.07 4.774 4.66 4.505 4.372 4.29 ] OV trace/nuclear -0.265 L2H5 Q/K/V/O norms [16.445 15.839 13.339 13.161] QK top [1.201 1.063 0.969 0.943 0.934 0.905 0.845 0.835] QK stable rank 13.55 QK symmetric fraction 0.704 OV top [4.84 4.693 4.49 4.483 4.358 4.288] OV trace/nuclear -0.292 L3H0 Q/K/V/O norms [15.747 15.258 14.874 14.87 ] QK top [0.79 0.738 0.734 0.707 0.695 0.688 0.673 0.667] QK stable rank 24.41 QK symmetric fraction 0.725 OV top [8.384 7.583 6.42 6.247 5.851 5.639] OV trace/nuclear -0.338 L3H1 Q/K/V/O norms [14.22 14.197 17.885 18.331] QK top [0.571 0.554 0.552 0.536 0.53 0.525 0.522 0.514] QK stable rank 31.82 QK symmetric fraction 0.778 OV top [13.854 13.612 13.378 12.359 12.256 12.167] OV trace/nuclear -0.217 L3H2 Q/K/V/O norms [12.743 12.446 15.523 15.585] QK top [0.808 0.724 0.627 0.583 0.54 0.503 0.472 0.46 ] QK stable rank 10.6 QK symmetric fraction 0.707 OV top [11.891 9.927 8.462 8.285 7.981 7.753] OV trace/nuclear -0.19 L3H3 Q/K/V/O norms [13.283 13.057 15.732 16.118] QK top [0.856 0.707 0.587 0.539 0.517 0.511 0.503 0.473] QK stable rank 10.64 QK symmetric fraction 0.702 OV top [12.434 10.397 9.009 8.061 7.769 7.575] OV trace/nuclear -0.21 L3H4 Q/K/V/O norms [14.356 13.874 15.175 15.3 ] QK top [0.919 0.738 0.695 0.661 0.629 0.593 0.577 0.566] QK stable rank 12.5 QK symmetric fraction 0.707 OV top [9.526 8.963 7.011 6.951 6.494 6.106] OV trace/nuclear -0.195 L3H5 Q/K/V/O norms [ 9.514 9.231 17.993 18.512] QK top [0.483 0.419 0.374 0.349 0.336 0.292 0.263 0.254] QK stable rank 9.4 QK symmetric fraction 0.716 OV top [21.979 19.201 17.421 16.747 16.065 15.831] OV trace/nuclear -0.182 ```