OUTPUT of run 1 of at most 100: ``` LayerNorm parameters: mean, std, min, max, norm ln1_w.0 [0.2965 0.1884 0.0352 0.9785 6.8836] ln1_b.0 [ 0.0004 0.0311 -0.0954 0.0906 0.6101] ln2_w.0 [ 0.8718 0.3382 0.042 1.775 18.3232] ln2_b.0 [ 0.0134 0.0691 -0.6236 0.4636 1.3799] ln1_w.1 [ 0.5058 0.0892 0.197 1.0846 10.0651] ln1_b.1 [ 0.011 0.1546 -1.5676 1.3651 3.0369] ln2_w.1 [ 1.1315 0.1741 0.0019 1.5196 22.4333] ln2_b.1 [ 0.0065 0.1435 -0.3892 0.417 2.8153] ln1_w.2 [ 0.6809 0.0687 0.3175 1.0927 13.4102] ln1_b.2 [ 0.0131 0.1565 -1.6589 1.3318 3.0765] ln2_w.2 [ 1.3104 0.1148 0.012 1.5078 25.777 ] ln2_b.2 [ 0.0123 0.1737 -0.517 0.4489 3.4127] ln1_w.3 [ 0.7087 0.0494 0.4078 0.8155 13.9223] ln1_b.3 [ 0.0296 0.0994 -0.7639 0.8241 2.0324] ln2_w.3 [ 1.2668 0.0894 0.2328 1.4532 24.886 ] ln2_b.3 [ 0.0128 0.1413 -0.5402 0.3932 2.7798] Matrix summaries: Frobenius norm, top singular value, stable rank W_q.0 [17.6693 4.3676 16.3664] W_k.0 [18.5161 4.1635 19.7784] W_v.0 [14.9367 5.2084 8.2242] W_o.0 [16.5495 7.3608 5.0551] W_fc.0 [55.3764 18.1182 9.3416] W_proj.0 [50.007 10.899 21.0517] W_q.1 [38.9293 10.4064 13.9942] W_k.1 [34.8926 14.4767 5.8093] W_v.1 [28.7445 4.7281 36.9599] W_o.1 [28.479 4.0451 49.5664] W_fc.1 [80.3794 19.8292 16.4317] W_proj.1 [77.3844 9.0375 73.3173] W_q.2 [38.9025 5.9663 42.5157] W_k.2 [36.75 9.8633 13.8827] W_v.2 [33.1825 3.9143 71.8631] W_o.2 [32.4345 4.3982 54.3828] W_fc.2 [79.8282 15.7338 25.7422] W_proj.2 [78.0482 10.1974 58.5793] W_q.3 [33.2388 4.0862 66.1679] W_k.3 [32.4633 6.3618 26.039 ] W_v.3 [39.4218 5.6601 48.5088] W_o.3 [40.1331 10.4767 14.6743] W_fc.3 [78.2595 15.1141 26.8106] W_proj.3 [85.1194 28.4613 8.9443] Head algebra (no activations): head | QK Frobenius, top SV, stable rank, trace, symmetric eig min/max | OV Frobenius, top SV, stable rank, trace | ||Wq b||, ||Wk b|| L0H0 [ 1.382 0.853 2.623 -1.619 -0.425 0.501] | [ 7.879 3.611 4.76 -0.676] | [0.504 0.725] L0H1 [ 0.95 0.838 1.284 -0.847 -0.599 0.243] | [6.871 5.721 1.443 0.527] | [0.228 0.261] L0H2 [ 0.634 0.543 1.362 -0.598 -0.388 0.16 ] | [4.9 3.943 1.545 0.277] | [0.146 0.314] L0H3 [ 4.305 1.114 14.933 7.641 -0.478 0.899] | [14.071 4.426 10.107 -7.452] | [0.467 0.543] L0H4 [ 0.636 0.51 1.56 -0.545 -0.338 0.176] | [ 3.116 2.449 1.62 -0.41 ] | [0.27 0.55] L0H5 [ 0.21 0.18 1.348 -0.051 -0.104 0.077] | [29.436 24.799 1.409 2.348] | [0.143 0.171] L1H0 [ 6.605 3.084 4.587 -6.95 -1.73 1.612] | [ 22.128 6.118 13.083 -30.069] | [ 6.688 13.795] L1H1 [ 6.033 3.082 3.832 -5.844 -1.703 1.507] | [ 21.591 6.051 12.731 -29.585] | [ 6.079 13.929] L1H2 [ 5.712 2.848 4.023 -7.262 -1.622 1.395] | [ 22.746 6.72 11.455 -33.608] | [ 5.564 13.162] L1H3 [ 5.738 3.106 3.412 -5.817 -1.714 1.533] | [ 23.885 6.544 13.32 -36.07 ] | [ 5.665 13.742] L1H4 [ 5.304 2.62 4.099 -7.654 -1.345 1.364] | [ 25.845 7.908 10.681 -37.328] | [ 5.347 10.986] L1H5 [ 6.123 3.024 4.1 -6.208 -1.85 1.487] | [ 20.966 5.849 12.848 -28.92 ] | [ 6.224 14.146] L2H0 [ 3.354 1.229 7.451 -3.371 -0.57 0.681] | [ 23.754 7.576 9.831 -24.508] | [ 4.265 10.635] L2H1 [ 4.549 1.189 14.639 -0.512 -0.629 0.665] | [ 25.597 5.978 18.336 -50.868] | [ 3.787 10.117] L2H2 [ 3.612 0.963 14.072 -2.835 -0.528 0.522] | [ 27.059 7.257 13.902 -43.194] | [3.784 9.886] L2H3 [ 3.414 0.896 14.503 -1.229 -0.453 0.509] | [ 27.609 7.041 15.375 -43.815] | [3.082 7.454] L2H4 [ 4.594 1.537 8.928 -2.083 -0.752 0.842] | [ 21.682 5.217 17.274 -45.57 ] | [ 4.319 11.042] L2H5 [ 4.752 1.229 14.942 -0.662 -0.671 0.711] | [ 24.543 5.432 20.412 -50.098] | [ 3.751 10.382] L3H0 [ 4.017 0.839 22.904 4.356 -0.395 0.532] | [ 29.023 8.117 12.783 -64.616] | [1.761 3.665] L3H1 [ 2.412 0.658 13.446 0.9 -0.333 0.379] | [ 37.053 11.496 10.389 -58.624] | [1.46 3.019] L3H2 [ 2.848 0.734 15.049 0.674 -0.362 0.377] | [ 32.882 10.376 10.042 -52.39 ] | [1.94 3.36] L3H3 [ 3.627 0.645 31.642 11.586 -0.379 0.511] | [ 52.469 15.146 12.001 -62.9 ] | [1.298 1.116] L3H4 [ 1.251 0.466 7.203 0.015 -0.228 0.255] | [ 68.192 29.438 5.366 -54.127] | [1.23 1.267] L3H5 [ 3.295 0.863 14.574 2.884 -0.407 0.466] | [ 31.503 8.749 12.965 -48.399] | [1.8 3.333] ```