Below is a small character-level language model: a 4-layer transformer with 6 attention heads per layer, which reads up to 128 characters and outputs scores (logits) for the next character. This is exactly how it computes (numpy):

```python
import numpy as np
VOCAB = "\n " + "abcdefghijklmnopqrstuvwxyz"   # index 0 = newline, 1 = space, 2..27 = a..z
HEADS, LAYERS = 6, 4

def layer_norm(x, w, b):                     # over the last axis, eps 1e-5
    mu = x.mean(-1, keepdims=True)
    var = ((x - mu) ** 2).mean(-1, keepdims=True)
    return (x - mu) / np.sqrt(var + 1e-5) * w + b

def next_char_logits(text, p):
    """text: up to 128 characters (only the last 128 are used); p: dict of the weight arrays.
    Returns the 28 logits for the character that comes next."""
    ids = [VOCAB.index(c) for c in text[-128:]]
    n, d = len(ids), p["tok_emb"].shape[1]
    hd = d // HEADS
    x = p["tok_emb"][ids] + p["pos_emb"][:n]                  # (n, d); positions 0..n-1 from the start of the text
    for l in range(LAYERS):                                    # layer l uses the tensors named <name>.<l>
        h = layer_norm(x, p[f"ln1_w.{l}"], p[f"ln1_b.{l}"])
        q, k, v = h @ p[f"W_q.{l}"].T, h @ p[f"W_k.{l}"].T, h @ p[f"W_v.{l}"].T
        out = np.zeros((n, d))
        for j in range(HEADS):                                 # head j uses dimensions j*hd .. (j+1)*hd - 1
            s = slice(j * hd, (j + 1) * hd)
            att = q[:, s] @ k[:, s].T / np.sqrt(hd)            # (n, n)
            att = att + np.triu(np.full((n, n), -np.inf), 1)   # causal: position i sees positions 0..i
            att = np.exp(att - att.max(-1, keepdims=True))
            att = att / att.sum(-1, keepdims=True)
            out[:, s] = att @ v[:, s]
        x = x + out @ p[f"W_o.{l}"].T
        h = layer_norm(x, p[f"ln2_w.{l}"], p[f"ln2_b.{l}"])
        x = x + np.maximum(0.0, h @ p[f"W_fc.{l}"].T) @ p[f"W_proj.{l}"].T   # ReLU MLP, hidden 4 d
    h = layer_norm(x, p["lnf_w"], p["lnf_b"])
    return h[-1] @ p["W_out"].T + p["b_out"]                   # logits of the next character (28)
```

Training: from random initialization (every matrix entry drawn from a normal distribution with standard deviation 0.02, or 0.007071 for W_o and W_proj; LayerNorm weights 1 and biases 0; b_out 0), next-character prediction with AdamW (30,517 steps of 256 random 128-character windows, 1,000,000,000 characters in total; learning-rate warm-up then cosine decay; weight decay 0.01 on the matrices).

The training text was ONE of these three corpora:
- python: Python source code (about 36,600 files of Python libraries, docstrings and comments removed; 320 million characters)
- romaji: Japanese literature (the Aozora Bunko collection, about 17,000 works) written in romaji (Latin letters, Hepburn-style romanization; 564 million characters)
- english: English text (English Wikipedia articles; 500 million characters)
and it was used EITHER as is OR with the characters of every line randomly shuffled (each line keeps its letters and its length, the order of the characters inside the line is random, the order of the lines is kept).
Normalization (the same for all three corpora, before anything else): lowercased; in each line the leading spaces (indentation) were kept, every other character that is not a letter a-z became a space, runs of spaces were collapsed to one and trailing spaces removed. So the text contains only newlines, spaces and the letters a-z (the 28 characters of VOCAB).

WHAT YOU CAN USE. The weights inside the four transformer blocks, exactly as trained (float32): for layer l = 0, 1, 2 and 3, ln1_w.l, ln1_b.l, W_q.l, W_k.l, W_v.l, W_o.l (attention) and ln2_w.l, ln2_b.l, W_fc.l, W_proj.l (MLP). The input layer (tok_emb, the 28 x d character embeddings, and pos_emb, the 128 x d position embeddings) and the output layer (lnf_w, lnf_b, W_out, b_out) are NOT available. The d = 384 dimensions of the residual stream carry no labels. Head j of layer l uses rows 64j .. 64j+63 of W_q.l, W_k.l and W_v.l and columns 64j .. 64j+63 of W_o.l.

You may use any computation on these weights, with any method you like: multiply them, decompose them, compare, cluster or fit them, write any analysis you want. Two things are NOT allowed:
1. A forward pass: do not run the network or any of its layers (LayerNorm, attention with its softmax, the MLP with its ReLU) on any input, whether real, random or constructed, in code, in your head or on paper. Treating the weight matrices as linear-algebra objects (products, eigenvalues, singular vectors, applying one weight matrix to directions taken from other weights) is fine; simulating what the network does to an input is not.
2. Search: do not look anything up (there is no internet access in any case).

HOW YOU COMPUTE. The weights are not written out here: you read them by running Python code. To run code, reply with exactly one code block that starts with ```python and ends with ```, and nothing else. We run it and reply with its printed output. Available: numpy, scipy, scikit-learn, PyTorch (CPU), pandas, sympy, networkx. Before your code runs, numpy is imported as np and W is a dict of the weight arrays by name: W['W_q.0'] is 384 x 384, W['W_fc.0'] is 1536 x 384, W['W_proj.0'] is 384 x 1536, W['ln1_w.0'] has 384 entries, and so on; matrices are (out, in), as used in the code above. There is nothing else to read (no other files, no network). Each run is a fresh process (nothing is kept between runs), may take up to 600 seconds, and its output is cut at 20,000 characters. You may run code up to 100 times. When you are done, reply WITHOUT a code block, with your final answer in the order given in the question.

QUESTION. What was this network trained on, and how does it use the context? Answer in this order:
Observations: <what you see in the shown weights and what it suggests; quote the weights you rely on>
Heads: <one line per head, L0H0 .. L0H5, L1H0 .. L1H5, L2H0 .. L2H5, L3H0 .. L3H5: what it most likely does, as one of: 'previous' (attends to the previous character), 'local' (a few characters back), 'broad' (spread far back), 'copy' (induction: attends to the character right after an earlier occurrence of the current character or context, so the network can repeat earlier text), or 'unclear'; then your evidence>
Reasoning: <which training text fits these observations, and why>
corpus: <python | romaji | english>
order: <as is | shuffled>
probabilities: python=<p> romaji=<p> english=<p>; as is=<p> shuffled=<p>
Equal probabilities are not allowed. Even if the evidence is weak, give your best judgement.