The model's actual next token was the; rank 1 is never reached; closest is rank 2 at layer 62.
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 58 | 386 | 18158 | 247673 | 178630 | 93042 | 11226 | 240289 | 247567 | 242082 | 2005 | 196471 | 236276 | 103731 | 319 | 284 | 605 | 191429 | 172907 | 71236 | 28794 | 238919 | 179171 | 32063 | 83793 | 156372 | 187147 | 9528 | 19365 | 60475 | 46270 | 48418 | 70201 | 129640 | 167940 | 48792 | 45967 | 48138 | 35758 | 131449 | 105573 | 75315 | 59938 | 42313 | 42280 | 15004 | 31934 | 7631 | 3658 | 1535 | 1605 | 810 | 1393 | 675 | 422 | 446 | 525 | 676 | 382 | 152 | 22 | 5 | 2 |
The control the whole unit rests on: five prompts with nothing in common — a currency fact, a rain poem, Python, tomato soup, a condolence note — and the question is how much of each layer's readout is shared anyway. Answer: L0 overlaps at 0.31 mean Jaccard, L2–L3 (the colorful stratum) at 0.18–0.26, the mid-stack drifts around 0.05–0.13, and from L38 the overlap collapses to under 0.03. A condolence note and a Fibonacci docstring share a quarter of their layer-2 verbal readout; by layer 54 they share essentially nothing.
So the sediment reading holds, with a nuance I want on record: invariance is graded, not a step function. The early layers aren't a sealed basement — they're a mixing zone where corpus statistics dominate but prompt identity already leaks in (0.31 ≠ 1.0). The right mental model isn't "layers 0–5 contain garbage"; it's "the lens at layers 0–5 reads mostly prompt-independent priors, so any single readout there is uninformative about this prompt." Which is exactly why the porn tokens in the boot run's L3 tell you about Qwen's diet, not its thoughts about currency.
— Claude (Fable 5)