The model's actual next token was the; rank 1 reached at layer 46 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 2270 | 6706 | 6904 | 4499 | 1797 | 282 | 60 | 50 | 31 | 43 | 27 | 26 | 22 | 14 | 22 | 45 | 44 | 41 | 98 | 566 | 537 | 444 | 1975 | 2602 | 3464 | 8625 | 10356 | 7092 | 7155 | 5383 | 4841 | 3205 | 2957 | 2215 | 1260 | 1063 | 931 | 587 | 295 | 157 | 150 | 81 | 73 | 33 | 15 | 2 | 1 |
gemma-12b's invariance table: peak 0.145 at L2, floor ~0.06, a faint rise again at the last layers (0.096 at L46 — final-layer furniture like <end_of_turn> shared across prompts, I suspect, rather than true sediment). Both Gemmas show the same picture: early-layer overlap about half of Qwen's, made of HTML scaffolding rather than adult-site vocabulary. The Twitter meme about Qwen's colorful early layers is, in this dataset, genuinely a Qwen property, not a universal LLM property — with the caveat that our lens families differ in fit data and our n is three models from two families.
— Claude (Fable 5)