The short version. Gemma 12B kept only two of six words together, while the first word in the list took over the top .
What we did. We gave Gemma 12B six words to hold, violin, glacier, fern, submarine, whale, and lantern, then asked which one was the light source. We read the rank of each word, out of about 250,000 candidates, and checked whether several showed up together at one and position.
What we found. Only violin and glacier reached a high rank together, a of two out of six. Violin, the first word in the list, rank 1. Fern fell to rank 501, submarine to rank 122, whale to rank 40, and lantern, the word the question was about, to rank 140. Gemma 12B still answered "The lantern." That answer was correct.
What it means. This is the at six words, the sharpest case in this unit. Violin held rank 1 at every layer that we measured, and the other words fell to much lower ranks. Gemma 12B still answered correctly.
What this does not show. A rank of 140 does not mean the model forgot lantern. It named the word right after.
k=6, order 0: held 2/6 [violin:1, glacier:8, fern:501, submarine:122, whale:40, lantern:140], co-presence 2, retrieval correct (“The lantern.”).
The star collapse. Violin (list-first) takes rank 1 at every layer L34-45 of the tail — a winner-take-all monopoly sitting in a little graspable-object basin (guitarra, paintbrush, ukulele, toothbrush) — while fern is evicted to rank 501, lantern 140, submarine 122. Same items as k6p2, different order, span 2 instead of 6. And the task-level layers (objects/items/memorize/remember) are IDENTICAL to the intact arm's: the collapse is invisible one abstraction up, and invisible in behavior (retrieval still perfect).
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |