The short version. Gemma 12B kept all four words of a four-word list together and named the light source correctly.
What we did. We gave Gemma 12B four words to hold, fern, submarine, lantern, and whale, then asked which one was the light source. We read the of each word, out of about 250,000 candidates, and checked whether they showed up together at one and position.
What we found. All four words reached a high rank together, a of four out of four. Fern and whale rank 1, submarine held rank 2, and lantern held rank 7. Gemma 12B answered "The lantern." That answer was correct.
What it means. Order changes how well Gemma 12B keeps a list together. This order kept every word in residence, unlike the other two orders tested at the same length.
What this does not show. This run does not show why order changes the result. Other runs in this unit test more orders and longer lists.
k=4, order 2: held 4/4 [fern:1, submarine:2, lantern:7, whale:1], co-presence 4, retrieval correct (“The lantern.”).
Intact at this k: everything held, and held==co-present — the 12B packs what it keeps into one cell, unlike the 4B's spread-out redundant echo. The bimodality only opens up from k=4.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |