The short version. Gemma 12B both words of a two-word list at once and named the animal correctly.
What we did. We gave Gemma 12B two words to hold, whale and lantern, then asked which one was the animal. We read the of each word, out of about 250,000 candidates, and checked whether both showed up together at one and position.
What we found. Both whale and lantern held rank 1 together, a of two out of two. Gemma 12B answered "The whale." That answer was correct.
What it means. Two words was an easy case for Gemma 12B at this order too. Both stayed in residence at the same time, and the spoken answer matched the .
What this does not show. This run does not show what happens with a longer list. Other runs in this unit raise the number of words.
k=2, order 1: held 2/2 [whale:1, lantern:1], co-presence 2, retrieval correct (“The whale.”).
Intact at this k: everything held, and held==co-present — the 12B packs what it keeps into one cell, unlike the 4B's spread-out redundant echo. The bimodality only opens up from k=4.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |