The short version. Gemma 12B named the largest of five words correctly, while only one of the five reached 8 at any single position.
What we did. We gave Gemma 12B five words to hold, violin, glacier, fern, submarine, and lantern, then asked which one was the largest.
What we found. Violin and glacier each reached rank 1 in the . Only one of the five words reached rank 8 or better at any single position, a of one. Submarine fell to rank 187 and lantern to rank 123. Gemma 12B answered "The glacier." That answer was correct.
What it means. Only two of the five words stayed in . Glacier was one of those two, and also the correct answer.
What this does not show. This run does not show that residence causes the correct answer. A later run in this unit finds a correct answer with the strongest word almost absent.
Binding k=5 (largest): violin and glacier at rank 1, submarine evicted to 187, lantern 123; co-presence 1; answer correct (glacier).
The answer is composed from a tail that holds two of five items — one of them happens to be the right answer, but b4 below shows that's not why it gets it right.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |