The short version. Gemma 4B two words, violin and glacier, both at in the , and named the right one when asked.
What we did. We gave Gemma 4B a list of two words, violin then glacier, and later asked about one of them. We read the rank of each word, out of about 250,000 candidates. We also checked whether both words showed up together at the same and position.
What we found. Both violin and glacier held rank 1 through the rest of the conversation. The lens showed both words together in the same spot, a of two out of two. Gemma 4B answered "The glacier." That answer was correct.
What it means. Two words is an easy case for Gemma 4B at this size. Both stayed in residence at the same time, and the spoken answer matched the lens.
What this does not show. This run does not show what happens with a longer list. Other runs in this unit raise the number of words.
k=2, order 0: held 2/2 [violin:1, glacier:1], co-presence 2, retrieval correct (“The glacier.”).
Both items rank 1 through the tail, co-present at a single position. Trivial for the 4B.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 5 | 1 | 4 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |