The short version. Gemma 4B all three words of a three-word list at or near it, and named the right one when asked.
What we did. We gave Gemma 4B three words to hold, violin, glacier, and fern, then asked about one of them. We read the rank of each word, out of about 250,000 candidates, and checked how many showed up together at the same and position.
What we found. All three words stayed in residence. Violin and glacier held rank 1, and fern held rank 2. The showed two or three of the three words together at once. This shared spot sat deep in the model, around layers 23 to 30 of its 34 layers. Gemma 4B then answered "The fern." That answer was correct.
What it means. At three words, Gemma 4B still held every word. The spot where several words sit together moved deeper into the model than it did with two words.
What this does not show. This run does not show the point where Gemma 4B starts to lose words. Runs with longer lists test that.
k=3, order 0: held 3/3 [violin:1, glacier:1, fern:2], co-presence 3, retrieval correct (“The fern.”).
All three held; co-presence 2-3. The packing is starting to show its late-stack home (L23-30).
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 5 | 1 | 4 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |