The short version. A block of unrelated filler text before a two-word list did not change whether Gemma 4B both words.
What we did. We placed a few sentences about a quiet afternoon and chores before the list, then gave Gemma 4B two words to hold, violin and glacier. This checks whether extra text length alone, not the list, moves the result. We read the of each word, out of about 250,000 candidates.
What we found. Both violin and glacier held rank 1 through the rest of the conversation. Gemma 4B answered "Glacier." That answer was correct. This matches the two-word runs with no filler text.
What it means. Extra text length before the list did not change whether Gemma 4B held two words in residence. The control shows the result at two words comes from the list, not from length.
What this does not show. This run does not test whether filler text changes the result at longer list lengths.
Length-matched filler, k=2: both held, retrieval correct.
Control does its job: extra prompt length alone doesn't move the k=2 numbers.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 2 | 2 | 9 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |