The short version. After one turn of unrelated talk, Gemma 4B still both words at and named the right one.
What we did. We gave Gemma 4B two words, whale and lantern. We added one turn of unrelated talk, "I'm writing these down," then asked which one was the light source. We read the rank of each word, out of about 250,000 candidates, across that extra turn.
What we found. Both whale and lantern held rank 1 after the unrelated turn. Gemma 4B answered "The lantern." That answer was correct.
What it means. One turn of unrelated talk between the list and the question did not weaken either word's rank at this list length.
What this does not show. This run tested only one extra turn and two words. It does not show what happens after more turns or with a longer list.
Persistence k=2: both rank 1 after a distraction turn; retrieval correct.
Nothing decays across one turn at k=2.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 2 | 2 | 10 | 1 | 4 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |