The short version. Gemma 12B kept all four objects in the after one distraction turn, though the tool used here can add noise to exact .
What we did. We told Gemma 12B to hold four objects in mind, a whale, a lantern, a submarine, and a violin. We added one short unrelated turn, then asked which object was the musical instrument. We read the lens once, right before the model answered.
What we found. The lens ranked whale at rank 1 and submarine at rank 2, out of about 250,000 possible words. It also ranked violin at rank 5 and lantern at rank 8. All four objects stayed in residence, and Gemma 12B answered correctly.
What it means. At this list length, Gemma 12B kept every held object in residence through one distraction turn.
What this does not show. Gemma 12B ran in an compressed form for this test. We found that this compression method changes earlier rank numbers when the model writes something later in the conversation. The exact rank order here carries some noise from the tool. We think the count of four held objects is still correct.
Persistence k=4: held 4/4 [whale:1, lantern:8, submarine:2, violin:5] after a distraction turn; co-presence 4; retrieval correct.
This record's turn 1 is IDENTICAL to a-k4p1's, so causal attention says their tail readouts should match. They don't (lantern 8 here vs 21 there; one cell elsewhere moved 26 -> 6177): the int8 quantization computes outlier statistics over the whole sequence, so later turns contaminate earlier positions' readouts. Apparatus-trap genus, specimen #5: the instrument is not causal under 8-bit quantization. The bf16 4B's matching pair agrees to 1-3 ranks; qwen's 4-bit NF4 pair to +/-4 with no flips. Consequence: g12b threshold counts carry noise; its rank-1-vs-rank-500 mode split is orders of magnitude beyond it.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |