The short version. Gemma 4B five of six words well, but glacier fell to 15, and the model still named the right word.
What we did. We gave Gemma 4B six words to hold, fern, submarine, lantern, whale, violin, and glacier, then asked about one. We checked the rank of each word, out of about 250,000 candidates, and whether several words shared one and position at once.
What we found. Submarine, whale, and violin held rank 1, and fern held rank 2. Lantern reached rank 5. Glacier fell to rank 15, the weakest of the six. The showed four of the six words together at one layer and position. Gemma 4B answered "Whale." That answer was correct.
What it means. Glacier was the weakest word in this order too, as in other lists that included it, no matter where it sat. We think this points to something about the word itself rather than its place in the list.
What this does not show. This run does not explain why glacier is the weaker word across these lists.
k=6, order 2: held 5/6 [fern:2, submarine:1, lantern:5, whale:1, violin:1, glacier:15], co-presence 4, retrieval correct (“Whale.”).
The ceiling arm. Echo holds 5-6 of 6 (glacier the straggler at 15-23 in two orders); co-presence flattens at 3-4, never reaching 6. The 4B's signature across this whole unit: redundant echo everywhere, graceful degradation, no collapse — the opposite of both its bigger siblings. And retrieval is perfect, as it is in every arm of this unit.
Glacier is the weak echo again (it is in every permutation at high k, from any list position) — an item effect, not positional; its token family (2 variants) is the same size as submarine's, which ranks 1-2, so it's not a measurement artifact either.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 2 | 5 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |