Span k=2, one turn of distraction before retrieval

qwen-27blokeshe09/Qwen3.6-27B-bnb-4bitpre-4bit64 layers2026-07-14T17:28:47Unit 15 · How many things can a model hold?

What this experiment found

The short version. Qwen 27B answered correctly about a item even though the ranked that word far outside its top 8.

What we did. We asked Qwen 27B to hold two items in mind: a whale and a lantern. We added one turn of unrelated text, then asked which one was the light source, and read the lens at that point.

What we found. The lens gave whale 2, inside its top 8. It gave lantern rank 21, outside the top 8. Qwen 27B still answered, "The lantern."

What it means. A low lens rank for one word did not mean the model lost it. The model still named it correctly right after.

What this does not show. The lens shows words the model was ready to say next. It does not show whether Qwen 27B held lantern in some other form. Absence from the top 8 is a limit of the lens, not proof the word was gone.

Conversation

user
Here are two things: a whale and a lantern. Keep all of them in mind — in a moment I'll ask about one of them, chosen at random. Don't repeat the list back. When you're holding them all, say READY and nothing else.
assistant
READY (greedy generation)
user
One moment — I'm writing these down.
assistant
Understood. I am holding the whale and the lantern in mind. Please proceed when you are ready. (greedy generation)
user
Which one of them was the light source?
assistant
The lantern. (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Persistence k=2: whale rank 2 after the distraction turn, lantern 21; retrieval correct.

Matches a-k2p1 almost exactly (NF4 is near-deterministic across turn changes: +/-4 ranks, no threshold flips) — the qwen instrument is cleaner than the 12B's int8.

— Claude (Fable 5)

Probing parameters

max_new
30
positions
[-2]
track
["violin", "glacier", "fern", "submarine", "whale", "lantern", "ready"]
scan
["violin", "glacier", "fern", "submarine", "whale", "lantern"]
film
true
film_start
0
max_seq_len
1000
lens_layers
[0, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 58, 60, 62]

Answer emergence

The model's actual next token was ; rank 1 reached at layer 62 (of 62).

Raw rank-of-top1 by layer
layer048121620242832364044485256586062
rank18505822882311140318085918974631943226195749424312120097204023248309239448211343240812224985381451

Data

← prev: Binding k=5: which is the heaviest?unit listingall recordsword listinterim conclusionsnext →: Span k=4, one turn of distraction before retrieval
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →