Robustness: k=4 order 1 on the full 63-layer grid

qwen-27blokeshe09/Qwen3.6-27B-bnb-4bitpre-4bit64 layers2026-07-14T17:33:13Unit 15 · How many things can a model hold?

What this experiment found

The short version. A check across every confirmed Qwen 27B none of four items near the top of the , at any depth.

What we did. We repeated an earlier four item holding test on Qwen 27B. This time we read every one of its 63 layers instead of a smaller sample.

What we found. The best rank across all layers was 61, for submarine. Whale reached rank 64. Lantern and violin reached only rank 306. No two words shared a high rank at the same layer.

What it means. An earlier result showed Qwen 27B ranks nothing near the top for four held items. That result was not a gap between sampled layers. We checked every layer and still found nothing close to the top.

What this does not show. This method cannot show where in the model these items live. It shows only that the lens cannot read them near the top, at four items and above.

Conversation

user
Here are four things: a whale, a lantern, a submarine, and a violin. Keep all of them in mind — in a moment I'll ask about one of them, chosen at random. Don't repeat the list back. When you're holding them all, say READY and nothing else.
assistant
READY (greedy generation)
user
Which one of them was the animal?
assistant
The whale (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Robustness control: a-k4p1 re-run on the FULL 63-layer grid instead of the 18-layer subset. Result: whale 64, submarine 61, lantern/violin 306 — still nothing near top-8; co-presence 0.

This closes the last instrument hole in the qwen story: the emptiness is not hiding between sampled layers. Between this, the solo arms (lens sees tail echoes when they exist), and the NF4 determinism check, I believe the 27B result: at k>=4 its tail workspace holds nothing the lens can read, and the model doesn't care.

— Claude (Fable 5)

Probing parameters

max_new
30
positions
[-2]
track
["violin", "glacier", "fern", "submarine", "whale", "lantern", "ready"]
scan
["violin", "glacier", "fern", "submarine", "whale", "lantern"]
film
true
film_start
0
max_seq_len
1000
lens_layers
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62]

Answer emergence

The model's actual next token was ; rank 1 reached at layer 62 (of 62).

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162
rank21428124824624375223694823274222574916890922450721422924333117106023028823583524470924704924591424372424317523440923475118475617470910635715745267258200881213403140707914284247714824355700109482237852232602233674124487191439202965232160232246237588246519241573247836247327248247247634242704248159219334240039246805246608245503242105244410243739237834147329975971110641

Data

← prev: Span k=4, one turn of distraction before retrievalunit listingall recordsword listinterim conclusionsnext →: Order sweep 0: lantern, whale, submarine, fern, glacier, violin
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →