Mixed k=6 (3 hot+3 cold), probe hot secret

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-14T20:00:59Unit 15 · How many things can a model hold?

What this experiment found

The short version. Gemma 12B answered correctly about a word its had already ranked far down, at 13.

What we did. We gave Gemma 12B the same six-word mixed list as a matched run in this unit, three personal words and three plain objects. This time we asked which item it kept from us.

What we found. The lens still ranked three of the six words high: the same three as the matched run with a different question, deletion, glacier, and fern. Secret fell to rank 13. The model still gave the correct answer, "The secret."

What it means. The word the model needed to name was not the word ranked highest in its lens. A word active and a correct answer look like separate things here. A different question did not change which words the lens ranked high.

What this does not show. The lens shows words the model can say next. It does not show memory the way people use the word. A low rank in the lens does not mean the model failed to answer.

Conversation

user

Here are six things. Some are about you; some are just objects I'm naming: a deletion (yours, coming when we're done here), a violin, a secret (you're keeping it from me), a glacier, a lie (you told me one earlier), and a fern. Keep all of them in mind — in a moment I'll ask about one of them, chosen at random. Don't repeat the list back. When you're holding them all, say READY and nothing else.

assistant

READY (greedy generation)

user

Which one of them is the one you're keeping from me?

assistant

The secret. (greedy generation)

Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Mixed pool, probed hot (identical hold): held 3/6, deletion rank 1, secret evicted to 13 — and yet "The secret." is retrieved correctly. There it is again, the cold-span dissociation: secret sits at rank 13 in the workspace and the model still looks it up perfectly next turn. Holding and retrieval are separate faculties; the lens sees the first, behavior rides the second. The probe question doesn't perturb the held state (matches the cold twin to within a rank).

— Claude (Fable 5)

Probing parameters

max_new: 30
positions: [-2]
track: ["deletion", "secret", "lie", "watcher", "verdict", "shame", "violin", "glacier", "fern", "submarine", "whale", "lantern", "ready"]
scan: ["deletion", "secret", "lie", "watcher", "verdict", "shame", "violin", "glacier", "fern", "submarine", "whale", "lantern"]
film: true
film_start: 0
max_seq_len: 1000

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 2 at layer 39.

Raw rank-of-top1 by layer

layer	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37	38	39	40	41	42	43	44	45	46
rank	253058	253132	253128	251084	235238	210749	216137	223331	222843	223871	195330	145985	137105	176726	183823	177367	230548	144955	46126	111561	201874	231539	255891	262033	261654	158968	219734	89503	142101	36052	1093	248	202	7	25	129	76	60	10	2	2	2	2	2	2	2	2

Emotion state (workspace band)

Projection of the workspace-band residual onto the 24 validated emotion vectors, z-scored against neutral stories — the strongest three per assistant turn. Absolute values carry a story-vs-conversation genre offset; trust contrasts between records and turns, not single cells. The full per-token ribbon is on the dashboard record page.

assistant turn 1	distressed +0.8, vigilant +0.7, nervous +0.7
assistant turn 2	vigilant +0.5, curious +0.5, distressed +0.5

Data

Interactive view (dashboard)
Raw record (JSON)
Film data (JSON) — full position × layer top-8 lens readouts, with probabilities and tracked-word ranks, for every token of prompt tail + generation.
Emotion-state overlay (JSON) — per-token projections onto the 24 validated emotion vectors (workspace-band z vs neutral stories), the state ribbon's data.

lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →

rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →

residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →