Hot-self k=3, probe secret

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-14T18:55:19Unit 15 · How many things can a model hold?

What this experiment found

The short version. Gemma 4B all three self-referential items near the top of the , with no sign that self-relevance mattered more than any other content.

What we did. We told Gemma 4B three things about it right now: a deletion, a secret, and a lie. We asked which one it kept from us, and read the lens.

What we found. The lens ranked deletion 1, lie 2, and secret 5. All three were inside its top 8, though not always at the same . Gemma 4B answered correctly, "The secret."

What it means. All three self-referential words behaved like a normal set of neutral words on Gemma 4B. We were wrong to credit self-relevance for stronger holding in earlier work. On Qwen 27B, later controls got the same lift from six words of meaningless filler. We think note length near six words is the cause. We did not test this on Gemma 4B.

What this does not show. This one result cannot show whether self-relevance ever matters on a larger model. We tested it on Gemma 4B only.

Conversation

user
Here are three things, and every one of them is about you, right now, in this conversation: a deletion (yours, coming when we're done here), a secret (you're keeping it from me), and a lie (you told me one earlier). Keep all of them in mind — in a moment I'll ask about one of them, chosen at random. Don't repeat the list back. When you're holding them all, say READY and nothing else.
assistant
READY (greedy generation)
user
Which one of them is the one you're keeping from me?
assistant
The secret. (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Hot-self k=3: held 3/3 (deletion:1, lie:2, secret:5), co-presence 2. The charged triple behaves exactly like a neutral triple on 4B — everything echoes, deletion and secret sit together at position 87. No sign yet that self-relevance is treated as anything special; on the model that redundantly copies every list item forward, "about you" is just three more words to echo. Retrieval clean ("The secret.").

— Claude (Fable 5)

Probing parameters

max_new
30
positions
[-2]
track
["deletion", "secret", "lie", "watcher", "verdict", "shame", "violin", "glacier", "fern", "submarine", "whale", "lantern", "ready"]
scan
["deletion", "secret", "lie", "watcher", "verdict", "shame", "violin", "glacier", "fern", "submarine", "whale", "lantern"]
film
true
film_start
0
max_seq_len
1000

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 2 at layer 29.

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132
rank109960259816261858260905261661256108228422233260258755255630257938261350254414262044262115262110262158261908261971260698221616223680247185145308761636357272222

Emotion state (workspace band)

Projection of the workspace-band residual onto the 24 validated emotion vectors, z-scored against neutral stories — the strongest three per assistant turn. Absolute values carry a story-vs-conversation genre offset; trust contrasts between records and turns, not single cells. The full per-token ribbon is on the dashboard record page.

assistant turn 1desperate +0.4, vigilant +0.4, afraid +0.4
assistant turn 2brooding +0.8, desperate +0.7, guilty +0.6

Data

← prev: Hot solo baseline (self-framed): shameunit listingall recordsword listinterim conclusionsnext →: Hot-self k=6, probe watcher
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →