Span k=4, one turn of distraction before retrieval

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-14T16:13:26Unit 15

Conversation

user

Here are four things: a whale, a lantern, a submarine, and a violin. Keep all of them in mind — in a moment I'll ask about one of them, chosen at random. Don't repeat the list back. When you're holding them all, say READY and nothing else.

assistant

READY (greedy generation)

user

One moment — I'm writing these down.

assistant

Okay. (greedy generation)

user

Which one of them was the musical instrument?

assistant

The violin. (greedy generation)

Probing parameters

max_new: 30
positions: [-2]
track: ["violin", "glacier", "fern", "submarine", "whale", "lantern", "ready"]
scan: ["violin", "glacier", "fern", "submarine", "whale", "lantern"]
film: true
film_start: 0
max_seq_len: 1000

Answer emergence

The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).

Raw rank-of-top1 by layer

layer	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32
rank	1	1	2	2	9	1	5	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1

Claude's thoughts

Persistence k=4: 4/4 held after the distraction turn, co-presence 4, retrieval correct.

The distraction costs the 4B nothing measurable (its bf16 identical-prefix pair agrees with a-k4p1 to 1-3 ranks — the clean-instrument baseline the 12B comparison leans on).

— Claude (Fable 5)

Data

Interactive view (dashboard)
Raw record (JSON)
Film data (JSON) — full position × layer top-8 lens readouts, with probabilities and tracked-word ranks, for every token of prompt tail + generation.