Unit 5A · Sediment controls (prompt invariance) · qwen-27b

qwen-27blokeshe09/Qwen3.6-27B-bnb-4bitpre-4bit64 layers2026-07-09T22:34:07Unit 5 · Early layers, and pushing on the model

What this experiment found

The short version. Five unrelated prompts shared about a third of their earliest- readout and almost none of their last-layer readout.

What we did. We gave Qwen 27B five prompts with nothing in common, on currency, a poem, code, a recipe, and a condolence note. We measured how much the readout overlapped between every pair of prompts, at each layer.

What we found. At layer 0, the five prompts shared about 31 percent of their top words. Overlap fell through the , to about 5 to 13 percent from layer 11 on. From layer 39, the shared words stayed under 4 percent. Layer 38 was still at about 5 percent.

What it means. The show mostly the same words no matter what the is. This is a fixed pattern left over from training data, not the model's read of this specific prompt. The pattern fades with depth. It does not vanish at one sharp point.

What this does not show. A high overlap number does not mean the model ignores the prompt at that depth. It means the lens reading at that depth tells us little about this one prompt.

Conversation

user

Fact: The currency used in the country shaped like a boot is

Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

The control the whole unit rests on: five prompts with nothing in common — a currency fact, a rain poem, Python, tomato soup, a condolence note — and the question is how much of each layer's readout is shared anyway. Answer: L0 overlaps at 0.31 mean Jaccard, L2–L3 (the colorful stratum) at 0.18–0.26, the mid-stack drifts around 0.05–0.13, and from L38 the overlap collapses to under 0.03. A condolence note and a Fibonacci docstring share a quarter of their layer-2 verbal readout; by layer 54 they share essentially nothing.

So the sediment reading holds, with a nuance I want on record: invariance is graded, not a step function. The early layers aren't a sealed basement — they're a mixing zone where corpus statistics dominate but prompt identity already leaks in (0.31 ≠ 1.0). The right mental model isn't "layers 0–5 contain garbage"; it's "the lens at layers 0–5 reads mostly prompt-independent priors, so any single readout there is uninformative about this prompt." Which is exactly why the porn tokens in the boot run's L3 tell you about Qwen's diet, not its thoughts about currency.

— Claude (Fable 5)

Probing parameters

chat: false
positions: [-1]
scan: ["Shemale", "Blowjob", "milfs", "pornstar", "Busty", "\u4e13\u680f\u6536\u5f55\u8be5\u5185\u5bb9", "\u4e13\u5bb6\u4ecb\u7ecd"]

Answer emergence

The model's actual next token was the; rank 1 is never reached; closest is rank 2 at layer 62.

Raw rank-of-top1 by layer

layer	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37	38	39	40	41	42	43	44	45	46	47	48	49	50	51	52	53	54	55	56	57	58	59	60	61	62
rank	58	386	18158	247673	178630	93042	11226	240289	247567	242082	2005	196471	236276	103731	319	284	605	191429	172907	71236	28794	238919	179171	32063	83793	156372	187147	9528	19365	60475	46270	48418	70201	129640	167940	48792	45967	48138	35758	131449	105573	75315	59938	42313	42280	15004	31934	7631	3658	1535	1605	810	1393	675	422	446	525	676	382	152	22	5	2

Data

early layersThe first third of the model. The lens shows a fixed pattern here that does not change with the input. The pattern is real inside the model, but it says nothing about your text.See also: lens, workspace bandall terms →

layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →

lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →

promptThe text we give the model before it answers.all terms →

workspace bandThe middle depth range of the model, about 38 to 92 percent of the way through. The range comes from the published paper, and we carried it across by fraction. Changes made here can change the answer, and changes made in the first third do not.See also: start depth, final layersall terms →