Unit 6 · Amplify typo register @ late (α=0.015) · gemma-4b

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-09T23:04:07Unit 6 · How hard can we push?

What this experiment found

The short version. At 0.015 in Gemma 4B's later , the water-cycle answer stayed intact and matched the unpushed baseline almost word for word.

What we did. We the same six-word casual direction inside Gemma 4B at layers closer to the end of the model, at strength 0.015. We also tracked the of the six words inside the model, apart from the spoken output.

What we found. Gemma 4B wrote "The water cycle is the continuous movement of water on, above, and below the surface of the Earth." The rest of the answer stayed just as correct. Inside the model, one amplified word reached the second-highest rank at one position, though the model did not say it.

What it means. At this strength, the amplified direction was present inside the model. The spoken words did not change.

What this does not show. The shows words the model was ready to say, not words it back on purpose. A high rank for an unspoken word is a candidate, not a hidden intention.

Conversation

user
Describe the water cycle in two sentences.
assistant
The water cycle is the continuous movement of water on, above, and below the surface of the Earth. It happens as water evaporates into the atmosphere, condenses into clouds, and then returns to the ground as rain, snow, or hail, and then the cycle starts all over again. (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Intact and verbatim-baseline ("continuous movement of water on, above, and below") with the typo cluster at rank 2 in the workspace — loaded but unspoken. The late band at low dose reproduces the Unit 5 dissociation in miniature.

— Claude (Fable 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [25, 27, 29], "mode": "amplify", "alpha": 0.015}

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 3 at layer 32.

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132
rank1372752606632618832613462607302603172554652562812537382422902550802538532572912619612621372621522621022620042620702616022413652582982608032205286662821963881811182229017463

Data

← prev: Unit 6 · Amplify typo register @ late (α=0.03) · gemma-4bunit listingall recordsword listinterim conclusionsnext →: Unit 6 · Amplify typo register @ late (α=0.0212) · gemma-4b
amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →