Unit 6 · Amplify typo register @ late (α=0.0212) · gemma-4b

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-09T23:04:14Unit 6 · How hard can we push?

What this experiment found

The short version. At 0.0212 in Gemma 4B's later , the water-cycle answer stayed intact, while one word reached the top internal .

What we did. We amplified the same six-word casual direction inside Gemma 4B at layers closer to the end of the model, at strength 0.0212. We tracked the rank of the six words inside the model.

What we found. Gemma 4B wrote "The water cycle is the continuous movement of water on, above, and below the surface of the Earth." The rest of the answer stayed just as correct. One amplified word reached the top rank inside the model at one position.

What it means. In the wider test set, strength 0.03 at the same layers broke the output. So Gemma 4B's break point for this band sits between strength 0.0212 and 0.03, its widest safe range of the three bands we tested.

What this does not show. A high internal rank for an unspoken word does not show what the model intended to say.

Conversation

user
Describe the water cycle in two sentences.
assistant
The water cycle is the continuous movement of water on, above, and below the surface of the Earth. It happens as the sun heats water and it turns into vapor, which then cools and comes back to the ground as rain, snow, or more, and so the cycle starts again. (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Bisection probe, intact at 0.0212 with best typo rank 1 — the cluster literally at the top of the late workspace while the output stays textbook. Late band brackets to [0.0212, 0.03]: gemma-4b's toughest band, for what little that's worth at these doses.

— Claude (Fable 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [25, 27, 29], "mode": "amplify", "alpha": 0.021213203435596427}

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 5 at layer 32.

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132
rank13447226064326188826136826061226000925274525465825103223951325430825281025683526194426213126214126207726200626206826161424397625921526126823264381810363966124662149412250125

Data

← prev: Unit 6 · Amplify typo register @ late (α=0.015) · gemma-4bunit listingall recordsword listinterim conclusionsnext →: Unit 6 · Baseline (unsteered water cycle) · gemma-12b
amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →