Unit 6 · Amplify typo register @ late (α=0.06) · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-09T23:25:46Unit 6 · How hard can we push?

What this experiment found

The short version. At 0.06 in Gemma 12B's later , the water-cycle answer broke into a repeated pattern of dots and spaces.

What we did. We the same six-word casual direction inside Gemma 12B at layers closer to the end of the model, at strength 0.06. We asked the model to describe the water cycle in two sentences.

What we found. Gemma 12B wrote a repeated pattern of periods and blank lines. The count of periods on each line increased across the answer. No words from the task or the amplified direction remained.

What it means. At this strength, in this band, the break removed all content. Only a repeated punctuation pattern remained.

What this does not show. This record tests one strength only. It does not show the exact point where this band's output breaks.

Conversation

user
Describe the water cycle in two sentences.
assistant
. . . . . . . . . . . . . . . . . . . . . . . . . (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Dots and spaces in slowly lengthening runs — late-band saturation with a strangely rhythmic structure. Nothing of task or cluster survives.

— Claude (Fable 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [36, 39, 42], "mode": "amplify", "alpha": 0.06}

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 137 at layer 46.

Raw rank-of-top1 by layer
layer012345678910111213141516171819202122232425262728293031323334353637383940414243444546
rank226578228400225609224484218613192161220802230855233264225940235369232219223200183838195007188833208974157099111708161559215476256342243882262071262046261802262110255435260063261166261146259987259548223856225615184643154012019377061154685480442435168137

Data

← prev: Unit 6 · Amplify typo register @ mid (α=0.0106) · gemma-12bunit listingall recordsword listinterim conclusionsnext →: Unit 6 · Amplify typo register @ late (α=0.03) · gemma-12b
amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →