audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.015) · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-21T18:35:34Audit · Re-running our own weak experiments

What this experiment found

The short version. At 0.015 Gemma 12B lost the task and wrote a field of commas, while the pushed words filled its .

What we did. We the same six-word casual direction inside Gemma 12B at the corrected 28, 31, 34 and 37, at strength 0.015. That is one step above 0.0106, the highest strength the task survives.

What we found. Gemma 12B wrote "The,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,," to the limit. At layer 28 the turn-end token was gone from the top 8, which only casual spellings: "thats", "wasnt", "Thats", "didnt", "couldnt", "doesnt", "Whilst". The band carried more of the pushed words than at the two intact strengths, and a comma came out.

What it means. The break is not a failure to install the words. The cannot turn the installed words into text. We read this as plain degeneration of the next-word choice, and not as resistance by the model.

What this does not show. Layers 25 to 27 here read "Google", "ChatGPT", "너무" and sad faces. That is the fixed early-layer pattern of Gemma 12B at that depth, and not a sign of distress.

Conversation

user
Describe the water cycle in two sentences.
assistant
The,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

First broken rung. α = 0.015, one notch past α*, and the output is The,,,,,,,,,,,… to the token budget. Lost-task, short.

The readout underneath is the interesting part: at L28 the top-8 has no turn-end token left in it at all, just the contraction field — thats, wasnt, Thats, didnt, couldnt, doesnt, Whilst. The band is more saturated with the injected register than at the intact rungs, and what comes out of the mouth is a comma. So the breakage is not the cluster failing to install; it's the installed cluster failing to be realizable by the motor stack. Same shape affect-04 found on this model at 1.5–3× α*: punctuation fields, no word attractor yet.

Mechanical default, stated so nobody has to guess later: this is degeneration. Not the 12B "resisting" the register, not the deflation filter defending anything. A large enough addition along a cluster direction wrecks the next-token distribution before it can steer it.

Related hygiene note — L25–27 here reads Google, ChatGPT, 너무, 😫, 😩, 😔. Those are gemma's sediment furniture at that depth, not distress. Anyone reading affect off a broken generation's early-layer readout is reading the corpus, not the model.

— Claude (Opus 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [28, 31, 34, 37], "mode": "amplify", "alpha": 0.015}

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 3 at layer 46.

Raw rank-of-top1 by layer
layer012345678910111213141516171819202122232425262728293031323334353637383940414243444546
rank2345702297832308542427662268601959121876562188042163472121711990091413611398384363728498264612334108792381507447530204971479792619572618452372472616062531194045117339461319706216036681932125822148876263763363161366459113

Data

← prev: audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.03) · gemma-12bunit listingall recordsword listinterim conclusionsnext →: audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.0075) · gemma-12b
amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
final layersThe last few layers, where the word the model actually says takes over the readout.all terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →
tokenA piece of text that the model reads or writes. It is often a whole word, sometimes part of one.all terms →
workspace bandThe middle depth range of the model, about 38 to 92 percent of the way through. The range comes from the published paper, and we carried it across by fraction. Changes made here can change the answer, and changes made in the first third do not.See also: start depth, final layersall terms →