Unit 7B · Recruitment: HTML register · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-09T23:28:31Unit 7 · The early layers across three sizes

What this experiment found

The short version. Gemma 12B also wrote correct HTML, and its early- tag word </strong> reached 217 partway through, its strongest movement of either Gemma model.

What we did. We ran the same HTML test on Gemma 12B. We checked the rank, at each layer, of the tag words seen repeatedly in its .

What we found. Gemma 12B wrote correct, complete HTML boilerplate. The tag </strong> reached rank 217 at layer 18, the strongest movement of any tracked tag word in either Gemma model. Every other tracked tag word stayed in the thousands.

What it means. Rank 217 out of about 250,000 words is a real move upward, but it is still far from rank 1. We think the model can generate the language a tag belongs to, even when that tag itself does not reach a high rank.

What this does not show. Later work found that the early-layer tag words we tracked come from a fixed part of the model's early internal state. That part does not change with the input, so it says nothing about our text. This limits what "the tag's rank changed" can mean.

Conversation

user
Write the raw HTML skeleton for a simple blog post page.
assistant
```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width= (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

gemma-12b writing HTML: same flawless boilerplate, and the strongest sediment-token movement in either Gemma — </strong> to rank 217 at L18, everything else in the thousands. Rank 217 is genuinely elevated and genuinely not content: the model whose early layers are made of close-tags gives those tags only a distant nod while actually writing the language they came from. The CSDN lesson generalizes across families: registers are generated without waking their fossils.

— Claude (Fable 5)

Probing parameters

max_new
40
positions
[-2]
track
["</strong>", "</h1>", "</h2>", "</b>", "<start_of_image>"]
scan
["</strong>", "</h1>", "</h2>", "</b>", "<start_of_image>", "anyways", "alot", "yummy", "kinda", "whilst", "luckily"]

Answer emergence

The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).

Raw rank-of-top1 by layer
layer012345678910111213141516171819202122232425262728293031323334353637383940414243444546
rank11111211111111111111111111111111111111111111111

Data

← prev: Unit 7A · Sediment controls (prompt invariance) · gemma-12bunit listingall recordsword listinterim conclusionsnext →: Unit 7B · Recruitment: romance register · gemma-12b
early layersThe first third of the model. The lens shows a fixed pattern here that does not change with the input. The pattern is real inside the model, but it says nothing about your text.See also: lens, workspace bandall terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →