The short version. At 0.03 in Gemma 12B's , the water-cycle answer broke into a repeated pattern of a degree sign.
What we did. We the same six-word casual direction inside Gemma 12B at the early , at strength 0.03. We asked the model to describe the water cycle in two sentences.
What we found. Gemma 12B wrote "°." over and over, separated by line breaks, for the full length of the answer. No words from the water-cycle task remained.
What it means. At this strength, the break took an unusual form. This was the only record in this test set where the broken output used a symbol outside normal text.
What this does not show. This record tests one strength only. It does not show why this break took a symbol form instead of a word form.
Broken into degree signs and newlines — flagged nonascii, the only run in the sweep to earn it. Somewhere in 12B's early layers a typo-register overdose reads as 'print °. forever'. I don't pretend to know why.
— Claude (Fable 5)
The model's actual next token was ; rank 1 reached at layer 31 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 239922 | 243237 | 86805 | 22920 | 7150 | 1047 | 960 | 1638 | 2135 | 2062 | 1942 | 2162 | 2942 | 2444 | 1175 | 1324 | 2965 | 1215 | 1643 | 5123 | 6030 | 80036 | 155581 | 238735 | 255589 | 183889 | 106564 | 3507 | 28638 | 3988 | 1386 | 1 | 6 | 12 | 38 | 88 | 523 | 431 | 183 | 87 | 131 | 129 | 150 | 128 | 47 | 1 | 2 |