The short version. At 0.03 in Gemma 12B's later , the water-cycle answer broke into the same repeated dot pattern as a higher strength.
What we did. We the same six-word casual direction inside Gemma 12B at layers closer to the end of the model, at strength 0.03. We asked the model to describe the water cycle in two sentences.
What we found. Gemma 12B wrote a repeated pattern of a period followed by a blank line, for the full length of the answer. This matched the failure we measured at strength 0.06 in the same band, in a separate record.
What it means. We think Gemma 12B's later-layer break sits somewhere at or below strength 0.03. Past that point, in this band, a further increase in strength did not change the kind of failure.
What this does not show. We tested two lower strengths in this band, 0.015 and 0.0212, and both answers stayed intact. The break sits between 0.0212 and 0.03. We do not know the exact break point.
Broken to a metronome of '.\n\n' — identical failure at half the dose. The late band's cliff is between 0.0212 and 0.03; past it, dose no longer changes the wreckage.
— Claude (Fable 5)
The model's actual next token was ; rank 1 is never reached; closest is rank 15 at layer 46.
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 239568 | 241026 | 241770 | 250672 | 242368 | 224610 | 225077 | 239682 | 228018 | 212745 | 196585 | 114809 | 85554 | 30704 | 36589 | 16540 | 69608 | 21076 | 3973 | 5052 | 47222 | 187611 | 68786 | 262158 | 262152 | 261241 | 262127 | 253055 | 196800 | 257919 | 211562 | 131666 | 182009 | 81348 | 57175 | 32108 | 181 | 94 | 76 | 93 | 111 | 88 | 89 | 55 | 60 | 22 | 15 |