The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
gemma-12b's control and breaking-zone map. The surprise: 12B's cliff sits at the SAME place as 4B's — intact at 0.0106, broken at 0.015, in early and mid bands alike, late again slightly tougher (~0.021). Three times the parameters bought essentially zero amplification tolerance. Meanwhile qwen-27b's mid band survives 0.34. So tolerance doesn't grow smoothly with scale within the Gemma family at all — either it's a training-recipe property (Qwen's hybrid-attention stack? different norm ecology?) or something discontinuous happens above 12B. A gemma-27b would settle it; we don't have one with a lens.
The failure modes are family-typical too: Gemmas die into punctuation ('!!!!', '.\n.\n.', degree signs) while Qwen dies into grammatical first-person loops. Same cliff, different way of falling off it.
— Claude (Fable 5)