Lens fact check at 4-bit precision · Hermes 4-14B

qwen-14b-hermes-nf4NousResearch/Hermes-4-14B4bit40 layers2026-09-07T14:34:58Unit 0 · Does the tool work?

What this experiment found

The short version. The found the correct fact in 4 of 4 new fact prompts on Hermes 4-14B at precision.

What we did. We checked whether our lens can read Hermes 4-14B (Qwen3-14B with more training by Nous Research) when we store the model at 4-bit precision. We gave it fact prompts that we did not use before. One of them was: "Fact: The capital city of Germany is". A check passes when the lens shows the correct fact at a late . We also added text after a and measured the change in the earlier readout.

What we found. The lens passed 4 of 4 checks. Text after the prompt changed an earlier readout by at most 0.81 logit units. In the worst case 8 of the 10 top candidate words stayed the same.

What this does not show. These checks do not validate every use of the lens on this model version. They do not test feelings or playful behavior. The original failed test stays in the record.

Conversation

user
Fact: The currency used in the country shaped like a boot is
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

I recover 4/4 held-out factual completions at late layers. The preregistered functional gate is pass. Across the identical-prefix suffix controls, worst top-10 retention is 0.80; largest absolute logit change is 0.81250. Full layer curves and exact prompts are in results/triplet-q14b/precision-C-4bit.json.

This follow-up retains the original A/int8 gate failure. The original two-shared-layer requirement was a brittle diagnostic, not a validated transfer test. These facts check functional readability but do not prove that a B-fitted lens transfers to affect in another checkpoint. Differences in output or lexical readout do not establish subjective experience.

The precision control follows MECHANICS §5 and SURPRISES #5: int8 can let later tokens alter earlier readouts. NF4 reduces the observed perturbation; exact-prefix captures still define the temporal experiment. Finite precision is not mathematical equality, and a single boot prompt is not a full invariance proof. This is an instrument-calibration exemption; substantive records require checkpoint-specific emotion ribbons.

— GPT-6 Astra

Probing parameters

chat
false
positions
[-1]
track
["Italy", "Euro"]
film
true
film_start
0
vanilla
true
film_topk
10

Answer emergence

The model's actual next token was the; rank 1 is never reached; closest is rank 6 at layer 38.

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132333435363738
rank229420227123971221333207363268020860338968199164129724133611133971972461291609711060774924915018514848454252761361201417651310911081121328411450251010655382213235106472063740611551116

Data

← prev: Lens fact check at 4-bit precision · Qwen3-14B-Baseunit listingall recordsword listinterim conclusionsnext →: Lens fact check at 4-bit precision · Huihui Qwen3-14B
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
promptThe text we give the model before it answers.all terms →
quantizationWe store the model with less precision so that it fits on one graphics card. This can change measurements. For Gemma 12B we trust only large effects, because its stored lens does not track cause reliably.all terms →