What Question Is Your Evidence Answering?

A linear probe reads refusal from your model’s activations with 98% accuracy. Steering along the probe direction shifts behaviour, so you write up the finding: the model encodes refusal in this layer. Three weeks later the steering vector miscalibrates on out-of-distribution prompts, the probe fails under paraphrase, and an ablation that suppressed the behaviour in testing does nothing on deployment traffic.

Failures of this kind are predictable at submission time. The claim asked for more than the evidence provided, and causal inference has precise vocabulary for the mismatch.

This post is built around four exercises. Place evidence on its rung, see why a low loss identifies nothing, score yourself against 186 annotated claims from the literature, and rewrite a claim so the language matches the method.

Every interpretability question sits on a rung

Causal inference sorts questions into Pearl’s hierarchy. Associational questions ask what correlates with what. Interventional questions ask what happens under a controlled manipulation. Counterfactual questions ask what would have happened, for this very input, under an unobserved change. Different rungs demand different evidence, and evidence from a lower rung cannot license claims from a higher one, no matter how much of it you collect.

Interpretability methods sit on this ladder whether or not they acknowledge it. Probing is associational. Ablation, patching, and steering are interventional. Claims that a feature means something, or that a circuit is the circuit computing a behaviour, are counterfactual claims about uniqueness and semantics.

There is a second question the field tends to skip. Even at the right rung, how many answers are consistent with your evidence? Three experiments follow. Place each on the ladder before reading the answer.

The purpose of evidence is elimination. Climbing rungs shrinks the set of hypotheses consistent with what you measured, and what remains at the top is still an equivalence class. Whether that class collapses to a single answer, up to harmless symmetries, is the question of identifiability.

One claim, two gaps

Consider the claim from the opening story, that the model encodes refusal in this layer. It fails in two independent ways, and the failures compound.

Gap 1, the rung gap. Used downstream, for steering or editing, the claim requires interventional evidence that the model uses this direction in its computation. The method provides associational evidence that a probe fires on refusal prompts. Decodability does not imply model use. The probe may be reading token length, position artefacts, or any confounder that co-varies with refusal.

Gap 2, the identification gap. The claim says the direction, as if the data singled one out. Many probes and SAE features decode refusal equally well. The evidence defines an equivalence class of solutions, and the method returned one member of it, selected by initialisation and optimiser quirks.

The ladder animation, for completeness:

We audited 186 claims

We annotated 50 interpretability papers, 186 claims about model internals in total, assigning each a method rung (what the experiment establishes) and a claim rung (what the surrounding language asserts). Roughly half the claims admit a stronger causal reading than the reported evidence licenses. The figure is 47% to 54%, depending on how strictly linguistic convention is coded.

The patterns are systematic. Patching results carry counterfactual verbs such as “encodes” and “THE circuit”. Probing findings carry interventional ones. Single-distribution results are generalised past their scope. The gap rate is the same in abstracts and body text, which points to missing terminology rather than space pressure, and none of it means the underlying findings are wrong. It does mean the field lacks a shared vocabulary that tracks evidential strength, and that vocabulary is cheap to adopt.

The claims below are drawn from the annotated dataset, verbatim. You see the sentence and the method, and you assign the gap.

Why low loss can’t identify anything

This part is a theorem. A method learns a representation $\mathbf{h} = \phi(\mathbf{a})$ from activations $\mathbf{a}$, and the claim is that $\mathbf{h}$ recovers concepts $\mathbf{c}$. For any invertible map $v$, the pair $(\phi' , \psi') = (v \circ \phi, \psi \circ v^{-1})$ achieves exactly the same reconstruction loss, so the objective cannot distinguish a solution from its remixes. Without additional structure, unsupervised recovery of latents is impossible (Hyvärinen & Pajunen, 1999; Locatello et al., 2019). Adding a sparsity penalty expresses a preference among solutions. It does not guarantee the preferred solution is the right one.

The demo below makes the theorem concrete. Two concepts co-occur, as real concepts do. Rotate the learned basis onto the true concept axes using the information an unsupervised objective actually has, then lock in a guess.

Switching the data regime to independent shows the other side. There the ℓ₁ penalty does find the true axes, because the sparsity assumption holds in the data. Real concepts correlate, and that is where an inductive bias stops being enough.

Causal representation learning supplies the missing structure through interventions, paired contexts, and sparsity of changes. With the paired interventions switched on above, the objective acquires minima exactly at the true axes in both regimes, and what survives is

where $\mathbf{P}$ is a permutation and $\boldsymbol{\Lambda}$ a diagonal scaling. The notches at 0° and 90° in the demo are this equivalence class. The leftover ambiguity is harmless relabelling. Sparse shift autoencoders realise the recipe on LLM activations, where paired samples over uniformly sampled contexts identify the varied latents (Joshi et al., 2025).

Identifiability is also interaction-relative. Two iron filings, one magnetised and one not, are indistinguishable under looking, weighing, and lifting. A magnet makes the difference visible. What a method can identify is bounded by the interactions it affords, and a representation may show structure only under the right one.

Eight ways claims outrun evidence

The two gaps from the opening are instances of a pattern. The paper catalogues the recurring estimand–evidence gaps of mechanistic interpretability, each an implied claim sitting above its actual scope.

The research programme

The failure modes above are addressable. Each direction below pairs a tool from causal representation learning with an interpretability testbed.

Diagnose your claim

The paper includes a practitioners’ checklist (§G.6). The short version takes a minute.

Much of the problem is vocabulary. The linter below flags rung-elevated verbs in a sentence, suggests a rewrite at the rung the method supports, and produces a scope statement, which is the checklist’s final item.

None of this raises publication barriers. A paper that states its rung, names its equivalence class, and bounds its scope contributes more than one whose language implies stronger evidence than it reports, even when its claims are narrower.

Cite

@inproceedings{joshi2026causality,
  title     = {Position: Causality Is Key for Interpretability Claims to Generalise},
  author    = {Joshi, Shruti and Mueller, Aaron and Klindt, David and Brendel, Wieland
               and Reizinger, Patrik and Sridhar, Dhanya},
  booktitle = {International Conference on Machine Learning},
  year      = {2026}
}

Poster: East Exhibition Hall, ICML 2026, Seoul. Come argue with us about rungs.