Decoding Concepts Under Superposition

A linear probe reaches 99% accuracy on your validation set. The concept it reads out is linearly encoded in the activations, by construction. When the model encounters a combination of concepts the probe never saw in training, the readout collapses to near chance.

Nothing malfunctioned. The failure is geometric, it is quantifiable in advance, and it follows from taking the linear representation hypothesis more seriously than the field usually does. This post walks through the argument of our UAI 2026 paper: why linear probes cannot work under superposition, why sparse autoencoders inherit a related failure through a different mechanism, and why the experiments end up indicting a component that neither story starts with.

Linear encoding without linear access

The linear representation hypothesis says that a model’s activations are linear mixtures of underlying concepts, $\mathbf{y} = \mathbf{W}\mathbf{z}$, where $\mathbf{z} \in \mathbb{R}^{d_z}$ collects the ground-truth latent concepts and $\mathbf{y} \in \mathbb{R}^{d_y}$ is the activation vector. The goal of interpretability, on this view, is to recover $\mathbf{z}$ from $\mathbf{y}$. Under superposition the model encodes more concepts than it has dimensions, $d_z > d_y$, so $\mathbf{W}$ projects a high-dimensional concept space into a lower-dimensional activation space. The system $\mathbf{y} = \mathbf{W}\mathbf{z}$ is then underdetermined. Each observation is consistent with infinitely many latent vectors, and there is no linear unmixing that maps activations back to concepts.

This is where a distinction that sounds pedantic becomes load-bearing. Concepts being linearly encoded in activations is a claim about the generative direction, from $\mathbf{z}$ to $\mathbf{y}$. Concepts being linearly accessible is a claim about the inverse direction, from $\mathbf{y}$ back to $\mathbf{z}$. The two coincide when $\mathbf{W}$ is invertible and come apart when it is not. The projection folds latent space, and a decision boundary that is linear in $\mathbf{z}$ need not remain linear in $\mathbf{y}$. The literature routinely conflates the weaker claim with the stronger one.

The demo below makes the fold visible. The label is linear in latent space, $t = \mathbb{1}\{z_1 > 0.5\}$, at most two concepts are active at a time, and the demo finds the best linear probe that exists by exhaustive search over directions and thresholds.

The sizes involved are worth pausing on, because compressed sensing gives the exact accounting. Recovering a $k$-sparse code by nonlinear methods requires $d_y = O(k \log(d_z/k))$ activation dimensions. Requiring the recovery to be linear, as a probe does, raises the requirement to $\Omega\big(\tfrac{k^2}{\log k}\log(\tfrac{d_z}{k})\big)$, which is quadratic in the number of simultaneously active concepts. For one million latent concepts with $k = 100$ active at once, nonlinear recovery is comfortable at roughly 920 dimensions, while linear recovery needs roughly 20,000. A transformer hidden size of 4,096 comfortably exceeds the first number and falls well short of the second. In that regime the concepts are present in the activations, and whether they are recoverable depends entirely on how you attempt to decode them.

The failure appears under compositional shift

In distribution, none of this geometry is visible. A probe is free to exploit whatever co-occurrence structure the training distribution offers, and its accuracy will look excellent. The problem surfaces when concepts recombine, which is precisely the situation interpretability tools are supposed to survive.

The construction is simple. Take three concepts. Train on samples where $z_1$ co-occurs with $z_2$ and where $z_2$ co-occurs with $z_3$, and hold out the combination $(z_1, z_3)$ for evaluation. The label depends only on $z_1$, so a decoder that actually isolates $z_1$ passes without noticing. A decoder that leans on the training co-occurrences does not. This is the withheld-co-occurrence logic of spurious-correlation benchmarks such as Waterbirds, applied to concepts rather than image backgrounds.

The probe was never reading $z_1$. It was reading a projection that aligned with $z_1$ under the training mixture, and the mixture changed. In the paper we show the same failure across each choice of target concept, including cases where the compression already damages in-distribution accuracy. Superposition is what turns a mild shortcut into a catastrophic one, because the folded geometry gives the probe no linearly accessible alternative to the shortcut.

What sparse coding guarantees

Classical sparse coding handles the fold by construction, and the theory is specific about what it buys. Two levels of identifiability matter here, and it is worth keeping them separate.

The first is code identifiability: given a known, well-behaved dictionary $\mathbf{W}$, can you uniquely recover the sparse code $\mathbf{h}$ from an observation $\mathbf{y}$? Sparsity restores uniqueness to an underdetermined system. If codes have at most $k$ nonzero entries and the dictionary satisfies the restricted isometry property, no two distinct $k$-sparse codes map to the same activation, and per-sample solvers, basis pursuit or ISTA and FISTA, provably converge to the unique solution. Random dictionaries satisfy the property with high probability once $d_y \geq O(k \ln(d_h/k))$. The bound is logarithmic in the dictionary size $d_h$, which makes the theory strikingly permissive about scale: with $d_y = 4{,}096$ and a few dozen concepts active at once, dictionaries with millions of atoms are within the feasible range. The 16 million features reported for GPT-4 and the 34 million for Claude 3 Sonnet are not the obstacle. The same bound can be rearranged into a ceiling on sparsity, roughly $k_{\max} \approx d_y / \ln(d_h/k)$, which for a 4,096-dimensional residual stream and a ten-million-atom dictionary allows on the order of 185 simultaneously active concepts. Beyond that, no inference procedure of any kind can guarantee unique recovery.

The second is dictionary identifiability: can you recover $\mathbf{W}$ itself from data? Classical results say yes when the data is sufficiently sparse and diverse, and the standard algorithm is alternating minimisation, estimating codes given the current dictionary and then updating the dictionary to reduce reconstruction error. Both levels matter for interpretability, because a method needs a good dictionary before per-sample guarantees mean anything. The experiments below pull these two levels apart.

Sparse autoencoders amortise the inference

Sparse autoencoders adopt the sparse coding model but replace per-sample optimisation with a fixed encoder that maps activations to codes in one forward pass. The encoder is fit on a finite training distribution, jointly with the dictionary. Nothing in that training constrains the encoder to implement sparse inference. It is free to learn any mapping that reconstructs the training data well, including mappings that exploit the training distribution’s co-occurrence statistics as shortcuts. The discrepancy between what the fixed encoder computes and what per-sample optimisation would compute is the amortisation gap, and the setting where it should widen is exactly the compositional shift above, where sparsity patterns change while the dictionary does not.

The demo below isolates the gap under the most favourable conditions we can grant the encoder. Both procedures use the same frozen ground-truth dictionary, so dictionary quality is not in play. The encoder is trained on in-distribution compositions. ISTA solves the lasso objective from scratch for each input.

The paper runs this comparison as a controlled sweep, and the pattern is consistent. Per-sample FISTA with the true dictionary exhibits the sharp phase transition that compressed sensing predicts: once the undersampling ratio $\delta = d_y/d_h$ passes a critical threshold, recovery jumps to near-perfect, and the empirical transition point matches the theoretical constant. SAE variants, ReLU, TopK, JumpReLU, and matching pursuit, also improve as $\delta$ grows, but they plateau at 0.2 to 0.5 MCC and never approach the transition, even deep inside the regime where compressed sensing guarantees exact recovery. Scaling the latent dimension does not change the conclusion. Neither does more data: the gap between per-sample and amortised methods is stable across two orders of magnitude of training samples, DL-FISTA jumps from 0.5 to 0.98 MCC once it has a thousand samples, and JumpReLU actually degrades as data increases, which suggests additional training settles it into a poor solution rather than correcting one. Across all of these sweeps, SAE codes offered no downstream advantage over probing raw activations, consistent with independent evaluations by Kantamneni et al.

Isolating the bottleneck

Everything so far points at amortisation, and it suggests an obvious fix: keep the SAE’s dictionary, discard its encoder, and run per-sample inference. That was our hypothesis, and it is wrong.

The experiment holds one component fixed while varying the other. Given a trained SAE, we compare three inference strategies on the same test input: the SAE encoder as trained, FISTA run from scratch on the frozen SAE decoder as a dictionary, and FISTA warm-started from the SAE encoder’s output. All three use the same dictionary, and all three land in the same place, around 0.15 to 0.4 MCC. Classical dictionary learning with per-sample inference, which learns its own dictionary on identical data, reaches 0.75 to 0.95. Swapping the encoder for an optimiser does not close the gap, and the result is robust to sweeping the regularisation strength across three orders of magnitude.

The reason is that the SAE-learned dictionary itself points in substantially wrong directions. At $d_z = 100$, classical alternating minimisation converges to dictionary columns with cosine similarity above 0.9 to ground truth, while SAE decoders plateau earlier. At $d_z = 5{,}000$, SAE support recovery is nearly destroyed: a ReLU SAE with true sparsity $k = 10$ activates around 120 of 5,000 features with support precision 0.009, and even TopK, which activates the correct number of features, selects almost entirely wrong atoms at precision 0.03. Re-estimating the magnitudes by least squares on the SAE’s chosen support makes things worse, which confirms that the support itself is incorrect. Per-sample inference cannot rescue wrong column directions, because the columns define which directions inference is allowed to use.

So the reframe the paper argues for: the SAE failure is a dictionary learning problem, and amortised inference is the visible symptom. The honesty required here runs in both directions. At $d_z = 5{,}000$, classical alternating minimisation begins to fail too, with cosine similarity around 0.25, and it converges to MCC comparable to the SAEs. No current method learns good dictionaries at that scale. The oracle results establish that the problem is solvable, since FISTA with the true dictionary is near-perfect at every scale we tested. No existing algorithm solves it at the sizes that matter for language models.

What the SAE decoder is still good for

Two findings temper the negative result. First, although the SAE decoder is a poor dictionary on its own, it encodes useful structure as an initialisation. Warm-starting dictionary learning from a TopK decoder begins at 0.87 OOD MCC and reaches 0.94 within five rounds of dictionary updates, where a random initialisation starts at 0.30 and needs around fifty rounds to catch up. For ReLU and JumpReLU the head start is smaller but consistent. Both initialisations converge to the same final quality, so the decoder biases the optimisation toward the right basin rather than trapping it in a wrong one. Warm-starting the inference from SAE codes, by contrast, buys little: the lasso objective is convex, so cold and warm starts reach the same solution given enough iterations.

Second, the classical recipe is already competitive on real language model activations, not only in synthetic sweeps. On the SAE Bench sparse-probing task over pythia-70m residual streams, dictionary learning with per-sample FISTA reaches 92.38% mean test accuracy against 92.21% for both SAEs and linear probes. On Gemma-2-2B, layer 12, it outperforms TopK SAEs on all eight datasets, by margins between 0.32 and 2.24 points and by 1.17 on average, and it edges out the raw-activation linear probe as well. Per-sample sparse inference on LLM activations is practical today, at the cost of an optimisation loop per input rather than a single forward pass.

What this changes

For practitioners, the immediate lessons are about where to place trust. In-distribution probe accuracy is an upper bound that compositional shift will revise downward, and the geometry guarantees that no validation set drawn from the training distribution can warn you. When a trustworthy dictionary is available, per-sample sparse inference dominates a one-pass encoder, and the compute cost is modest. When evaluating an SAE, evaluate its decoder as a dictionary, since column directions and support recovery are where the failure lives, not in the encoder’s approximation error.

For the field, the paper’s conclusion is a redirection of effort. The recent wave of negative results on SAEs is real, but it does not indict the superposition hypothesis or the sparse coding framework. Compressed sensing says unique recovery under superposition is theoretically cheap at LLM scale, with millions of concepts, a few dozen active at once, and hidden sizes that already exist. What is missing is a dictionary learning algorithm that remains accurate as the latent dimension grows past a few thousand. That is a concrete, well-posed open problem with a defined success criterion, and progress on it converts directly into the thing interpretability keeps promising: concept recovery that survives novel combinations of concepts.

This post is the geometric half of an argument whose inferential half is our ICML 2026 position paper. There we argue that decodability never licensed causal claims about model internals. Here, superposition shows that decodability itself degrades in quantifiable ways once compositions shift. Both papers point toward the same practice, which is stating and testing identifiability assumptions instead of extrapolating accuracy numbers past the distribution that produced them.

Cite

@inproceedings{barinpacela2026stopprobing,
  title     = {Stop Probing, Start Coding: Why Linear Probes and Sparse
               Autoencoders Fail at Compositional Generalisation},
  author    = {Barin-Pacela, Vit{\'o}ria and Joshi, Shruti and Camacho, Isabela
               and Lacoste-Julien, Simon and Klindt, David},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year      = {2026}
}