Research

In statistical machine learning, identifiability results specify what can be recovered under assumptions about the data and the model. On synthetic benchmarks, recovery is evaluated against known latent factors. I study whether the metrics used for that comparison measure the property the theory guarantees, by formalising settings with unknown ground-truth generative factors.

In Who Guards the Guardians? (UAI 2026), we characterise the assumptions that common identifiability metrics make about both the latent factors and the encoder. Using analytical results and controlled synthetic encoders, we distinguish structural misspecification from finite-sample effects and provide a reproducible evaluation suite.

This work studies the consequence of identification without a ground-truth in evaluation. This is important to study as a first step in understanding the scope of current tools when we seek to apply them to interpreting pretrained models, or use them for mechanism discovery.

I am formalising equivalent representations relative to responses to interventions. Two candidate abstractions are equivalent when they imply the same observable outcome distributions under every intervention in that set. The target is therefore defined by what the learner can distinguish through interaction, rather than by a prescribed set of latent coordinates.

Identifiability is an asymptotic guarantee, and thus its finite-sample achieavability is a separate question. I study the gap between what the evidence can identify and what a particular architecture and optimiser learn from finite data.

My work on Sparse Shift Autoencoders uses an explicit sparsity constraint rather than a fixed regularisation penalty, giving direct control over the sparsity budget. I am studying how such optimisation choices affect identification and hyperparameter transfer across datasets and model scales.

Using the identification criteria I am developing, I aim to quantify representation error and study how it varies with sample size, model width and optimisation, and whether these relationships admit scaling laws. For learners that collect their own data, I also ask how intervention choice changes these relationships. The target is a notion of representation quality, which may be complementary to generalisation error.

I think language models provide an excellent testbed for these questions: when does an extracted feature support a reliable intervention, rather than being merely predictive of an observed behaviour? In Causality is Key for Interpretability Claims to Generalise (ICML 2026), we use causal inference to specify what observations and interventions establish, and what additional assumptions counterfactual claims require. Identifying a feature does not, by itself, establish its causal role in the model's computation.

Sparse Shift Autoencoders learn sparse representations of differences between embeddings. Under sufficient conditions on the diversity of concept changes in these embeddings (which may result from different layers of an LLM, different input prompts, or different tokens of the same prompt), we prove identification of concept shifts up to permutation and rescaling from pairs that differ in multiple unknown concepts. This supports single-concept steering without requiring single-concept contrastive training pairs.

In two further studies, I explore where existing methods fall short in providing generalisable answers to interpreting an LLM's activations. Stop Probing, Start Coding (UAI 2026) examines failures on unseen concept combinations, showing that the gap between sparse autoencoders and classical sparse coding persists as training data increases. From Isolation to Entanglement (ACL 2026) shows that features that appear selective under correlational evaluation can affect several concepts when steered. These results motivate evaluating representations through the interventions they support, not through reconstruction or probing scores alone.