Publications
Identifiability and interpretability
-
Causality is Key for Interpretability Claims to Generalise
Shruti Joshi, Aaron Mueller, David Klindt, Wieland Brendel, Patrik Reizinger, Dhanya Sridhar
Interpretability studies draw counterfactual conclusions from interventional experiments. Pearl’s hierarchy says which claims a given experiment licenses, and causal representation learning says which variables can be recovered from activations at all.
Interpretability research on large language models (LLMs) has yielded important insights into model behaviour, yet recurring pitfalls persist: findings that do not generalise, and causal interpretations that outrun the evidence. Our position is that causal inference specifies what constitutes a valid mapping from model activations to invariant high-level structures, the data or assumptions needed to achieve it, and the inferences it can support. Specifically, Pearl's causal hierarchy clarifies what an interpretability study can justify. Observations establish associations between model behaviour and internal components. Interventions (e.g., ablations or activation patching) support claims how these edits affect a behavioural metric (eg, average change in token probabilities) over a set of prompts. However, counterfactual claims -- i.e., asking what the model output would have been for the same prompt under an unobserved intervention -- remain largely unverifiable without controlled supervision. We show how causal representation learning (CRL) operationalises this hierarchy, specifying which variables are recoverable from activations and under what assumptions. Together, these motivate a diagnostic framework that helps practitioners select methods and evaluations matching claims to evidence such that findings generalise.
@inproceedings{ joshi2026position, title={Position: Causality Is Key for Interpretability Claims to Generalise}, author={Shruti Joshi and Aaron Mueller and David Klindt and Wieland Brendel and Patrik Reizinger and Dhanya Sridhar}, booktitle={Forty-third International Conference on Machine Learning Position Paper Track}, year={2026}, url={https://openreview.net/forum?id=6l7cKIR6nf} }, } -
Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation
Shruti Joshi*, Vitória Barin Pacela*, Isabela Camacho, Simon Lacoste-Julien, David Klindt
Sparse autoencoders replace per-sample sparse inference with a single learned encoder. We show that the resulting amortisation gap survives more training data, and that it accounts for the failures of these methods on concept combinations held out from training.
The linear representation hypothesis states that neural network activations encode high-level concepts as linear mixtures. However, under superposition, when the number of concepts exceeds the activation dimension, recovering underlying latent factors from the activations requires sparse nonlinear inference, making methods such as linear probes insufficient. In this setting, classical sparse coding methods with per-sample iterative inference leverage compressed sensing guarantees to recover latent factors. Sparse autoencoders (SAEs), on the other hand, perform nonlinear inference, but amortise it into a fixed encoder, introducing a systematic amortisation gap. We show this gap persists as the number of training samples is increased, causing SAEs to fail under out-of-distribution (OOD) compositional shifts. Our results demonstrate that the recent OOD failures of SAEs can be attributed to amortisation failures: per-sample inference at test time substantially improves OOD performance, even when using a dictionary learned by an SAE.
@InProceedings{pmlr-v337-barin-pacela26a, title = {Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalization}, author = {Barin-Pacela, Vit'{o}ria and Joshi, Shruti and Camacho, Isabela and Lacoste-Julien, Simon and Klindt, David}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {364--412}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/barin-pacela26a/barin-pacela26a.pdf}, url = {https://proceedings.mlr.press/v337/barin-pacela26a.html}, abstract = {Foundational to interpreting pretrained representations of deep generative models, the linear representation hypothesis states that neural network activations encode high-level concepts as linear mixtures. However, linear representation does not imply linear accessibility of such concepts: under superposition, when the number of concepts exceeds the activation dimension, recovering the underlying latent factors requires sparse nonlinear inference, making methods such as linear probes insufficient. Sparse autoencoders ({SAEs}) perform nonlinear inference but amortize it into a fixed encoder, introducing a systematic amortization gap. We show this gap dominates all other error sources and persists as the number of training samples is increased, causing {SAEs} to fail under out-of-distribution ({OOD}) compositional shifts. In contrast, classical sparse coding with per-sample iterative inference leverages compressed sensing guarantees to recover latent factors robustly, maintaining near-zero gaps in the accuracy between in and out of distribution. Our results demonstrate that the recent {OOD} failures of {SAEs} can be attributed to amortization failures: per-sample inference at test time substantially improves {OOD} performance, even when using a dictionary learned by an {SAE}. This is observed along a spectrum of hybrid approaches that progressively undo amortization and recover {OOD} performance.} } , } -
Who Guards the Guardians? The Challenges of Evaluating Identifiability of Learned Representations
Shruti Joshi, Théo Saulus, Wieland Brendel, Philippe Brouillard, Dhanya Sridhar, Patrik Reizinger
MCC, R² and DCI are the metrics used to certify that a representation has been identified. Each one encodes assumptions about the data-generating process and about the encoder, and outside those assumptions it reports success and failure for the wrong reasons.
Identifiability in representation learning is commonly evaluated using standard metrics (e.g., MCC, R², DCI) on synthetic benchmarks with known ground-truth factors. These metrics are assumed to reflect recovery up to the equivalence class guaranteed by identifiability theory. We show that this assumption holds only under specific structural conditions: each metric implicitly encodes assumptions about both the data-generating process (DGP) and the encoder. When these assumptions are violated, metrics become misspecified and can produce systematic false positives and false negatives. Such failures occur both within classical identifiability regimes and in post-hoc settings where identifiability is most needed. We introduce a taxonomy separating DGP assumptions from encoder geometry, use it to characterize the validity domains of existing metrics, and release an evaluation suite for reproducible stress testing and comparison.
@InProceedings{pmlr-v337-joshi26a, title = {Who Guards the Guardians? {The} Challenges of Evaluating Identifiability of Learned Representations}, author = {Joshi, Shruti and Saulus, Th'{e}o and Brendel, Wieland and Brouillard, Philippe and Sridhar, Dhanya and Reizinger, Patrik}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {2618--2660}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/joshi26a/joshi26a.pdf}, url = {https://proceedings.mlr.press/v337/joshi26a.html}, abstract = {Identifiability in representation learning is commonly evaluated using standard metrics (e.g., *MCC, $R^2$, DCI*) on synthetic benchmarks with known ground-truth factors. These metrics are assumed to reflect recovery up to the equivalence class guaranteed by identifiability theory. We show that this assumption holds only under specific structural conditions: each metric implicitly encodes assumptions about both the data-generating process ({DGP}) and the encoder. When these assumptions are violated, metrics become misspecified and can produce systematic false positives and false negatives. Such failures occur both within classical identifiability regimes and in post-hoc settings where identifiability is most needed. We introduce a taxonomy separating {DGP} assumptions from encoder geometry, use it to characterise the validity domains of existing metrics, and release an evaluation suite for reproducible stress testing and comparison.} } , } -
Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations
Shruti Joshi, Andrea Dittadi, Sébastien Lachapelle, Dhanya Sridhar
Sparse codes over differences between embeddings are identifiable from pairs of observations that vary in several unknown concepts. This yields steering of one concept at a time without supervised contrastive data.
Steering methods manipulate the representations of large language models (LLMs) to induce responses that have desired properties, e.g., truthfulness, offering a promising approach for LLM alignment without the need for fine-tuning. Traditionally, steering has relied on supervision, such as from contrastive pairs of prompts that vary in a single target concept, which is costly to obtain and limits the speed of steering research. An appealing alternative is to use unsupervised approaches such as sparse autoencoders (SAEs) to map LLM embeddings to sparse representations that capture human-interpretable concepts. However, without further assumptions, SAEs may not be identifiable: they could learn latent dimensions that entangle multiple concepts, leading to unintentional steering of unrelated properties. We introduce Sparse Shift Autoencoders (SSAEs) that instead map the differences between embeddings to sparse representations. Crucially, we show that SSAEs are identifiable from paired observations that vary in multiple unknown concepts, leading to accurate steering of single concepts without the need for supervision. We empirically demonstrate accurate steering across semi-synthetic and real-world language datasets using Llama-3.1 embeddings.
@misc{joshi2026sparseshiftautoencodersidentifying, title={Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations}, author={Shruti Joshi and Andrea Dittadi and Sébastien Lachapelle and Dhanya Sridhar}, year={2026}, eprint={2502.12179}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2502.12179}, } -
From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?
Aaron Mueller, Andrew Lee, Shruti Joshi, Ekdeep Singh Lubana, Dhanya Sridhar, Patrik Reizinger
Concept representations are usually evaluated one concept at a time, under an implicit assumption that the concepts are independent. Holding the correlations between concepts under control, sparse autoencoder features affect many concepts at once when steered.
A central goal of interpretability is to recover representations of causally relevant concepts from the activations of neural networks. The quality of these concept representations is typically evaluated in isolation, and under implicit independence assumptions that may not hold in practice. Thus, it is unclear whether common featurization methods - including sparse autoencoders (SAEs) and sparse probes - recover disentangled representations of these concepts. This study proposes a multi-concept evaluation setting where we control the correlations between textual concepts, such as sentiment, domain, and tense, and analyze performance under increasing correlations between them. We first evaluate the extent to which featurizers can learn disentangled representations of each concept under increasing correlational strengths. We observe a one-to-many relationship from concepts to features: features correspond to no more than one concept, but concepts are distributed across many features. Then, we perform steering experiments, measuring whether each concept is independently manipulable. Even when trained on uniform distributions of concepts, SAE features generally affect many concepts when steered, indicating that they are neither selective nor independent; nonetheless, features affect disjoint subspaces. These results suggest that correlational metrics for measuring disentanglement are generally not sufficient for establishing independence when steering, and that affecting disjoint subspaces is not sufficient for concept selectivity. These results underscore the importance of compositional evaluations in interpretability research.
@inproceedings{mueller-etal-2026-isolation, title = "From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?", author = "Mueller, Aaron and Lee, Andrew and Joshi, Shruti and Lubana, Ekdeep Singh and Sridhar, Dhanya and Reizinger, Patrik", editor = "Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David", booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)", month = jul, year = "2026", address = "San Diego, California, United States", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2026.acl-long.782/", doi = "10.18653/v1/2026.acl-long.782", pages = "17188--17210", ISBN = "979-8-89176-390-6", abstract = "A goal of interpretability is to recover disentangled representations of latent concepts (features) from the activations of neural networks. The quality of features is typically evaluated in isolation, and under implicit independence assumptions that may not hold in practice. Thus, it is unclear to what extent common featurization methods such as sparse autoencoders (SAEs) and probes disentangle one concept from another. We propose a multi-concept evaluation setting using concepts such as sentiment, domain, voice, and tense. We evaluate how well featurizers produce disentangled representations of each concept, observing that features are typically sensitive to only one concept, but also that concepts are distributed across many features. Then, we steer these features, measuring whether each concept is independently manipulable, and whether features interact. Even in idealized settings, steering a feature often affects many concepts, despite a near absence of interaction effects. These results suggest that correlational metrics are insufficient to establish steering selectivity, and that demonstrating that two features operate in separate spaces is insufficient to claim that they will be selective for one concept. These results underscore the importance of multi-concept evaluations in interpretability research." }
Earlier work
-
Learning robust dynamics through variational sparse gating
Arnav Kumar Jain, Shivakanth Sujit, Shruti Joshi, Vincent Michalski, Danijar Hafner, Samira Ebrahimi Kahou
Learning world models from their sensory inputs enables agents to plan for actions by imagining their future outcomes. World models have previously been shown to improve sample-efficiency in simulated environments with few objects, but have not yet been applied successfully to environments with many objects. In environments with many objects, often only a small number of them are moving or interacting at the same time. In this paper, we investigate integrating this inductive bias of sparse interactions into the latent dynamics of world models trained from pixels. First, we introduce Variational Sparse Gating (VSG), a latent dynamics model that updates its feature dimensions sparsely through stochastic binary gates. Moreover, we propose a simplified architecture Simple Variational Sparse Gating (SVSG) that removes the deterministic pathway of previous models, resulting in a fully stochastic transition function that leverages the VSG mechanism. We evaluate the two model architectures in the BringBackShapes (BBS) environment that features a large number of moving objects and partial observability, demonstrating clear improvements over prior models.
@article{jain2022learning, title={Learning robust dynamics through variational sparse gating}, author={Jain, Arnav Kumar and Sujit, Shivakanth and Joshi, Shruti and Michalski, Vincent and Hafner, Danijar and Ebrahimi Kahou, Samira}, journal={Advances in neural information processing systems}, volume={35}, pages={1612--1626}, year={2022} } -
Function contrastive learning of transferable meta-representations
Muhammad Waleed Gondal, Shruti Joshi, Nasim Rahaman, Stefan Bauer, Manuel Wüthrich, Bernhard Schölkopf
Meta-learning algorithms adapt quickly to new tasks that are drawn from the same task distribution as the training tasks. The mechanism leading to fast adaptation is the conditioning of a downstream predictive model on the inferred representation of the task's underlying data generative process, or function. This meta-representation, which is computed from a few observed examples of the underlying function, is learned jointly with the predictive model. In this work, we study the implications of this joint training on the transferability of the meta-representations. Our goal is to learn meta-representations that are robust to noise in the data and facilitate solving a wide range of downstream tasks that share the same underlying functions. To this end, we propose a decoupled encoder-decoder approach to supervised meta-learning, where the encoder is trained with a contrastive objective to find a good representation of the underlying function. In particular, our training scheme is driven by the self-supervision signal indicating whether two sets of examples stem from the same function. Our experiments on a number of synthetic and real-world datasets show that the representations we obtain outperform strong baselines in terms of downstream performance and noise robustness, even when these baselines are trained in an end-to-end manner.
@inproceedings{gondal2021function, title={Function contrastive learning of transferable meta-representations}, author={Gondal, Muhammad Waleed and Joshi, Shruti and Rahaman, Nasim and Bauer, Stefan and Wuthrich, Manuel and Sch{\"o}lkopf, Bernhard}, booktitle={International Conference on Machine Learning}, pages={3755--3765}, year={2021}, organization={PMLR} } -
Dynamic inference with neural interpreters
Nasim Rahaman, Muhammad Waleed Gondal, Shruti Joshi, Peter Gehler, Yoshua Bengio, Francesco Locatello, Bernhard Schölkopf
Modern neural network architectures can leverage large amounts of data to generalize well within the training distribution. However, they are less capable of systematic generalization to data drawn from unseen but related distributions, a feat that is hypothesized to require compositional reasoning and reuse of knowledge. In this work, we present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules, which we call functions. Inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. The proposed architecture can flexibly compose computation along width and depth, and lends itself well to capacity extension after training. To demonstrate the versatility of Neural Interpreters, we evaluate it in two distinct settings: image classification and visual abstract reasoning on Raven Progressive Matrices. In the former, we show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner. In the latter, we find that Neural Interpreters are competitive with respect to the state-of-the-art in terms of systematic generalization.
@article{rahaman2021dynamic, title={Dynamic inference with neural interpreters}, author={Rahaman, Nasim and Gondal, Muhammad Waleed and Joshi, Shruti and Gehler, Peter and Bengio, Yoshua and Locatello, Francesco and Sch{\"o}lkopf, Bernhard}, journal={Advances in Neural Information Processing Systems}, volume={34}, pages={10985--10998}, year={2021} } -
Online utility-optimal trajectory design for time-varying ocean environments
Mohan Krishna Nutalapati, Shruti Joshi, Ketan Rajawat
This paper considers the problem of online optimal trajectory design under time-varying environments. Of particular interest is the design of energy-efficient trajectories under strong and uncertain disturbances in ocean environments and time-varying goal location. We formulate the problem within the constrained online convex optimization formalism, and a modified online gradient descent algorithm is motivated. The mobility constraints are met using a carefully chosen step-size, and the proposed algorithm is shown to incur sublinear regret. Different from the state-of-the-art algorithms that entail planning and re-planning the full trajectory using forecast data at each time instant, the proposed algorithm is entirely online and relies mostly on the current ocean velocity measurements at the vehicle locations. The trade-off between excess delay incurred in reaching the goal and the overall energy consumption is examined via numerical tests carried out on real data obtained from the regional ocean modelling system. As compared to the state-of-the-art algorithms, the proposed algorithm is not only energy-efficient but also several orders of magnitude computationally efficient.
@inproceedings{nutalapati2019online, title={Online utility-optimal trajectory design for time-varying ocean environments}, author={Nutalapati, Mohan Krishna and Joshi, Shruti and Rajawat, Ketan}, booktitle={2019 International Conference on Robotics and Automation (ICRA)}, pages={6853--6859}, year={2019}, organization={IEEE} }
Software
-
TriFinger Simulation
Shruti Joshi, Felix Widmaier, Vaibhav Agrawal, Manuel Wüthrich
The simulation package for the TriFinger platform, used for the Real Robot Challenge and maintained by the Open Dynamic Robot Initiative.
@misc{trifinger-simulation, author = {Joshi, Shruti and Widmaier, Felix and Agrawal, Vaibhav and W{\"u}thrich, Manuel}, title = {TriFinger Simulation}, year = {2020}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/open-dynamic-robot-initiative/trifinger_simulation}} }