Interpretability for Improving Retrieval

Anja uses mechanistic interpretability as a tool to study information retrieval models, with the goal of improving them. Anja presented an analysis of a language model–based retrieval approach called Generative Retrieval, in which the model generates document identifiers instead of relying on an external document index. She demonstrated how these models perform retrieval, the extent to which they pre-plan when generating a sequence of tokens, and the roles MLPs and self-attention play in knowledge lookup. Finally, she showcased how these insights can be applied to model editing of generative retrieval models, making them more applicable to real-world use cases.

Verbalizable Representations Form a Global Workspace in Language Models

Out of everything the human brain processes, only a small fraction is consciously accessible — available for verbal report, deliberate control, and flexible reasoning. Gurnee et al. present evidence that an analogous distinction has emerged in large language models, introducing the Jacobian lens, an interpretability technique that identifies the representations a model is poised to verbalize at any point in its processing. These representations — collectively the J-space — can be reported, deliberately summoned and held, used to carry the intermediate steps of silent reasoning, and passed as arguments to arbitrary downstream computations, while automatic processing such as text parsing proceeds without them; they also show the structural signatures global workspace theory associates with conscious access, carrying coherent content only in an intermediate band of layers, holding on the order of tens of concepts at a time, and being broadcast more widely than other representations. In alignment audits this makes the J-space a practical window into a model's unspoken thinking, revealing strategic deliberation, evaluation awareness, and trained-in misaligned dispositions that never appear in the model's outputs.

How Feature Learning Discovers Sparsity: Width and Scaling Laws in Quadratic Neural Networks

Feature learning allows an overparameterized neural network to adapt its representation to the structure of the task, effectively reducing the number of parameters that need to be learned. Emanuele discussed that mechanism in a simple two-layer quadratic neural network, where the learned representations can be characterized exactly. The model displays a rich learning phenomenology, and the emergence of task-relevant features can be tracked directly in the spectrum of the learned weights, where informative directions separate from a noisy background — a concrete way to connect the structure of the learned representation to generalization. The interplay between informative spectral features and the noise bulk produces distinct learning regimes and allows analytical generalization scaling laws to be derived. In the high-dimensional limit the problem becomes simple enough to solve while retaining this phenomenology, making it possible to study in detail how increasing the width changes the learned representation and the resulting generalization behavior.

Interpretability of Protein Language Models: The State of the Art, Open Problems, and a New Window into the Machines of Life

Protein language models (pLMs) such as the ESM family have been instrumental in developing new methods to study proteins and predict their properties, yet interpreting their operations remains a consistent challenge. Interpretability of pLMs would open the door to a new language for describing protein structure and function, a new lens into protein evolution, and mechanistic insights that could pave the way to new drugs for protein-associated diseases. Shivam surveyed the state of pLM interpretability through techniques such as sparse autoencoders and cross-layer transcoders, then turned to open problems current methods have not solved — in particular interpreting mutations and genetic interactions (epistasis) within proteins — along with hypothesized approaches for addressing them. The discussion also extended to interpretability of diffusion models of protein structure, given their increasing use for tasks such as structure prediction via AlphaFold3.

Evaluating Chain-of-Thought Faithfulness and Monitorability

Chains of thought (CoTs) have become central in interpreting and auditing behaviors of large language models. Yet growing evidence suggests that these traces often fail to faithfully represent the computations behind a model's predictions. Several faithfulness metrics have been proposed, but whether they indeed measure faithfulness remains unknown. Answering this requires ground-truth labels, which are hard to obtain since internal computations are not directly observable. Yoav discussed efforts to address these problems by defining faithfulness in a more tractable way and establishing ground truth in multiple settings, enabling evaluation of faithfulness metrics and monitors.

Vision or Language? On Modality Roles in Multimodal Models

When a multimodal model performs a complex behavior, is the key computation happening in the vision or the language component? Modern multimodal models combine vision and language through increasingly unified architectures, yet it remains unclear what parts of their underlying computations are carried out by each modality. Tamar will present three studies that collectively ask: which components contribute to different aspects of multimodal computation? Across vision-language reasoning in VLMs, text-to-image generation, and in-context image editing in diffusion models, she shows that key multimodal behaviors often emerge from components not traditionally associated with them, and that identifying these mechanisms can, in some cases, enable targeted improvements of model performance.

Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability

Mechanistic interpretability seeks to explain the internal computations of neural networks through faithful, human-understandable models, yet it has lacked a unified theoretical foundation. This presentation discusses a recent paper proposing causal abstraction as that foundation. Geiger et al. generalize causal abstraction beyond intervention-based accounts to arbitrary transformations of internal mechanisms, providing a broader framework for relating low-level neural computations to higher-level explanatory models. They also offer formal definitions of central concepts in mechanistic interpretability — including polysemantic neurons, modular features, the linear representation hypothesis, and graded faithfulness — and demonstrate how a wide range of interpretability methods, from activation patching and causal tracing to sparse autoencoders and circuit analysis, can be understood within a common causal framework.

Using Interpretability to Identify a Novel Class of Alzheimer's Biomarkers

Researchers trained an AI model (Pleiades) to detect Alzheimer's disease from blood samples, then used interpretability techniques to understand what drove its decisions. They found DNA fragment length patterns dominate its decision-making. Distilling these insights into a simpler fragmentomics-based classifier (cell-free DNA fragmentation patterns) yielded better generalization to independent test cohorts than prior biomarker approaches — achieving AUROC 0.78 standalone, 0.84 combined with methylation features.

Tomer Ashuach

Tomer uses interpretability to uncover the internal mechanisms of LLMs — specifically how knowledge is acquired, represented, and edited. His talk covers two angles: (1) unlearning in LLMs via REVS (ACL Findings 2025), which locates neurons storing memorized sensitive information and surgically removes it, and CRISP (ACL 2026), which performs concept unlearning via Sparse Autoencoders; and (2) Privileged Knowledge in LMs (ACL 2026), asking whether models carry an internal signal about their own correctness that no external observer can retrieve.

Michael Toker

Michael's research focuses on mechanistic interpretability of multi-modal models. He presented Diffusion Lens (ACL 2024), Padding Tone (NAACL 2025), and Follow the Flow (ACL 2026) — methods that reveal what text-to-image models learn and how computations are performed internally, with applications to solving problems like semantic leakage.

Old Habits and Hard Choices: How LLMs Navigate Management Decision-Making and Earlier Interactions

We will start with evaluating what happens in LLM decision-making in realistic, human-validated managerial scenarios via ManagerBench, indicating that frontier LLMs perform poorly when navigating the safety-pragmatism trade-off. We next turn to a multi-turn setting to investigate the "carryover-effect" — the phenomenon in which, for example, hallucinations in earlier interactions shape subsequent model responses. To study this, we developed HISTORY-ECHOES, a framework for analyzing how conversational history biases later generations. The framework approaches this from two angles: a probabilistic view, modeling conversations as Markov chains to quantify state consistency, and a geometric view, measuring the consistency of consecutive hidden representations — indicating a correlation between the two angles.

Modality-aware methods for interpretability

Interpretability promises a lever to control models, monitor their behavior, and learn from them. As such, we often imagine finding features in models such as "honesty," "conciseness," or "speaking in French." However, state-of-the-art methods find features that are local, syntactic, and uninteresting (e.g., "the phrase 'The' at the start of a sentence"), a critical gap between the proposed utility of interpretability and reality. More precisely, while we care about semantic features in models, we instead recover syntactic ones. Drawing on theory from computational linguistics, we propose a simple model of language production: semantic features are globally smooth across time, whereas syntactic features are local. We use this model to propose Temporal Sparse Autoencoders (T-SAEs), which employ a temporal contrastive loss to isolate smooth features in LLM representations. Importantly, T-SAEs recover semantic information despite being trained with only a self-supervised context-similarity objective, and successfully disentangle semantic and syntactic features. We present experiments demonstrating T-SAEs' ability to track semantic shifts over text sequences and applications to safety monitoring and steering. Finally, I will discuss ongoing extensions to vision models and alternative invariances suited for visual semantics, including opportunities to understand visual processing and predict neural activity.

Layer-wise efficient coding in early olfactory processing

The architecture of early olfactory processing is a striking example of convergent evolution. Typically, a panel of broadly tuned receptors is selectively expressed in sensory neurons (each neuron expressing only one receptor), and each glomerulus receives projections from just one neuron type. Taken together, these three motifs—broad receptors, selective expression, and glomerular convergence—constitute "canonical olfaction," since a number of model organisms including mice and flies exhibit these features. The emergence of this distinctive architecture across evolutionary lineages suggests that it may be optimized for information processing, an idea known as efficient coding. In this talk, I explain how, by maximizing mutual information one layer at a time, efficient coding recovers several features of canonical olfactory processing under realistic biophysical assumptions. In the second part of the talk, I will speculate about what other aspects of olfactory processing might be understood using mutual information.

How Optimality Structures Sparse Dictionaries: A Theory for Understanding SAE Representations

Sparse Autoencoders (SAEs) have found widespread success parsing neural representations into interpretable concepts, providing a basis for understanding and control. However, what exactly an SAE extracts, and, correspondingly, the scientific conclusions we can draw, is not obvious. Empirically, the proof is in the pudding: SAEs do learn interpretable features. Theoretically, we lack a clear account of what properties a 'concept' must satisfy for an SAE to extract it. There is an extensive body of work studying sparse coding identifiability; in particular, given data generated under sparsity assumptions, when will an algorithm recover the true factors? However, SAEs are trained on internet-swallowing representations that are poorly approximated by simple generative models. Rather than assuming a hypothesised ground truth, we ask what properties any dictionary learning optimum must satisfy without data-assumptions. Concretely, we extend existing local optimality analyses to the nonnegative joint-optimisation problem that vanilla SAEs approximate, and derive constraints relating optimal SAE features to their distributions. We use these to explain a range of observed SAE behaviours - hierarchical splitting & absorption, the structure of residuals, and dense antipodal features - each reflecting how L1+nonnegativity interact with data to structure optimal dictionaries. Further, we identify a novel convex formulation of the problem, and use it to ask: will larger SAEs ever stop splitting? We find the answer can be yes, with a limiting dictionary state that clusters data along rays. In sum, we hope this framework can tease model assumptions from unexpected observations, letting us learn more from SAEs' successes.

Arithmetic in the Wild: Llama uses Addition to Reason About Cyclic Concepts

Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is three months after November?"). Even though Llama-3.1-8B has circular representations for these concepts, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using non-modular addition (three + November = 14). Then, it maps back to cyclic concept space (14 → February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums—in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12). Furthermore, we identify a sparse, reusable set of 28 MLP neurons (~0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters that each compute the sum for a Fourier feature with a different period.

Unveiling Value Functions in Social Cognition with Multi-Agent Inverse Reinforcement Learning

This paper introduces MAIRL, a multi-agent inverse reinforcement learning framework that recovers interpretable, decomposed value maps from behavior, revealing how latent value functions conditioned on social roles govern multi-agent interactions across species.

Symmetry in Language Statistics Shapes the Geometry of Model Representations

This paper shows that statistical symmetries inherent in natural language directly induce geometric structure in the representation spaces of language models.

Multilevel Interpretability of Artificial Neural Networks: Leveraging Framework and Methods From Neuroscience

This paper proposes a multilevel interpretability framework for ANNs by adapting concepts and methods from neuroscience to systematically analyze representations across scales.

Large Language Models Generate Harmful Content Using a Distinct, Unified Mechanism

A presentation on how harmful behavior in LLMs relies on a compact set of internal weights, helping explain why safety safeguards are brittle and why narrow fine-tuning can trigger broad misalignment.

Decomposing Query-Key Feature Interactions Using Contrastive Covariances

A presentation on decomposing the query-key space in Transformer attention heads into low-rank, human-interpretable components — and attributing attention scores to identified semantic features.

Model Circuits Interpretability, and the Road to Scale It Up

A tour of circuit analysis for neural network interpretability — from gaining scientific insights into arithmetic reasoning in LLMs to diagnosing performance gaps in VLMs on visual tasks, with a look toward scaling circuit analysis to complex real-world behaviors.

Shared Global and Local Geometry of Language Model Embeddings

This paper reveals systematic geometric similarities across the token embeddings of large language models — both in global relative orientations and local intrinsic structure — and introduces EMB2EMB to linearly transfer steering vectors between models.

Geometric Approaches to Neural Network Training and MLLM Fine-tuning

A tour of geometric and manifold-based approaches to parameter-efficient fine-tuning, tracing how orthogonality, constrained optimization, and algebraic structure can reduce catastrophic forgetting and improve efficiency beyond vanilla LoRA.

Model-Optimized Stimuli for Comparing Brain-Alignment of Encoding and Generative Models

A comparison of closed-loop stimulus synthesis approaches — generative models vs. encoding model-based feature accentuation — as stringent tests of brain–model alignment across the ventral visual stream.

Interpretable Anomaly Detection for Scientific Discovery, Open-World Novelty Detection, and Generative Model Validation

This work introduces SparKer, a sparse ensemble of Gaussian kernels grounded in a semi-supervised Neyman–Pearson framework, designed to detect and interpret anomalies in high-dimensional representation spaces with sparsity, locality, and competitive allocation.

Toy Models of Superposition

The inaugural PRISM session: a presentation of the foundational "Toy Models of Superposition" paper, exploring how neural networks represent more features than they have dimensions — and what this reveals about the geometry of representations.