This is an automated archive made by the Lemmit Bot.
The original was posted on /r/machinelearning by /u/jamesvoltage on 2025-06-06 12:55:36+00:00.
https://arxiv.org/abs/2505.24293
https://github.com/jamesgolden1/llms-are-llms
Hello all, I'd like to share my new research describing an alternative approach to LLM interpretability. I show that transformer decoder LLMs can be made locally linear at inference time without changing outputs or weights.
Result: LLMs can be converted into nearly exactly equivalent linear systems that reconstruct the next-token output for any given input text sequence. Instead of 25+ layers of nonlinear computations, this method computes a single set of matrix multiplications that linearly operates on the input embedding vectors and nearly exactly reconstructs the output embedding for a single token prediction.
Method: A "linear path" through the transformer is identified, the nonlinear components are detached from the gradient, and the Jacobian with respect to the input embeddings is computed. This yields the "detached Jacobian", which is the set of matrices that operate linearly on input embeddings to reproduce the predicted output embedding with ~10⁻⁶ error for float32 models.
Interpretability: This method provides nearly-exact token attribution rather than approximate attention weights - tools from linear algebra like the SVD are used to understand which concepts drive predictions
Scope: Works across Qwen 3, Gemma 3, Llama 3, Phi 4, Ministral and OLMo 2 (tested up to 70B parameters at q4).
Practical: The method works on free Colab T4 instances for Gemma 3 4B and Llama 3.2 3B models.
Concept steering: Preliminary results are shown for using the detached Jacobian as a linear conceptual steering operator in mid to late layers for guided generation of 8B models.
Trade-offs and costs: The detached Jacobian linear system is only valid for that specific input sequence (and must be computed from scratch for each new sequence). This is slow (10 sec to compute the Jacobian for Llama 3.2 3B on a T4, up to minutes for models > 30B parameters), VRAM intensive and currently limited to very short sequences, but I plan to continue working on this aspect.
Applications: In addition to steering, there is some potential for safety analysis (bias detection, deceptive content).
Background: This extends prior work on adaptive linear networks (Mohan, Khadkhodaie, Simoncelli et al.) and locally linear image diffusion models (Khadkhodaie, Simoncelli, et al.) to transformer decoder architectures, building on decoder circuit analysis (Elhage Nanda Olsson et al).
Abstract
We demonstrate that the inference operations of several open-weight large language models (LLMs) can be mapped to an exactly equivalent linear system for an input sequence without modifying the model weights or altering output predictions. Extending techniques from image diffusion models that exhibit local or piecewise linearity, we strategically alter the gradient computation with respect to a given input sequence for a next-token prediction such that the Jacobian of the model nearly exactly reproduces the forward prediction with a linear system. We demonstrate this approach across models (Llama 3, Gemma 3, Qwen 3, Phi 4, Mistral Ministral and OLMo 2, up to Llama 3.3 70B Q4) and show through the singular value decomposition of the detached Jacobian that these LLMs operate in extremely low-dimensional subspaces where many of the largest singular vectors decode to concepts related to the most-likely output token. This approach also allows us to examine the operation of each successive layer (and its attention and MLP components) as nearly-exact linear systems and observe the emergence of semantic concepts. Additionally, we present preliminary results on the detached Jacobian as a steering operator for inserting concepts into inference responses. Despite their expressive power and global nonlinearity, modern LLMs can be interpreted through nearly-exact locally linear decompositions that provide insights into their internal representations and reveal interpretable semantic structures in the next-token prediction process.
