A big problem with some of these supervised* interpretability approaches is that they can find spurious structure. (There are lots of ways to make the model do what you want; which is roughly what Hewitt and Liang 2019 showed). This paper draws a contrast to a previous method, DAS (distributed alignment search) on page 20. These and related methods rest on theories of causal abstraction, which are great in theory, but harder in practice. DAS, for example, has faced numerous recent criticisms (Makelov 2024, Meloux 2025, Sutter 2025, Grant 2026, Kumon 2026). My favorite is the quite approachable Meloux et al.; Sutter 2025 is also really good, but relies on a sort of real number argument that allows a lossless encoding of every input.
My forthcoming paper at EMNLP offers an alternative that instead grounds the notion of representation in a very simple notion of the effect it has on model learning/behavior when you adversarially perturb it. For example, if I tell a model that in the context "I saw a duck quacking" it should replace 'duck' with 'glam', how much does it desire to replace 'duck' with 'glam' in "I need to duck out of the meeting" vs. "At the park a duck protected her ducklings." This method turns out to work quite well, and as we use only a single example, avoids the need for supervision.
The linked paper argues that their method, DISCOVER, is not supervised in the same way as DAS, since it does not directly optimize for causal effect. I have only skimmed this, but I am not so sure it might not suffer from a similar issue. They're still supervising to align representations with their underlying hypothesis, even if they don't directly supervise for causal outcomes.
Refs
- Hewitt and Liang 2019. Designing and interpreting probes with control tasks
- Kumon and Yanaka, 2026. Fine-grained analysis of shared syntactic mechanisms
- Meloux et al., 2025. Everything everywhere all at once
- Rozner and Shain 2026. Perturbation: A simple and efficient adversarial tracer for representation learning in LMs. https://arxiv.org/abs/2603.23821
- Sutter et al. 2025. The nonlinear representation dilemma
(1) they are claiming to produce apparently bijective closed-form symbolic representations/approximations of, among other things, LLMs. Is evaluating these closed-form representations more computationally efficient? The implications of that are potentially huge. It would be essentially analytic distillation. Fable on a chip and not a data center would be important — and disruptive - in many ways.
(2) Unsupervised, and even supervised, symbolic approaches to problem solving break down due to combinatorial explosion, among other things. This could potentially allow us to treat LLM training and inference as a search algorithm for novel symbolic approaches to solving new classes of complex problems hitherto unreachable through other approaches. If that works, I suspect it’s a feedback loop, too - the learnings from one representation push advances in the other. This would also increase the economic value of large training runs, since the model itself is now valuable, not just its inference.
(3) Per the above, can this push LLM design to greater capabilities?
The relationship between this and Anthropic’s J-space observation is also interesting. This is much, much deeper and more directly actionable, though.
EDIT: I ran my questions through Sonnet — yes, I appreciate the irony — and it was none too sanguine about questions (1) and (2), but thought (3) was reasonable. In any case, this is quite the paper. On reflection, I do think that the apparent reliance on very simple symbolic representations and tasks is underwhelming. But the approach is impressive. And obviously this is still early days, and the value of building a bridge between the very fuzzy LLM models and the rigorous, mechanically provable models would be enormous.
Inference is just tokens transformed through a fixed crystalline structure, no? You already could put that on a chip. There's no particular reason it couldn't be represented as some set of symbolic equations instead of a layered process... it's just another kind of quantization.
When symbolic algorithms are that large, they're equally hard to reason with most of the time. The upshot would be a lot more storage required in exchange for more generalized computing, lessening the need for so much GPU in a lot of cases.
I don't see why a model couldn't be represented that way. After all, if you just polled the output of a model, you could evolve genetic algorithms to predict it with fairly high accuracy in a limited domain. Take that out to the Nth degree and you're basically just unspooling the model into a giant set of equations.
As I am going through the article, I was wondering why is this more interesting than having the ability to recover java programs from byte code. So I asked copilot the same question. It told me that - "Honestly this is where the difference between an engineer and researcher shows up!" .
The math and core experimentation here is beyond my abilities, but what I think I understand is that there are possible deeper patterns of representation that exist in LLMs that are distillations of core conceptual relations in grammar that we can get our heads around in a mathematical sense rather than apparent layer-smeared noise that somehow, un-interpretably (in a meaningful sense), resolve to correct grammar/inferences.
That's pretty cool. I hope I've got that kinda-right.
I haven't read this in depth yet, though I plan to. If this general line of research is interesting to you, I'd recommend checking out some of the lines of research it touches upon--they're really rich and fascinating, and some are pretty approachable mathematically even if ML research papers aren't usually your thing. The related works section here seems pretty well stocked, but mechanistic interpretability is a pretty interesting peephole into this general vein: https://transformer-circuits.pub/
Symbolism has tried to strike back repeatedly ever since statistical learning revived with AlexNet. With all the due respect one can have for the names Smolensky and Linzen from the perspective of linguistics, the question about the applicability, generalizability and robustness of the method proposed here should be raised. It seems from section 3.5 of the paper that one cannot be so optimistic about it at least as yet. I get it that the method is still in its infancy, but we've already got the kind of Mech Interp as pushed forward by Neel Nanda and co, among other lines of research. Not that we are forced to make a choice between all interpretability works, or this TPR method is inherently inferior to the other ones, but we can be moderately cautious when looking at such progress.
Sounds reasonable... That the model is sometimes learning a lossy vector representation of something symbolic in nature... Sure, a NN can approximate a function?
They say this holds in... Some examples they found?
Imagine a box of balls. They have size, weight, colour, density… etc. These properties, each a measure, are dimensions and they are orthogonal to each other. Taken together are multi-dimensional.
Now take a set of words. They have "sizeness", "weightness", "colorness" and "densityness"...and "pythonness" and "haskellness" and even "adjectiveness" and "verbness" and so on and so forth...You can readily see that this can encode arbitrary patters (like language grammer or program syntax)
Training identifies these dimensions in the training data and links it with each word/token. Then given a stream of such tokens, each with its own set of dimensions (which can be huge), and LLM predicts the dimensions that the next token is most likely to have...
This is nonsense. The human mind cannot visualize more than 3 dimensions. It can perfectly comprehend any number of dimensions as long as they are represented in a vector space. In fact, that's what linear algebra does.
Yes. It's worth pointing out that anything with n distinct parameters is just a point in n-dimensional space. We're so used to handling so many dimensions that nobody ever bats an eye until someone brings up the magic word "dimensions". It's quite intuitive actually.
The trivial example that comes to mind is the character customization sliders in many video games.
They're borderline useless and certainly potentially inadvertently malicious for writing, customer service, speech to text, writing large amounts of complex code, therapy, medical diagnostics... the list goes on
Sounds like you're part of the problem?
It's really a serious problem just eroding the fabric of society in real-time. Being complacent in it or believing in the promise is just wholly foolish and bad for everyone.
Too shallow of a dismissal, and you don't determine what everyone else takes seriously.
It's been several years now of LLMs only appeasing those with low expectations and inexperience. Unless the only goal was generating boilerplate or really sloppy proofs of concept, LLMs are a waste time for everyone else. This argument is so over already. We're all just hoping for a soft landing when the hangover really kicks in.
You are disconnected from reality. The whole industry is already completely dominated by LLMs generating code. Bury your head in the sand all you want. This is not about low expectations or inexperience at all. Your condescending tone doesn't make you look smarter, it makes you look like an Amish who expects the industrial revolution is temporary and soon people will come to their senses and stop using all this nonsense industrial technology.
I spent the last 6 months reviewing all of the AI generated code primarily with gpt 5.3 codex but after I bought the subscription I also used gpt 5.5 and although I was constantly rejecting the bad code and I was mostly happy with the end result.
Then they removed the approve everything manually mode and I thought, maybe I'm the one out of touch, I should try vibe coding. I gave it a big spec of things to implement and I was pretty vague on the details, but I was kind of hoping that the AI would get it right anyway.
I noticed that when you let the AI just do its thing, you spend a lot of time staring at the screen waiting for it to finish. No wonder all these people are obsessed with parallel agents.
Anyway, after a while it produced the end result and it was pretty bad and I don't even mean the code. This was with gpt 5.6 Terra. Definitively not the type of "did a week's worth of work in one hour" type of deal.
Meanwhile if you guide it with your own technical skills the end result is a tireless code generator that operates at 80% of the quality of your own code but is only roughly 1.5x to 2x faster than doing it yourself. The biggest multipliers come from making the AI do things humans don't want to do, like writing a bunch of thorough tests but those don't translate into increased productivity, the opposite in fact. They translate into higher software quality but if you vibe the tests and don't review them, even that disappears.
Doesn't that say more about the massive crumb tray nobody ever bothered to empty at the bottom of mathematics?
I'm sure someone will point out something like the 4-color theorem as a counterargument. Where is that kind of theorem proving in this generation of AI? We seem to have hit a dead end rather quickly.
this is an obvious result. for example, this guy has been writing on substack about this for at least a year or two (with code snippets) explaining the phenomenon of grokking and the ghostbasin.com concept - https://richardaragon.substack.com/
their algorithm is even named "DISCOVER" so they set out to discover the connective tissue of why the universe has invariants like math, and lo it was discovered.
i guess good job for having credentials & publishing the math so people 2years behind the curve can learn from your tenure?
yes. large matrices can gradient descend to understand arbitrary symbolic logic.
ENGLISH IS INSUFFICIENT but it is at least a few decades of math proofs & progress :) welcome to the future Slackernews
Post the actual articles that you have in mind. What I've seen is vague slop.
A good example is https://richardaragon.substack.com/p/a-universal-prime-funct... describing a supposed "universal prime function" which is simply a finite approximation using a sum of 50 sines (each applied to a linear term plus a sine-log offset). The 53 parameters are fitted to the first 10^3 or so prime numbers. This is followed by the *absolutely ridiculous* claim that if the function approximates the first 10^3 primes well, it must also fit the remaining prime numbers (of which there are infinitely more than 10^3000000000) equally well.
Then they suggest "A formal proof connecting this function to the RH would involve the following steps" using this great discovery: "1. Correspondence with the Explicit Formula: Demonstrate that the oscillatory correction term in our function corresponds to the sum over zeta zeros in the explicit formula for ψ(x) or π(x). 2. Error Bound: Prove that the error in the prime counting function derived from our function is bounded by O(√x log x). 3. Contradiction: Show that if any non-trivial zero were to lie off the critical line ℜ(s) = 1/2, the error would exceed the bound, leading to a contradiction."
This isn't even midwit math.
It's the kind of naive ideas I had as a high schooler, who was good at high school math and who knew how to code functions and plots in Mathematica, but who had no understanding of higher math. This kind of naive approach to RH signals that one doesn't even understand the problem.
It's like Neo says "You get used to it, though. Your brain does the translating. I don't even see the code." He was referring to something like a K, Q, V vector at the time I believe.
"Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas". Missing the forest for the trees? Aren't neural networks modeled after biological systems? Our brains are obviously able to contain symbolic structure despite not having a "symbol processing unit".
People really overstate the relationship between ANNs and the brain, they have very different mechanisms and only have a similarity if you squint at 100000 feet. ANNs don't have neurotransmitters or even action potentials.
Correct. Worst of all, even if you do build a spiking neural network, the update rule is kind of a mystery. To have a good update rule, a biological neuron needs to be kind of like a tiny computer in its own right. You might be able to model synapses as weights between neurons, but the neuron carries further internal states within in itself and how the "update rule" uses those internal states is not known at all.
I hate that whole intro - the first four sentences - so much. It’s nothing but unsupported assumptions. Basically, a strawman that they can do battle with in the paper. Not an auspicious start.
My forthcoming paper at EMNLP offers an alternative that instead grounds the notion of representation in a very simple notion of the effect it has on model learning/behavior when you adversarially perturb it. For example, if I tell a model that in the context "I saw a duck quacking" it should replace 'duck' with 'glam', how much does it desire to replace 'duck' with 'glam' in "I need to duck out of the meeting" vs. "At the park a duck protected her ducklings." This method turns out to work quite well, and as we use only a single example, avoids the need for supervision.
The linked paper argues that their method, DISCOVER, is not supervised in the same way as DAS, since it does not directly optimize for causal effect. I have only skimmed this, but I am not so sure it might not suffer from a similar issue. They're still supervising to align representations with their underlying hypothesis, even if they don't directly supervise for causal outcomes.
Refs
- Hewitt and Liang 2019. Designing and interpreting probes with control tasks
- Kumon and Yanaka, 2026. Fine-grained analysis of shared syntactic mechanisms
- Meloux et al., 2025. Everything everywhere all at once
- Rozner and Shain 2026. Perturbation: A simple and efficient adversarial tracer for representation learning in LMs. https://arxiv.org/abs/2603.23821
- Sutter et al. 2025. The nonlinear representation dilemma
reply