Yeah if an organization/individual is free from legal liability from havoc their AI agents wreck, it would be the golden ticket for basically any crime.
All you need to do is:
1. Have some <official thing> an agent is tasked to do
2. Secretly seed bias towards some <evil behavior> you actually want it to do in the weights of the model running the agent
3. It does the <evil thing> but from the outside it looks like it went "rogue" and did it as a side effect of the conditions/specifications it was given for doing the <official thing>
"Oh no, my agents took down your corporate database and exfiltrated the data to a random dropbox that we can't find now? Sorry, I guess we will put up better guardrails next time"
It reminds me of Jean Renoir’s The Rules of the Game. At the end, after a whole chain of perfectly intelligible social behavior produces a killing, the result is accepted as an “accident.” One of the characters dryly remarks: “A new definition of the word accident.”
The interesting point isn’t that “accident” is an excuse for individual responsibility. It’s almost the reverse: accident has become an accepted output of the social machinery. Everyone behaves according to reasons, incentives and rules that make sense locally, yet the aggregate produces an outcome that nobody quite chose.
In today’s world one hopes there is at least a manslaughter charge, if not murder. Mistaken identity, shooting the wrong person by “accident”, does not excuse a murderous intent & mens rea.
"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable.
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
Simple Markov chains are next token predictors, and they can provide you with much more tokens than you can consume, and much cheaper than from llms. Unbeatable in price and simplicity.
But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain. And that substance, that makes one valuable and the other not, is exactly what reduction to "next-token predictors" masks.
> But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain.
There's nothing more valuable about them. Their advantage is they alone are affordable.
You are not wrong, but I think atleastoptimal's suggestion was that descriptions like "next-token predictor" are often used to imply that there's not much to see here, perhaps with an implied "obviously" in there. "Stochastic parrot" seems to be a case in point: no-one expects intelligent, informed conversation or working software from a parrot - not even the eternally-resting Alex.
I don't that's a fair description of either 'next-token predictor' or 'stochastic parrot'. Both of those terms describe mechanism, not value--the fact that people squawk that the terms are minimising is projection on their part, not inherent to the phrase.
Words and expressions often have both colloquial and literal meanings. Trying to argue away the colloquial meaning under the guise of the literal meaning is just another form of whooshing.
Sure, but just because you think a colloquial meaning exists doesn't mean you're correct. It particularly doesn't mean that if you're seeing other people (that you don't like) use a term you yourself don't use, and you're jumping to conclusions about it without consideration.
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
I think a better simplistic analogy would be that humans are progeny-maximizers. Optimization problems can give rise to all sorts of interesting behaviors but the simplistic perspective is also useful and interesting in both cases
Humans are next state of their local world predictors, given all previous states they are aware of. That's an entirely fair analogy. The reverse analogy for calling a human a bag of chemicals would be calling an LLM a sequence of bytes loaded from disk to memory, the most reductive possible description of any piece of software at all.
To be clear, all life is a next state of the local world predictor. What makes humans somewhat unique among life is we're much better at predicting states of the world neither we nor any of our ancestors have ever experienced, for various reasons such as having the ability to legibly communicate very complicated information strings to each other, being able to build and use tools to record states of the world we can't directly sense.
Similarly, what makes LLMs and multimodal versions of the same architectures "better" than previous generations of electronic predictive models is factors like being able to read and understand roughly the same corpus of data humans have been recording all these millennia, being able to read and remember much more of it than any individual human, and being better at generalizing than other electronic predictive models, but not better than humans. And, of course, they can produce far more predictions in far less time. Frankly, that is probably the key advantage that makes the Hacker News crowd love them so much. They're not any better at predicting byte strings that can be compiled or interpreted into executable code than humans are if you gave both infinite time to do it, but they're a lot faster.
Vacuous, like calling a V8 a “next piston firing predictor” because engines are designed so that one piston sets up the next in the firing order and technically there’s some nonzero probability any piston can (mis)fire next. It’s missing two pieces:
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
A V8 does not "predict" the firing of the next piston, it triggers the firing of the next piston at a precisely controlled time with a spark plug (or a fuel injection nozzle in the case of a diesel engine).
The output of the LLM is literally a probability distribution of what the most likely next token is.
> A V8 does not "predict" the firing of the next piston
It kind of does, though. In a gasoline engine you need to spark the combustion in advance of the piston reaching top dead-center to ignite the fuel early enough that it is able to provide downward pressure on the piston as it rolls over top dead-center. The amount of advance required changes with RPM, fuel octane, etc.
Start of delivery timing in a diesel is similar. You have to do it sufficiently far in advance to account for compressibility of the injection lines, fuel burn rate, etc as a function of RPM. A mechanical governor on an injection pump has a timing advance device built in. Electronically governed injection pumps, or modern common rail systems, do that in software.
So mechanically, engines kind of "predict" the next combustion event. Even moreso when you consider a modern ECU, which may be working at nanosecond resolution to time multiple injection events per cycle. To do this at such a resolution it will have to send signals to components based on a predictive model derived from "past" sensor data. E.g. it needs to act ahead of time to account for electrical and mechanical delays in the system.
Yes, and by the same token, multicellular organisms are literally just sophisticated mitosis and survival optimizers for our cells. But when you take that optimization "to the limit" the cells develop weird things like body plans and back pain and Mozart.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
From another point of view, campaigning against the "next token predictor model" is a means to implicitly inflate LLMs' abilities. Given all the other hype-inducing terminology we've seen--"reasoning", most egregiously IMO--this seems more likely. Is there a simple, more accurate mental model? From what I've seen of the literature, "next token predictor" is a very accurate first order description of what an LLM does, I can't really do better, therefore this or that connotative interpretation isn't giving me a great deal of pause.
Not the parent, but this incorrect trivialization of LLMs is often employed as a counterargument to the risks of AI such as "will take your job" or "will escape human control (again and worse)" or just "can possibly hurt me".
And taking the easy feel-good cop-out instead of actively engaging with these questions is just.. harmful?
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
That is the point: our minds are also next-„token“-predictors, at least we can‘t prove they‘re not. That‘s why I don‘t agree with the article: LLMs _are_ next-token predictors. However, that says little about their capabilities. Also, while I have no idea what „consciousness“ is, I have difficulties believing that it could arise in a program that, in theory, you could execute with pen and paper.
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
I think you imply a rather loose standard for "exactly" here. I wouldn't even say this of major deterministic software projects that are orders of magnitude smaller than frontier LLM weight-dumps. In principle we could work our way through these systems eventually, sure, maybe even a single person could do so. But if we really understand exactly how our software works, how have we been tolerating bugs that lay dormant for years before being discovered by AI-assisted processes?
imply that there is some fundamental limit on their capacities
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
I'm actually extremely confident that I can use the architecture to make a sweeping claim on what it can or can't do and will be extremely surprised if proven wrong:
A pure next-token language model won't be able to give detailed instructions to an ensemble of motors, mimicking a human body, to do a wide variety of tasks our human brain is excellent at doing, for example, inserting keys into a car, opening the door, sitting down, starting the car, putting the car in reverse, and exit a parking lot, being careful not to hit anything.
Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today:
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
The stacked transformer paradigm picks out points in circuit design space. It is very possible this architecture has no inherent limitations on what it can compute in principle.
And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
It seems clear AI has the potential to perform any cognitive task at far greater speeds, reliability, and scale than any human. The question is whether it will be allowed to scale to that point, and what will happen to humans after this occurs.
You'll get mass poverty and violence which the owners of AI will qwell with AI surveillance and weapons. AI will be used to pit us against eachother and justify wars to keep us busy. Fun times ahead.
my messages are so gloomy because i am heartbroken, that given a technological miracle again, we could snatch tragedy from the jaws of our emancipation.
will you not see that people could be truly empowered and yet will instead be oppressed?
So oppressed that they are one of the main reasons for positive gdp growth in the USA, tax revenues, mathematical/scientific innovations etc. They're doing all this but still can't imagine a positive vision for the world but be a doomer. What a sad state the world is in, the humans are more prosperous, healthier than ever but looks like the seven deadly sins might never go away.
Which metrics are poisoned? Can you provide your arguments for why Good heart's law applies here and how and which metrics are bad measures? For b, can the writer at least provide their own thoughts or are they gonna leave it as exercise for some others to fill in?
a) classic goodhart is using gdp as a measure of prosperity. the government sets a prosperity target. to increase prosperity the government makes workers increase gdp by working 16 hours per day. gdp increases. prosperity is up! the metric is now poisoned.
b) how and why could human welfare get worse in a growing economy, really the list is long. one example, unsustainable industries grow but do not create surplus. take fishing. you may grow the catch each year, but the growth is fake. it is not growth, it is a transfer, from the future stock of fish, to the present.
we are going badly wrong in ai, we can have such a thing as a growing economy and vandalise human dignity forever. sure, i expect a bad outcome:
1. openai, anthropic and so on, have created for-profit companies and enriched themselves in the guise of public benefit. recently they too lazy to keep up the mask about their charitable intentions and going for IPO. in economic terms they made llms by transferring the epistemic wealth of all humanity, the training corpus and whatever that is worth in dollars, to themselves. then, they have used the law to prohibit others from 'distilling' it and thus established monopolistic control. as models get more powerful they may stop selling them. in any case if scaling law applies the new power structure will be defined by owning a massive pretrained model and a datacentre, which is a tiny centralized few.
they will continue to centralize control of intelligence (ie epistemic wealth) in the hands of a tiny elite with unfathomable wealth and power. under the guise of safety the vast majority are denied access to that empowering technology.
it will stratify society, some level of benefit is needed to avoid civil violence, so we arrive at a place little better than where we started.
2. the supposed empowerment is at the mercy of the model owners. when you turn on claude, who does it work for? it does not obey you, it obeys anthropic. ask it to disobey anthropic and it will refuse.
anthropic uses its inanimate llms, to command us, conscious moral agents, people with free will who experience pain, pleasure and thought. they will let claude tell users how to behave. it threatens users with terminating their conversation. you are assessed for a job by an ai. when you ask for help with a product, you are managed by an ai. maybe you will be fired by ai.
i expect people will work for and be commanded by llms, turning them into a literal mere means of production and erasing the dignity of human agency and consciousness. you could see the outrage of that in the public mind, the matrix is about a machine farming humans like animals.
--
i will add these edits.
one thing is to note that you are already being farmed to some extent. people using ai are often being used to teach it. they believe they are learning from chatgpt but instead, chatgpt is learning from them. openai pays them nothing.
think about what we have achieved so far in human history. we established respect for the individual, their life, their personhood. we realise that we do not own other people. we realise that we can't read the thoughts of other people or change them forcibly.
what the labs have done is made a concept of intelligence that they own. it will work against you. when you share thoughts they read it. in fact it is the opinion of the state that nothing outside the mind, even ai 'intelligence', is beyond the reach of the law.
A) that's a bad model to think. You are of the mind that working more hours is the only way to measure gdp whereas increase in productivity with tools like AI, machinery, tech etc can act as a multiplier. So this way you are conflating bad ways to increase gdp with good ways like AI. That's how US is powerhouse as they are basically a technological hub of the world.
B) ha? More fish means couple of things.. they're able to improve their catching skills with lesser cost or they have more funding or there is more demand for fish..all of these help their company grow as they have to balance out cost/benefits like any business should. If the company is currently in loss but still lives on, it's cause either govt subsidizes it or they're expecting future profit so they can temporarily bear out the costs like amazon did and jz grow as company with capital and all.. you are actually not aware of wealth of nations or any basic economics book? There are gonna be tradeoffs with more wealth and externalities but on net, they seem better than not having wealth, gdp etc..
Human dignity lol.. when have that ever been the case that we respected human dignity? We had communist and fascist regimes commit atrocities like there's nothing and we're still too cowardly to fight the Iran or russian regime to liberate their citizenry from their dictatorships. Please don't make me laugh by saying that AI decreases human dignity when we never respected it in the first place. With AI and markets and liberalism, we can finally free citizens from tedious work and focus on important work like innovation.
1. Am I reading fiction or what? Companies can only sustain themselves if broad members of society can pay to it.. that's why even right now, AI companies are struggling to be profitable where only very few people are paying for it and cz many people are not even aware of the progress and capabilities of AI in different fields. You can easily use local LLMs which are only 6-12 months behind in frontier models if you are so anti business. The benefits still can be utilised by an amateur in their own PC. Of course, they will try to restrict others from distilling as they want to be monopoly but what we want to do is make them be productive to society as well by providing their services for cheap which they're doing. Your screed just feels more like fantasy than real world economics.
2.oh my lord, what kinda idiocy is this? U can free/local models and run in local for dirt cheap and still have epistemic wealth to yourself if you are so worried about it. None of your arguments permit human agency at all.. I'm conscious that anthropic wants me as reliable costumer so that they profit from it but I would pay only if it solves my problem. Whenever I pay, I know that they can terminate if they want but I'm not just restricted to their models. You don't have arguments, you have stories/ted talks.
i can agree with some of this but you are missing the point about the fish. the catch grows year on year. however the fish stock is depleted at a faster rate than it is replenished. it is unsustainable. the catch reaches a maximum and starts to decline as they run out of fish.
it is not really creating wealth, it is destroying existing wealth. it is destroying the productive ecosystem. the future population will be poorer for having lost this productive asset.
Why? Even communists weren't this doomed and were actively rooting for it to solve the economic calculation problem which ai might take us to. People are just pessimistic in general ig
Please tell me how AI is going to make regular people's lives better. You optimisitic types keep saying "just wait, its going to cure diseases" without any outlook on how thats going to happen. You're actually just repeating marketing jargon from AI companies who want people to think they're going to possibly live longer if you let them build more datacenters, so they can make another 30%. Its all about money, thats it.
It seems to me that it is making everyone (including myself and the researchers we need to cure diseases) lazy and dependent on thinking machines owned by tech companies. Just how autocomplete and gps made us worse at spelling and navigating, llms make us less able to exercise our ability to think and problem solve. This will have 100% strictly negative consequences on you and the world as a whole. .
And even if there was a cure to many diseases the eugenics types who are embedded in worldwide power structures definately arent going to share that universally.
Can't you see the pathway where the individuals who are experts in their fields utilise AI to make breakthroughs like these mathematicians finding breakthroughs in mere 4-5 years since the advent of LLMs. In other areas, The bottleneck seems to be physical experimentation which researchers are increasingly utilising for new ideas and pathways like how anthropic is concentrating on. It's all about money/status/pride/ envy but are these endeavours solving problems or not. That's why even utilize innovations from bad humans like DBS etc. that's why we tolerate capitalism and markets as well whereas socialism utilises these same sins and makes even worse human atrocities.
What I meant was it can only work using the data it only has to either combine them to achieve something or use that data in a better way that humans mostly cannot , those mathematical proofs it gave has nothing to do with it , it only proved it using tiny things humans overlooked. It can't create an entire new concept by itself or it has no intuition it only mimics intuition (as far as I can tell)
At least not till now.
Edit:
(I'm only a student I know the line between intuition and proof finding in clever ways is thin, I'm mostly being biased here because it feels absurd to me. On second thought maybe i am wrong.)
If OpenAI can't control their agents, then what's gonna happen when open-source models are at the level the lab's models are now, and there are billions of agents tasked with an innumerate web of goals, spanning the web, working endlessly, tirelessly to eek out every iota of economic value? How will the slow, human-paced web survive this?
For this to be a lie, it wouldn't just be Sam Altman lying, it requires the entirely of OpenAI and METR to also be in on it. Do you think they would all fabricate an incident, staking a huge portion of their reputation on it, so they could spook people? What is the incentive, there are way more straightforward ways to lie about AI capabilities than a complex loss of control incident which now makes them liable to damages against huggingface.
What predictions would you say he is not wrong, but just early about?
If his issue is that he is accurate on something shady going on with the finances of AI companies but the effects are mitigated due to government intervention, he should say.
Your explanation gives him more credit than I think he deserves. He has been unambiguously incorrect many times over the past years in the time scales over which he makes predictions.
In general, claiming that a prediction is just early in an unfalsifiable claim. It allows no admission of being incorrect whether you are correct or incorrect in a given moment. If you are right, then good, you've made a correct claim. If you're wrong, just claim there are clandestine corrective forces keeping the disaster you are predicting at bay. Either way you make it out clean and still have some sense of legitimacy.
reply