On one side of this debate is Yudkowsky & Soares, who think that (if AI progress continues) we’re on a direct path to egregiously-misaligned, scheming, out-of-control, rogue superintelligence (ASI), not even slightly nice, in the absence of yet-to-be-invented breakthrough technical alignment ideas.

On the other side of this debate is almost everyone who works on or studies LLMs. Some of them are very concerned about egregious scheming, others much less so, and as a group they’re equally or more concerned about lots of other potential AI problems—AI-assisted bioterrorism, AI-assisted dictatorships, etc. And if they’re concerned about egregious misalignment and scheming, they’ll probably say that it would come about through race dynamics, careless programmers, bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that”.

Here’s my brief idiosyncratic take on this debate. I think BOTH of the following are true:

• (1) If you really think carefully about the properties of ASI, you really do find good reasons to strongly expect it to be egregiously misaligned, scheming, and ruthless, in the absence of yet-to-be-invented breakthrough technical alignment ideas.
• (2) If you really think carefully about the properties of current LLMs, you really do find good reasons to think that existing technical alignment techniques are adequate now, and may well continue to be adequate in the future.

So then here are three (caricatured) positions:

My position:

(1) and (2) are both totally true. And we can reconcile them by saying that LLMs won’t scale to ASI.

Yudkowsky & Soares’s position [caricatured]:

(1) is totally true. We know this with great confidence, having spent decades thinking about it.

So it follows that (2) must be wrong or irrelevant.

Why is (2) wrong or irrelevant? Hard to say! There’s no ASI yet, and nobody knows in detail how it will appear. Sometimes it’s easier to predict what happens eventually than the detailed path. An ice cube in warm water will melt eventually, but don’t ask me to predict how many seconds it will take to melt, etc.

So anyway, one possibility is that (2) is wrong because LLMs will kinda ‘wake up’, or something, when the core pieces of true intelligence finally come together. And then their behavior would change drastically for the worse. And maybe we’re already starting to see glimmers of that in existing LLMs?

Or another possibility [cf. Eliezer tweet] is that LLMs will invent non-LLM ASI. And then (2) will be simply irrelevant!

…Or something else! Again, we don’t know! But we do know that (1) is definitely right.

LLM people’s position [caricatured]:

(2) is totally true. We know this with great confidence, because we are LLM experts and we have thought about these alignment plans in great detail, including matching our theories against real-world data.

So it follows that (1) must be incorrect.

Why is (1) incorrect? I don’t really know! Man, I read Yudkowsky and Soares, and it’s all these words, words, words, and I’m reading along and trying to match those words to my knowledge of LLMs and it just doesn’t make any damn sense. I can and will try to respond to their points in detail, but honestly the core issue is that they’re guilty of head-in-the-clouds armchair theorizing gone off the rails.

Conclusion

…So I think that both sides of the debate are basically coming from a reasonable and sympathetic place, with a big kernel of truth.

Bonus section: Further commentary

…That said, I can still complain at both sides!

My “true objection” to Yudkowsky & Soares:

For the record, my “true objection” to Yudkowsky & Soares is that if we’re talking about ASI, then LLMs are basically irrelevant and we shouldn’t even be talking about LLMs at all. And meanwhile, their plans are misguided because delaying ASI is possible on the margin but mostly hopeless, although I guess I’m happy that they’re trying anyway. Meanwhile, my hunch is that they’re overstating the intractability of finding that technical alignment breakthrough, although I haven’t found it yet, so I guess time will tell.

My within-frame complaint at Yudkowsky & Soares:

…But I’ll put that aside for the sake of argument, and bring up a narrower complaint within their frame:

I think their suggestions that LLMs may become completely egregiously misaligned in the future via … umm … the ‘true core of intelligence’ coming together, and ‘waking up’? Like Skynet or something?? That was mean, sorry, but in any case, I don’t think this idea hangs together either theoretically or empirically.

For the former (theory), see my discussion of the extreme weirdness of the LLM pretraining algorithm in Foom & Doom §2.3.2. I think Yudkowsky & Soares have not internalized how weird this type of learning algorithm is, and if they had, then Yudkowsky would not be occasionally suggesting that we should think of an LLM as an actress playing characters.

For the latter (empirical), I think the most fair assessment is that current LLMs are nice and obedient in some contexts, and LLMs are mean, defiant, and just plain weird in other contexts. You can straightforwardly go from that observation to “maybe there will be egregious misalignment and scheming in the future”, but not to “there will definitely be egregious misalignment and scheming in the future, absent new breakthrough technical alignment ideas”.

I think that if Yudkowsky & Soares stopped treating current LLMs as direct evidence for technical alignment being definitely completely unsolved, and instead treated it as either mixed evidence or entirely off-topic, then their public messaging would come across to policymakers and general audiences as somewhat more convoluted and confusing. But I think it would be more accurate. Oh well.

My “true objection” to LLM people:

For the record, my “true objection” to the LLM people is that I don’t really care about anything they say, because I’m working on the ASI alignment problem, and LLMs won’t scale to ASI.

(I’m overstating a bit. I’m generally happy for people to work on making LLM-world a place of wisdom and goodness, especially because LLM-world is the world in which ASI will someday be invented.)

My within-frame complaint at LLM people:

…But I’ll put that aside for the sake of argument, and bring up a narrower complaint within their frame:

I think the LLM people are not pricing in the predictable consequences of ever more RLVR and/or the predictable consequences of ever more “real” open-ended continual learning, should the latter ever be solved (which I don’t think it will be, but never mind that).

In other words, lots of LLM-focused people say “LLMs will eventually be able to do the things that human society did over the last 5000 years: open-endedly and autonomously build new knowledge and ideas on top of new knowledge and ideas, in an endless tower, with no need for human ground truth anywhere in that process. And how exactly will the future LLMs do that? Uhh, I don’t know, people are working on it, they’ll probably figure something out.”

…And bam, that blank spot in the map is where the pea gets hidden under the thimble.

Because if you want the LLMs to gain ever more knowledge, whether through a perpetual RLVR loop or some other yet-to-be-invented type of continual learning, there has to be some kind of ground truth, or else it will go off the rails into nonsense. And that ground truth, whatever it is, will basically amount to an objective function (a.k.a. cost function, reward function, whatever). And when the LLM updates enough on that ground truth, then whatever human-niceness that the LLM inherited from pretraining will get diluted away in favor of ruthless maximization of that objective function.

(See also: Why we should expect ruthless sociopath ASI.)

Thanks Zack M. Davis for a brief discussion that inspired this post.