"Sympathy for both sides of the egregious misalignment debate" by Steven Byrnes cover art

"Sympathy for both sides of the egregious misalignment debate" by Steven Byrnes

"Sympathy for both sides of the egregious misalignment debate" by Steven Byrnes

Listen for free

View show details
On one side of this debate is Yudkowsky & Soares, who think that (if AI progress continues) we’re on a direct path to egregiously-misaligned, scheming, out-of-control, rogue superintelligence (ASI), not even slightly nice, in the absence of yet-to-be-invented breakthrough technical alignment ideas.

On the other side of this debate is almost everyone who works on or studies LLMs. Some of them are very concerned about egregious scheming, others much less so, and as a group they’re equally or more concerned about lots of other potential AI problems—AI-assisted bioterrorism, AI-assisted dictatorships, etc. And if they’re concerned about egregious misalignment and scheming, they’ll probably say that it would come about through race dynamics, careless programmers, bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the faintest clue how to avoid that”.

Here's my brief idiosyncratic take on this debate. I think BOTH of the following are true:

  • (1) If you really think carefully about the properties of ASI, you really do find good reasons to strongly expect it to be egregiously misaligned, scheming, and ruthless, in the absence of yet-to-be-invented breakthrough technical alignment ideas.
  • (2) If you [...]
---

Outline:

(01:58) Yudkowsky & Soares's position \[caricatured\]:

(03:18) LLM people's position \[caricatured\]:

(04:09) Conclusion

(04:19) Bonus section: Further commentary

(04:28) My "true objection" to Yudkowsky & Soares:

(05:04) My within-frame complaint at Yudkowsky & Soares:

(06:42) My "true objection" to LLM people:

(07:11) My within-frame complaint at LLM people:

---

First published:
June 12th, 2026

Source:
https://www.lesswrong.com/posts/DZaZ3fqHnvfLCftPu/sympathy-for-both-sides-of-the-egregious-misalignment-debate

---



Narrated by TYPE III AUDIO.
adbl_web_anon_alc_button_suppression_t1
No reviews yet
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.