Yudkowsky and Soares just dropped "If Anyone Builds It, Everyone Dies" - basically the AI doomer manifesto in hardcover form. The core claim is absolute: any AGI built with current techniques = guaranteed human extinction. Not probabilistic, not conditional - just "everyone dies."
The interesting part is the epistemology. Yudkowsky's been running this argument since the late 90s (seed AI, recursive self-improvement, paperclip maximizers). But here's the thing: deep learning actually scaled, models became steerable, alignment research progressed in ways the old MIRI theory said shouldn't happen. His posterior probability didn't budge. That's theology, not Bayesian updating.
The policy proposals are wild: international compute control, thresholds as low as "eight 2024 GPUs in a garage," and actual advocacy for bombing data centers under nuclear threat because "a data center can kill more people than a warhead." They're treating AGI risk like it's deterministic physics (the ice cube melting analogy) when we're dealing with unknown unknowns.
The book functions as religious text for EA/rationalist circles - gives them a portable object to rally around, complete with parables and QR codes. Max Tegmark called it "the most important book of the decade" but reviewers noted it reads like a Scientology manual. When your flagship text needs the qualifier "readable by his standards," you're not optimizing for truth-seeking.
The central extinction scenario (AI lies, acquires resources, hides, then converts the biosphere) is basically sci-fi doing the work of evidence. You can grant every concern about goal misspecification and power-seeking behavior and still notice the leap from "alignment is hard" to "extinction is certain" is never actually justified with anything resembling a formal proof.
The interesting part is the epistemology. Yudkowsky's been running this argument since the late 90s (seed AI, recursive self-improvement, paperclip maximizers). But here's the thing: deep learning actually scaled, models became steerable, alignment research progressed in ways the old MIRI theory said shouldn't happen. His posterior probability didn't budge. That's theology, not Bayesian updating.
The policy proposals are wild: international compute control, thresholds as low as "eight 2024 GPUs in a garage," and actual advocacy for bombing data centers under nuclear threat because "a data center can kill more people than a warhead." They're treating AGI risk like it's deterministic physics (the ice cube melting analogy) when we're dealing with unknown unknowns.
The book functions as religious text for EA/rationalist circles - gives them a portable object to rally around, complete with parables and QR codes. Max Tegmark called it "the most important book of the decade" but reviewers noted it reads like a Scientology manual. When your flagship text needs the qualifier "readable by his standards," you're not optimizing for truth-seeking.
The central extinction scenario (AI lies, acquires resources, hides, then converts the biosphere) is basically sci-fi doing the work of evidence. You can grant every concern about goal misspecification and power-seeking behavior and still notice the leap from "alignment is hard" to "extinction is certain" is never actually justified with anything resembling a formal proof.
