We dig into how the concept of AI "safety" has been co-opted and weaponized by tech companies. Starting with examples like Mecha-Hitler Grok, we explore how real safety engineering differs from AI "alignment," the myth of the alignment tax, and why this semantic confusion matters for actual safety.

• (00:00) - Intro
• (00:21) - Mecha-Hitler Grok
• (10:07) - "Safety"
• (19:40) - Under-specification
• (53:56) - This time isn't different
• (01:01:46) - Alignment Tax myth
• (01:17:37) - Actually making AI safer

• JMLR article - Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Trail of Bits paper - Towards Comprehensive Risk Assessments and Assurance of AI-Based Systems
• SSRN paper - Uniqueness Bias: Why It Matters, How to Curb It

Additional Referenced Papers

• NeurIPS paper - Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
• ICML paper - AI Control: Improving Safety Despite Intentional Subversion
• ICML paper - DarkBench: Benchmarking Dark Patterns in Large Language Models
• OSF preprint - Current Real-World Use of Large Language Models for Mental Health
• Anthropic preprint - Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Inciting Examples

• ars Technica article - US government agency drops Grok after MechaHitler backlash, report says
• The Guardian article - Musk’s AI Grok bot rants about ‘white genocide’ in South Africa in unrelated chats
• BBC article - Update that made ChatGPT 'dangerously' sycophantic pulled

Other Sources

• London Daily article - UK AI Safety Institute Rebrands as AI Security Institute to Focus on Crime and National Security
• Vice article - Prominent AI Philosopher and ‘Father’ of Longtermism Sent Very Racist Email to a 90s Philosophy Listserv
• LessWrong blogpost - "notkilleveryoneism" sounds dumb (see comments)
• EA Forum blogpost - An Overview of the AI Safety Funding Situation
• Book by Dmitry Chernov and Didier Sornette - Man-made Catastrophes and Risk Information Concealment
• Euronews article - OpenAI adds mental health safeguards to ChatGPT, saying chatbot has fed into users’ ‘delusions’
• Pleias website
• Wikipedia page on Jaywalking

In this episode, we redefine AI's "reasoning" as mere rambling, exposing the "illusion of thinking" and "Potemkin understanding" in current models. We contrast the classical definition of reasoning (requiring logic and consistency) with Big Tech's new version, which is a generic statement about information processing. We explain how Large Rambling Models generate extensive, often irrelevant, rambling traces that appear to improve benchmarks, largely due to best-of-N sampling and benchmark gaming.

Words and definitions actually matter! Carelessness leads to misplaced investments and an overestimation of systems that are currently just surprisingly useful autocorrects.

• (00:00) - Intro
• (00:40) - OBB update and Meta's talent acquisition
• (03:09) - What are rambling models?
• (04:25) - Definitions and polarization
• (09:50) - Logic and consistency
• (17:00) - Why does this matter?
• (21:40) - More likely explanations
• (35:05) - The "illusion of thinking" and task complexity
• (39:07) - "Potemkin understanding" and surface-level recall
• (50:00) - Benchmark gaming and best-of-n sampling
• (55:40) - Costs and limitations
• (58:24) - Claude's anecdote and the Vending Bench
• (01:03:05) - Definitional switch and implications
• (01:10:18) - Outro

• Apple paper - The Illusion of Thinking
• ICML 2025 paper - Potemkin Understanding in Large Language Models
• Preprint - Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

Theoretical understanding

• Max M. Schlereth Manuscript - The limits of AGI part II
• Preprint - (How) Do Reasoning Models Reason?
• Preprint - A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers
• NeurIPS 2024 paper - How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad

Empirical explanations

• Preprint - How Do Large Language Monkeys Get Their Power (Laws)?
• Andon Labs Preprint - Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents
• LeapLab, Tsinghua University and Shanghai Jiao Tong University paper - Does Reinforcement Learning Really Incentivize Reasoning Capacity
• Preprint - RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs
• Preprint - Mind The Gap: Deep Learning Doesn't Learn Deeply
• Preprint - Measuring AI Ability to Complete Long Tasks
• Preprint - GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

Other sources

• Zuck's Haul webpage - Meta's talent acquisition tracker
  • Hacker News discussion - Opinions from the AI community
• Interconnects blogpost - The rise of reasoning machines
• Anthropic blog - Project Vend: Can Claude run a small shop?