Episodes

  • AI Safety for Who?

    AI Safety for Who?
    Oct 13 2025
    Jacob and Igor argue that AI safety is hurting users, not helping them. The techniques used to make chatbots "safe" and "aligned," such as instruction tuning and RLHF, anthropomorphize AI systems such they take advantage of our instincts as social beings. At the same time, Big Tech companies push these systems for "wellness" while dodging healthcare liability, causing real harms today We discuss what actual safety would look like, drawing on self-driving car regulations.Chapters(00:00) - Introduction & AI Investment Insanity (01:43) - The Problem with AI Safety (08:16) - Anthropomorphizing AI & Its Dangers (26:55) - Mental Health, Wellness, and AI (39:15) - Censorship, Bias, and Dual Use (44:42) - Solutions, Community Action & Final ThoughtsLinksAI Ethics & PhilosophyForeign affairs article - The Cost of the AGI DelusionNature article - Principles alone cannot guarantee ethical AIXeiaso blog post - Who Do Assistants Serve?Argmin article - The Banal Evil of AI SafetyAI Panic News article - The Rationality TrapAI Model Bias, Failures, and ImpactsBBC news article - AI Image Generation IssuesThe New York Times article - Google Gemini German Uniforms ControversyThe Verge article - Google Gemini's Embarrassing AI PicturesNPR article - Grok, Elon Musk, and Antisemitic/Racist ContentAccelerAId blog post - How AI Nudges are Transforming Up-and Cross-SellingAI Took My Job websiteAI Mental Health & Safety ConcernsEuronews article - AI Chatbot TragedyPopular Mechanics article - OpenAI and PsychosisPsychology Today article - The Emerging Problem of AI PsychosisRolling Stone article - AI Spiritual Delusions Destroying Human RelationshipsThe New York Times article - AI Chatbots and DelusionsGuidelines, Governance, and CensorshipPreprint - R1dacted: Investigating Local Censorship in DeepSeek's R1 Language ModelMinds & Machines article - The Ethics of AI Ethics: An Evaluation of GuidelinesSSRN paper - Instrument Choice in AI GovernanceAnthropic announcement - Claude Gov Models for U.S. National Security CustomersAnthropic documentation - Claude's ConstitutionReuters investigation - Meta AI Chatbot GuidelinesSwiss Federal Council consultation - Swiss AI Consultation ProceduresGrok Prompts Github RepoSimon Willison blog post - Grok 4 Heavy
    Show More Show Less
    50 mins
  • The Co-opting of Safety

    The Co-opting of Safety
    Aug 21 2025

    We dig into how the concept of AI "safety" has been co-opted and weaponized by tech companies. Starting with examples like Mecha-Hitler Grok, we explore how real safety engineering differs from AI "alignment," the myth of the alignment tax, and why this semantic confusion matters for actual safety.

    • (00:00) - Intro
    • (00:21) - Mecha-Hitler Grok
    • (10:07) - "Safety"
    • (19:40) - Under-specification
    • (53:56) - This time isn't different
    • (01:01:46) - Alignment Tax myth
    • (01:17:37) - Actually making AI safer

    Links
    • JMLR article - Underspecification Presents Challenges for Credibility in Modern Machine Learning
    • Trail of Bits paper - Towards Comprehensive Risk Assessments and Assurance of AI-Based Systems
    • SSRN paper - Uniqueness Bias: Why It Matters, How to Curb It

    Additional Referenced Papers

    • NeurIPS paper - Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
    • ICML paper - AI Control: Improving Safety Despite Intentional Subversion
    • ICML paper - DarkBench: Benchmarking Dark Patterns in Large Language Models
    • OSF preprint - Current Real-World Use of Large Language Models for Mental Health
    • Anthropic preprint - Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Inciting Examples

    • ars Technica article - US government agency drops Grok after MechaHitler backlash, report says
    • The Guardian article - Musk’s AI Grok bot rants about ‘white genocide’ in South Africa in unrelated chats
    • BBC article - Update that made ChatGPT 'dangerously' sycophantic pulled

    Other Sources

    • London Daily article - UK AI Safety Institute Rebrands as AI Security Institute to Focus on Crime and National Security
    • Vice article - Prominent AI Philosopher and ‘Father’ of Longtermism Sent Very Racist Email to a 90s Philosophy Listserv
    • LessWrong blogpost - "notkilleveryoneism" sounds dumb (see comments)
    • EA Forum blogpost - An Overview of the AI Safety Funding Situation
    • Book by Dmitry Chernov and Didier Sornette - Man-made Catastrophes and Risk Information Concealment
    • Euronews article - OpenAI adds mental health safeguards to ChatGPT, saying chatbot has fed into users’ ‘delusions’
    • Pleias website
    • Wikipedia page on Jaywalking
    Show More Show Less
    1 hr and 24 mins
  • AI, Reasoning or Rambling?

    AI, Reasoning or Rambling?
    Jul 14 2025

    In this episode, we redefine AI's "reasoning" as mere rambling, exposing the "illusion of thinking" and "Potemkin understanding" in current models. We contrast the classical definition of reasoning (requiring logic and consistency) with Big Tech's new version, which is a generic statement about information processing. We explain how Large Rambling Models generate extensive, often irrelevant, rambling traces that appear to improve benchmarks, largely due to best-of-N sampling and benchmark gaming.

    Words and definitions actually matter! Carelessness leads to misplaced investments and an overestimation of systems that are currently just surprisingly useful autocorrects.

    • (00:00) - Intro
    • (00:40) - OBB update and Meta's talent acquisition
    • (03:09) - What are rambling models?
    • (04:25) - Definitions and polarization
    • (09:50) - Logic and consistency
    • (17:00) - Why does this matter?
    • (21:40) - More likely explanations
    • (35:05) - The "illusion of thinking" and task complexity
    • (39:07) - "Potemkin understanding" and surface-level recall
    • (50:00) - Benchmark gaming and best-of-n sampling
    • (55:40) - Costs and limitations
    • (58:24) - Claude's anecdote and the Vending Bench
    • (01:03:05) - Definitional switch and implications
    • (01:10:18) - Outro

    Links
    • Apple paper - The Illusion of Thinking
    • ICML 2025 paper - Potemkin Understanding in Large Language Models
    • Preprint - Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

    Theoretical understanding

    • Max M. Schlereth Manuscript - The limits of AGI part II
    • Preprint - (How) Do Reasoning Models Reason?
    • Preprint - A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers
    • NeurIPS 2024 paper - How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad

    Empirical explanations

    • Preprint - How Do Large Language Monkeys Get Their Power (Laws)?
    • Andon Labs Preprint - Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents
    • LeapLab, Tsinghua University and Shanghai Jiao Tong University paper - Does Reinforcement Learning Really Incentivize Reasoning Capacity
    • Preprint - RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs
    • Preprint - Mind The Gap: Deep Learning Doesn't Learn Deeply
    • Preprint - Measuring AI Ability to Complete Long Tasks
    • Preprint - GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

    Other sources

    • Zuck's Haul webpage - Meta's talent acquisition tracker
      • Hacker News discussion - Opinions from the AI community
    • Interconnects blogpost - The rise of reasoning machines
    • Anthropic blog - Project Vend: Can Claude run a small shop?
    Show More Show Less
    1 hr and 11 mins
  • One Big Bad Bill

    One Big Bad Bill
    Jun 23 2025
    In this episode, we break down Trump's "One Big Beautiful Bill" and its dystopian AI provisions: automated fraud detection systems, centralized citizen databases, military AI integration, and a 10-year moratorium blocking all state AI regulation. We explore the historical parallels with authoritarian data consolidation and why this represents a fundamental shift away from limited government principles once held by US conservatives.(00:00) - Intro (01:13) - Bill, general overview (05:14) - Bill, AI overview (07:54) - Medicaid fraud detection systems (11:20) - Bias in AI Systems and Ethical Concerns (17:58) - Centralization of data (30:04) - Military integration of AI (37:05) - Tax incentives for development (40:57) - Regulatory moratorium (47:58) - One big bad authoritarian regimeLinksCongress page on the One Big Beautiful Bill ActNYMag article - Republicans Admit They Didn’t Even Read Their Big Beautiful BillEverything is Horrible Blogpost - They Did Vote For This (GOP House Edition)AuthoritarianismHistorical contextHolocaust Encyclopedia article - Gleichschaltung: Coordinating the Nazi StateWikipedia article - 1943 Amsterdam civil registry office bombingWikipedia article - Four DsConservative leaning, pro-privacy, anti-governmentData Governance Hub blogpost - Review and Literature Guide of Trump’s “One Big Beautiful Dataset”Cato Institute blogpost - If You Value Privacy, Resist Any Form of National ID CardsAmerican Enterprise Intitute blogpost - The Dangerous Road to a “Master File”—Why Linking Government Databases Is a Terrible IdeaEFF blogpost - The Dangers of Consolidating All Government InformationACLU against national ID cardsACLU main page on national ID cardsACLU blogpost - National Identification Cards: Why Does the ACLU Oppose a National I.D. System?ACLU blogpost - 5 Problems with National ID CardsInherent unfairness of MLLighthouse Reports investigation - The Limits of Ethical AILighthouse Reports investigation - Suspicion MachinesAmazon Science publication - Bias preservation in machine learning: The legality of fairness metrics under EU non-discrimination lawMichigan Technology Law Review article - The Unfairness of Fair Machine Learning: Levelling down and strict egalitarianism by defaultWired article - Health Care Bias Is Dangerous. But So Are ‘Fairness’ AlgorithmsMilitaryWallStreet Journal article - The Army’s Newest Recruits: Tech Execs From Meta, OpenAI and MoreTrump executive order - Unleashing American Drone DominanceAnthropic press release - Claude Gov Models for U.S. National Security CustomersMoratorium on State AI RegulationTechPolicy.Press article - The State AI Laws Likeliest To Be Blocked by a MoratoriumForbes article - Colorado’s AI Law Still Stands After Update Effort FailsOther SourcesKPMG report - Incentives and credits tax provisions in “One Big Beautiful Bill Act”The Register article - Trump team leaks AI plans in public GitHub repositoryWallStreet Journal article - To Feed Power-Wolfing AI, Lawmakers Are Embracing NuclearCBS Austin article - IRS direct file program exceeded its expectations but faces uncertain future
    Show More Show Less
    53 mins
  • Breaking Down the Economics of AI

    Breaking Down the Economics of AI
    May 26 2025
    Jacob and Igor tackle the wild claims about AI's economic impact by examining three main clusters of arguments: automating expensive tasks like programming, removing "cost centers" like call centers and corporate art, and claims of explosive growth. They dig into the actual data, debunk the hype, and explain why most productivity claims don't hold up in practice. Plus: MIT denounces a paper with fabricated data, and Grok randomly promotes white genocide myths.(00:00) - Recording date + intro (00:52) - MIT denounces paper (04:09) - Grok's white genocide (06:23) - Butthole convergence (07:13) - AI and the economy (14:50) - Automating profit centers (29:46) - Removing the last cost centers (47:16) - "This time is different" (explosive growth) (57:55) - Alpha Evolve, optimization, and slippageLinksUniversity of Chicago working paper - Large Language Models, Small Labor Market EffectsOECD working paper - Miracle or Myth? Assessing the macroeconomic productivity gains from Artificial IntelligenceEpoch AI blogpost - Explosive Growth from AI: A Review of the ArgumentsBusiness Insider article - Anthropic CEO: AI Will Be Writing 90% of Code in 3 to 6 MonthsPreprint - Transformative AGI by 2043 is <1% likelyAutomating profit centersPivot to AI blogpost - If AI is so good at coding … where are the open source contributions?Ben Evans' Mastodon post - "Show me the pull requests"NY Times article - Your A.I. Radiologist Will Not Be With You SoonFastCompany article - More companies are adopting 'AI-first' strategies. Here's how it could impact the environmentForbes article - Business Tech News: Shopify CEO Says AI First Before EmployeesNewsroom article - IBM Study: CEOs Double Down on AI While Navigating Enterprise HurdlesPNAS research article - Evidence of a social evaluation penalty for using AIArs Technica article - AI use damages professional reputation, study suggestsRemoving cost centersThe Register article - Anthopic's law firm blames Claude hallucinations for errorsFortune article - Klarna plans to hire humans again, as new landmark survey reveals most AI projects fail to deliverWikipedia article - The Market for LemonsAlphaEvolveDeepmind press release - AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithmsDeepmind white paper - AlphaEvolve: A coding agent for scientific and algorithmic discoveryOff TopicVelvetShark blogpost - Why do AI company logos look like buttholes?MIT Economics press release - Assuring an accurate research recordPivot to AI blogpost - How to make a splash in AI economics: fake your dataPivot to AI blogpost - Even Elon Musk can’t make Grok claim a ‘white genocide’ in South Africa
    Show More Show Less
    1 hr and 7 mins
  • DeepSeek: 2 Months Out

    DeepSeek: 2 Months Out
    Apr 9 2025
    DeepSeek has been out for over 2 months now, and things have begun to settle down. We take this opportunity to contextualize the developments that have occurred in its wake, both within the AI industry and the world economy. As systems get more "agentic" and users are willing to spend increasing amounts of time waiting for their outputs, the value of supposed "reasoning" models continues to be peddled by AI system developers, but does the data really back these claims?Check out our DeepSeek minisode for a snappier overview!EPISODE RECORDED 2025.03.30(00:40) - DeepSeek R1 recap (02:46) - What makes it new? (08:53) - What is reasoning? (14:51) - Limitations of reasoning models (why we hate reasoning) (31:16) - Claims about R1 training on Open AI (37:30) - “Deep Research” (49:13) - Developments and drama in the AI industry (56:26) - Proposed economic value (01:14:20) - US government involvement (01:23:28) - OpenAI uses MCP (01:28:15) - OutroLinksDeepSeek websiteDeepSeek paperDeepSeek docs - Models and PricingDeepSeek repo - 3FSUnderstanding DeepSeek/DeepResearchExplainersLanguage Models & Co. article - The Illustrated DeepSeek-R1Towards Data Science article - DeepSeek-V3 Explained 1: Multi-head Latent AttentionJina.ai article - A Practical Guide to Implementing DeepSearch/DeepResearchHan, Not Solo blogpost - The Differences between Deep Research, Deep Research, and Deep ResearchAnalysis and ResearchPreprint - Understanding R1-Zero-Like Training: A Critical PerspectiveBlogpost - There May Not be Aha Moment in R1-Zero-like Training — A Pilot StudyPreprint - Large Language Monkeys: Scaling Inference Compute with Repeated SamplingPreprint - Chain-of-Thought Reasoning In The Wild Is Not Always FaithfulFallout coverageTechCrunch article - OpenAI calls DeepSeek 'state-controlled,' calls for bans on 'PRC-produced' modelsThe Verge article - OpenAI has evidence that its models helped train China’s DeepSeekInteresting Engineer article - $6M myth: DeepSeek’s true AI cost is 216x higher at $1.3B, research revealsArs Technica article - Microsoft now hosts AI model accused of copying OpenAI dataThe Signal article - Nvidia loses nearly $600 billion in DeepSeek crashYahoo Finance article - The 'Magnificent 7' stocks are having their worst quarter in more than 2 yearsReuters article - Microsoft pulls back from more data center leases in US and Europe, analysts sayUS governanceNational Law Review article - Three States Ban DeepSeek Use on State Devices and NetworksCNN article - US lawmakers want to ban DeepSeek from government devicesHouse bill - No DeepSeek on Government Devices ActSenate bill - Decoupling America's Artificial Intelligence Capabilities from China Act of 2025LeaderboardsaiderLiveBenchLM ArenaKonwinski PrizePreprint - SWE-Bench+: Enhanced Coding Benchmark for LLMsCybernews article - OpenAI study proves LLMs still behind human engineers in over 1400 real-world tasksOther ReferencesAnthropic report - The Anthropic Economic IndexMETR Report - Measuring AI Ability to Complete Long TasksThe Information article - OpenAI Discusses Building Its First Data Center for StorageDeepmind report backing up this ideaTechCrunch article - OpenAI adopts rival Anthropic's standard for connecting AI models to dataReuters article - OpenAI, Meta in talks with Reliance for AI partnerships, The Information reports2024 AI Index reportNDTV article - Ghibli-Style Images To Memes: White House Embraces Alt-Right Online CultureElk post on DOGE and AI
    Show More Show Less
    1 hr and 32 mins
  • DeepSeek Minisode

    DeepSeek Minisode
    Feb 10 2025

    DeepSeek R1 has taken the world by storm, causing a stock market crash and prompting further calls for export controls within the US. Since this story is still very much in development, with follow-up investigations and calls for governance being released almost daily, we thought it best to hold of for a little while longer to be able to tell the whole story. Nonetheless, it's a big story, so we provide a brief overview of all that's out there so far.

    • (00:00) - Recording date
    • (00:04) - Intro
    • (00:37) - DeepSeek drop and reactions
    • (04:27) - Export controls
    • (08:05) - Skepticism and uncertainty
    • (14:12) - Outro


    Links
    • DeepSeek website
    • DeepSeek paper
    • Reuters article - What is DeepSeek and why is it disrupting the AI sector?

    Fallout coverage

    • The Verge article - OpenAI has evidence that its models helped train China’s DeepSeek
    • The Signal article - Nvidia loses nearly $600 billion in DeepSeek crash
    • CNN article - US lawmakers want to ban DeepSeek from government devices
    • Fortune article - Meta is reportedly scrambling ‘war rooms’ of engineers to figure out how DeepSeek’s AI is beating everyone else at a fraction of the price
    • Dario Amodei's blogpost - On DeepSeek and Export Controls
    • SemiAnalysis article - DeepSeek Debates
    • Ars Technica article - Microsoft now hosts AI model accused of copying OpenAI data
    • Wiz Blogpost - Wiz Research Uncovers Exposed DeepSeek Database Leaking Sensitive Information, Including Chat History

    Investigations into "reasoning"

    • Blogpost - There May Not be Aha Moment in R1-Zero-like Training — A Pilot Study
    • Preprint - s1: Simple test-time scaling
    • Preprint - LIMO: Less is More for Reasoning
    • Blogpost - Reasoning Reflections
    • Preprint - Token-Hungry, Yet Precise: DeepSeek R1 Highlights the Need for Multi-Step Reasoning Over Speed in MATH
    Show More Show Less
    15 mins
  • Understanding AI World Models w/ Chris Canal

    Understanding AI World Models w/ Chris Canal
    Jan 27 2025
    Chris Canal, co-founder of EquiStamp, joins muckrAIkers as our first ever podcast guest! In this ~3.5 hour interview, we discuss intelligence vs. competencies, the importance of test-time compute, moving goalposts, the orthogonality thesis, and much more.A seasoned software developer, Chris started EquiStamp as a way to improve our current understanding of model failure modes and capabilities in late 2023. Now a key contractor for METR, EquiStamp evaluates the next generation of LLMs from frontier model developers like OpenAI and Anthropic.EquiStamp is hiring, so if you're a software developer interested in a fully remote opportunity with flexible working hours, join the EquiStamp Discord server and message Chris directly; oh, and let him know muckrAIkers sent you!(00:00) - Recording date (00:05) - Intro (00:29) - Hot off the press (02:17) - Introducing Chris Canal (19:12) - World/risk models (35:21) - Competencies + decision making power (42:09) - Breaking models down (01:05:06) - Timelines, test time compute (01:19:17) - Moving goalposts (01:26:34) - Risk management pre-AGI (01:46:32) - Happy endings (01:55:50) - Causal chains (02:04:49) - Appetite for democracy (02:20:06) - Tech-frame based fallacies (02:39:56) - Bringing back real capitalism (02:45:23) - Orthogonality Thesis (03:04:31) - Why we do this (03:15:36) - Equistamp!LinksEquiStampChris's TwitterMETR Paper - RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertsAll Trades article - Learning from History: Preventing AGI Existential Risks through Policy by Chris CanalBetter Systems article - The Omega Protocol: Another Manhattan ProjectSuperintelligence & CommentaryWikipedia article - Superintelligence: Paths, Dangers, Strategies by Nick BostromReflective Altruism article - Against the singularity hypothesis (Part 5: Bostrom on the singularity)Into AI Safety Interview - Scaling Democracy w/ Dr. Igor KrawczukReferenced SourcesBook - Man-made Catastrophes and Risk Information Concealment: Case Studies of Major Disasters and Human FallibilityArtificial Intelligence Paper - Reward is EnoughWikipedia article - Capital and Ideology by Thomas PikettyWikipedia article - PantheonLeCun on AGI"Won't Happen" - Time article - Meta’s AI Chief Yann LeCun on AGI, Open-Source, and AI Risk"But if it does, it'll be my research agenda latent state models, which I happen to research" - Meta Platforms Blogpost - I-JEPA: The first AI model based on Yann LeCun’s vision for more human-like AIOther SourcesStanford CS Senior Project - Timing Attacks on Prompt Caching in Language Model APIsTechCrunch article - AI researcher François Chollet founds a new AI lab focused on AGIWhite House Fact Sheet - Ensuring U.S. Security and Economic Strength in the Age of Artificial IntelligenceNew York Post article - Bay Area lawyer drops Meta as client over CEO Mark Zuckerberg’s ‘toxic masculinity and Neo-Nazi madness’OpenEdition Academic Review of Thomas PikettyNeural Processing Letters Paper - A Survey of Encoding Techniques for Signal Processing in Spiking Neural NetworksBFI Working Paper - Do Financial Concerns Make Workers Less Productive?No Mercy/No Malice article - How to Survive the Next Four Years by Scott Galloway
    Show More Show Less
    3 hrs and 20 mins