How good is AI at Hacking? Benchmarking Models for Vulnerability Detection with Philippe Dourassov cover art

How good is AI at Hacking? Benchmarking Models for Vulnerability Detection with Philippe Dourassov

How good is AI at Hacking? Benchmarking Models for Vulnerability Detection with Philippe Dourassov

Listen for free

View show details

This Season's Biggest Deal | $0.99/mo for 3 months + $20 Audible bonus

Auto-renews at $8.99/mo. after 3 months - terms apply.
Summary

Are frontier AI models really the best at hacking and finding security vulnerabilities? In this episode of Bad Dependencies, host Mackenzie Jackson speaks with Philippe Dourassov, head of technical AI pentesting at Aikido, to break down real-world AI benchmarking results.Philippe explains how they evaluate models by testing whether agents can rediscover recently disclosed CVEs in open-source software without prior training data or cheating. You will learn why an agent harness is critical for exploring codebases and orchestrating runs, and why massive models like Mythos or GPT-6 Astra, despite high scores, might not be the most practical choice.Discover how running cheaper open-weight models like DeepSeek V4 Pro across multiple non-deterministic attempts beats expensive frontier models at a fraction of the cost, completely changing the economics of offensive security.

No reviews yet
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.