My Instruments Lied to Me Six Times
Building and Testing LLM Systems in Production, and How to Find Out What Is Actually True
Failed to add items
Sorry, we are unable to add the item because your shopping cart is already at capacity.
Add to basket failed.
Please try again later
Add to Wish List failed.
Please try again later
Remove from Wish List failed.
Please try again later
Follow podcast failed
Unfollow podcast failed
$0.99/mo for 3 months + $20 Audible bonus voucher
Auto-renews at $8.99/mo after 3 months. Cancel anytime.
Offer ends on 28 December, 2026.
My Instruments Lied to Me Six Times
Narrated by:
AI Voice A synthetic voice
Buy Now for $13.73
-
Narrated by:
-
AI Voice A synthetic voice
-
By:
-
Nikolaos Broikos
This title uses virtual voice narration
Virtual voice is computer-generated narration for audiobooks.
Summary
One of them was the strongest finding in the entire project: a perfect result, every trial, in the direction the whole field expects. It was completely false, and the only thing that caught it was opening a file and reading what the model had actually said.
This is the instrument book. How to build a comparison that means something. How to find out what a wrong answer scores on your own tests, before you report a score at all. How to prove a grader can fail before you let it grade anything. Why an evaluator's error rate tells you almost nothing and its error direction tells you everything.
Every method in it is a few hundred lines of ordinary code with no dependencies, and every one of them exists because something went wrong first.
It is also, unusually, a book that documents its own failures at full length: the detector that looked for one set form of words and reported ninety-six fabrications where there were none, the check that compared a string against a pair of values and therefore could never fail, the average built from six outliers of ninety-six. Not as a confession, but because the pattern in how they were found is the most transferable thing here.
For anybody who has to decide something about an AI system and would rather decide it on evidence than on somebody's confidence.
No reviews yet
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.