Discussion about this post

User's avatar
Raghav Mehta's avatar

The excerpt says you cannot read every output but still need to recognize the common ways the agent fails. How would you sample emails for that review so rare cases like long quoted threads or multilingual signatures actually show up instead of mostly clean ones? Stratifying the sample by thread length and language seems like a reasonable starting point.

Justin Lassiter's avatar

nobody wants to read the traces and it's the only thing that ever actually tells you what's wrong, like we had an agent summarizing client emails last week and the summary kept pulling in the signature block and nobody noticed for days because nobody looked. the three gulfs thing gives me a name for that at least, now I can say oh that's a specification problem and not just the AI being weird

10 more comments...

Ready for more?