The failure mode that gets underestimated is partial success: valid JSON, confident tone, wrong content. Timeouts and rate limits at least surface as errors you can retry on, while a plausible wrong answer passes straight through unless something downstream validates it. Retry logic needs care too, since retrying a non-idempotent tool call can duplicate the side effect rather than recover from it.
Moving beyond the naive request-prompt-response loop requires treating structural validation as a runtime state machine rather than a simple try-catch. Enforcing deterministic JSON schemas at generation time paired with fallback routing is critical for sustaining production-grade SLAs at scale.
Wasim's point about partial success passing through undetected maps onto something happening in code review. AI-assisted changes wait 5.3 times longer for review than manual ones, and only a third clear review within a month against 85 percent of human-written work. A survey of 200 enterprise leaders found 94 percent rating AI-generated code as higher quality at review time while 78 percent reported production incidents from it. Same shape: the system reports success while being wrong, and nothing fires an exception.
The fix looks the same. One payments company got to a 70 percent acceptance rate on 4,000 automated changes in two months by funding review capacity to match generation, not by using a better model. Your fallback chains and circuit breakers assume the failure announces itself. Semantic failure and bad code share that assumption's blind spot, and both need verification funded ahead of volume, not added after.
The moment the LLM returns invalid JSON is the moment you remember the old rule: never trust a colleague who answers the question you didn't ask. Retrying the same prompt and hoping for a different personality is not resilience, it's optimism with a retry budget.
The failure mode that gets underestimated is partial success: valid JSON, confident tone, wrong content. Timeouts and rate limits at least surface as errors you can retry on, while a plausible wrong answer passes straight through unless something downstream validates it. Retry logic needs care too, since retrying a non-idempotent tool call can duplicate the side effect rather than recover from it.
Moving beyond the naive request-prompt-response loop requires treating structural validation as a runtime state machine rather than a simple try-catch. Enforcing deterministic JSON schemas at generation time paired with fallback routing is critical for sustaining production-grade SLAs at scale.
Wasim's point about partial success passing through undetected maps onto something happening in code review. AI-assisted changes wait 5.3 times longer for review than manual ones, and only a third clear review within a month against 85 percent of human-written work. A survey of 200 enterprise leaders found 94 percent rating AI-generated code as higher quality at review time while 78 percent reported production incidents from it. Same shape: the system reports success while being wrong, and nothing fires an exception.
The fix looks the same. One payments company got to a 70 percent acceptance rate on 4,000 automated changes in two months by funding review capacity to match generation, not by using a better model. Your fallback chains and circuit breakers assume the failure announces itself. Semantic failure and bad code share that assumption's blind spot, and both need verification funded ahead of volume, not added after.
The moment the LLM returns invalid JSON is the moment you remember the old rule: never trust a colleague who answers the question you didn't ask. Retrying the same prompt and hoping for a different personality is not resilience, it's optimism with a retry budget.
LLM failure modes are an ops problem. Cross-asset, what usually matters first is which systems still work when the demo breaks.