Silent Tool Failures in AI Agents — Why Your Agent Lies About Success
Most AI agent bugs never throw an error — the tool call "succeeds" with empty, truncated, or stale data and the agent confidently reports the task as done. Here's how to catch failures that don't look like failures.
Most write-ups on tool reliability focus on the tool call that visibly fails — the thrown exception, the 500 response, the timeout. Those are the easy cases: the agent gets an error string and can retry, ask for help, or give up. The failures that actually cause damage in production are the ones that never throw anything at all.
Step 1: Recognize That "No Error" Doesn't Mean "Correct"
A tool call returning successfully only tells you the code path completed without an exception — it says nothing about whether the content is right.
If customer_id is malformed and the query legitimately returns zero rows, this function returns exactly the same shape as if the customer simply has no orders. The agent sees {"orders": []} in both cases and has no signal that one of them is a bug.
Step 2: Understand Why the Model Papers Over It
LLMs are trained to produce a plausible, helpful-sounding continuation for almost any input they're given — including a tool result that's empty, truncated, or subtly wrong. Nothing in the model's training pushes it to say "this result looks suspicious" unless the prompt or system explicitly asks it to.
That response reads as complete and confident. The user has no reason to doubt it, and the agent has no internal signal that the empty result might be a bug rather than a fact. This is the core mechanic behind silent failures: the model's fluency actively hides the failure instead of surfacing it.
Step 3: Add Sentinel Checks on the Tool's Own Output
Before a tool result reaches the model's context, validate that it's shaped the way a correct result should be shaped — not just that the call didn't throw.
The extra existence check turns an ambiguous empty array into an explicit, distinguishable signal. The agent can now tell "this customer has zero orders" apart from "this customer_id doesn't exist," and only one of those should be reported to the user as a fact.
Step 4: Watch for Truncation, Not Just Emptiness
Empty results are the obvious case. Truncated results are more dangerous because they look complete.
If the upstream API has a response size limit and silently cuts the document off mid-sentence, this function returns a perfectly well-formed dict with a truncated string inside it. Nothing errors. The agent summarizes the partial document as if it were the whole thing.
A length check against a known limit doesn't guarantee you catch every truncation case, but it converts a category of failure that was completely invisible into one the agent can flag to the user instead of confidently summarizing incomplete data.
Step 5: Trace the Full Input/Output Pair, Not a Pass/Fail Flag
When an agent reports false success, the only way to find out what actually happened is to look at what the tool actually returned — which means your tracing has to log the real payload, not just whether the call succeeded.
A trace that only records {"tool": "get_recent_orders", "status": "success"} tells you the call didn't throw and nothing else. A trace that records the actual {"orders": []} payload lets you go back after a user complaint and see exactly what the model was working from — which is usually the fastest way to tell a real "no orders" case apart from a bug that returned an empty array by mistake.
Step 6: Add an Eval That Compares Claims Against Trace Data
Sentinel checks catch the failures you anticipated. The ones you didn't anticipate still need a way to surface, and manually reading every trace doesn't scale.
This kind of check won't be exhaustive, but running a handful of claim-vs-trace comparisons across recent sessions catches the specific pattern that matters most: the agent stating something confidently that the tool data doesn't actually support.
Key Takeaways
Silent tool failures don't show up as exceptions, status codes, or anything a standard try/catch will ever see — the tool call completes normally, and the model's fluency turns wrong, empty, or truncated data into a confident-sounding response with no visible warning sign. Catching this failure mode means validating a tool's output shape and content before it reaches the model's context, distinguishing "legitimately empty" from "broken" with explicit checks, and tracing the actual input/output payload — not a pass/fail flag — so a false claim of success can be traced back to the exact tool call that produced it.
