Production AI agents fail on tool calls 3–15% of the time. That's not a failure rate you fix — it's a reality you design around.
The teams that have designed around it have circuit breakers: token budgets, retry limits, cost anomaly alerts wired to incident response.
The teams that haven't find out from their AWS bill.
This article is about the reliability infrastructure between those two outcomes.
The Retry Loop Failure Mode
When an AI agent calls a tool and gets an ambiguous response — not an error, not a success, just something unexpected — most agents do what they're designed to do: they try again. And again. And again.
Without a hard retry limit, this becomes a loop. Without a token budget cap, the loop has no ceiling. Without observability instrumentation specific to retry signatures, your standard dashboards show nothing unusual until the cost spike appears.
In documented production deployments, the cost spike is the first operational signal that something has gone wrong. By that point, if the agent has write permissions and has queued remediation actions, the incident may have worsened before anyone noticed the loop.
This is the reliability problem behind the cost problem. The bill is the symptom. The missing circuit breaker is the cause.
Why Standard SLIs Don't Catch It
Request latency: normal. The agent is responding within SLO. Error rate: zero. Every call returns something — just not what the agent expected. Availability: 100%. The agent is up and running.
The retry loop produces none of the infrastructure-layer signals your existing alerts are watching.
What it does produce is a Tool Invocation Efficiency (TIE) anomaly — your agent is making 4, 6, 8 tool calls per task when its baseline is 2. That ratio climbing is your early warning. It fires before the billing cycle closes. It fires before the incident escalates.
This is why TIE is a first-class SLI in the agentsre library. It catches what latency and error rate miss.
The Three Circuit Breakers
Every production AI agent needs three reliability controls specifically for the retry loop failure mode:
1. Hard Token Budget Per Session
Set a maximum token count per agent session. Not a soft recommendation in the system prompt — a hard limit enforced at the infrastructure layer. When the agent hits the limit, it stops executing and routes to your escalation path.
The budget should be sized at 3x your P95 task token usage. A task that normally uses 2,000 tokens gets a 6,000-token ceiling. Anything above that is a signal, not normal operation.
from agentsre import AgentSLICollector, TaskRecord
# Track token usage as part of your task record
collector.record(TaskRecord(
task_id="t-001",
task_class="incident-analysis",
tool_calls=8, # elevated — baseline is 2.3
decision_confidence=0.71,
completed=True,
))
# TIE will catch the retry signature before the bill does
results = collector.collect("incident-analysis")
for r in results:
if r.breached:
trigger_circuit_breaker(r)
2. Retry Loop Signature in Observability
A retry loop has a distinctive signature: tool call count per task climbing above baseline, task completion time extending beyond P99, and decision confidence declining across sequential attempts.
Configure a CloudWatch alarm on TIE drift: when tool calls per task exceed 2x baseline for 10 consecutive minutes, fire an alert. This is your early warning before the cost spike and before the incident escalates.
# CloudWatch alarm for retry loop detection
aws cloudwatch put-metric-alarm \
--alarm-name "AgentRetryLoopDetected" \
--metric-name "ToolInvocationEfficiency" \
--namespace "AgentReliability" \
--statistic Average \
--period 300 \
--threshold 2.0 \
--comparison-operator GreaterThanThreshold \
--evaluation-periods 2 \
--alarm-actions arn:aws:sns:REGION:ACCOUNT:AgentAlerts
3. Cost Anomaly as Incident Trigger
Wire your AWS Cost Anomaly Detection to your incident management system. An AI agent whose cost per hour doubles is experiencing a reliability event — treat it as one.
Set a cost anomaly threshold at 150% of your rolling 7-day average for the relevant Lambda functions and Bedrock invocations. When it fires, it routes to the same on-call channel as your availability alerts — because it is an availability signal.
The Numbers Behind This
40% of agentic AI projects are expected to be cancelled by 2027. Cost overruns and inadequate risk controls rank in the top three reasons. These are not independent failure modes — they're the same failure mode at different stages of the same incident.
The retry loop causes the cost overrun. The missing circuit breaker causes the retry loop. The missing circuit breaker exists because teams treat AI agent reliability as an application problem rather than an infrastructure problem requiring SRE governance.
What To Do Before Your Next Agent Goes Live
Three checks before any AI agent touches production:
Check 1: Does this agent have a hard token budget enforced at the infrastructure layer? Not a prompt instruction — a hard limit.
Check 2: Is TIE instrumented per task class with a 2x-baseline breach alert configured?
Check 3: Is cost anomaly detection wired to your incident management system for this agent's associated AWS resources?
If any answer is no — the agent is not production-ready. It is demo-ready.
The circuit breaker for the retry loop costs an afternoon to build. The absence of it costs the project.
Open-source implementation: github.com/Ajay150313/agentsre — the agentsre library instruments TIE, DQR, HER, and AQDD out of the box with AWS CloudWatch integration.
What ceiling do you have today when an agent starts looping?
Top comments (0)