A resilience ladder for streaming agent calls¶
Problem¶
A long-running agent call streams messages back. Three different things go wrong, and treating them alike gets one of them wrong:
- an unknown message type appears in the stream — a new event the client library does not model yet
- the subprocess or connection dies mid-stream
- the model cannot complete the task at all
Retrying an unknown message type wastes the whole call for a frame you did not need. Skipping a connection death loses everything. Failing hard on either throws away work that a cheaper path could still finish.
Technique¶
Three rungs, in order, each matched to a failure that behaves differently.
Skip on message-parse errors. Log the frame and keep consuming the stream: one unmodelled event should not discard the messages after it.
Retry on process and connection errors, with exponential backoff. These are
usually transient. Match on the error types the client raises and on the
message text of generic exceptions it re-raises — client libraries routinely
flatten a subprocess crash into a plain exception whose only distinguishing
feature is a string like exit code or command failed.
Fall back when retries are exhausted: a single-turn, non-streaming call that produces something adequate without tools or multi-step reasoning.
The ladder is ordered by cost. Each rung is tried only when the cheaper one does not apply.
When it applies¶
Any streaming call to a subprocess-backed or network-backed agent runtime where partial output is still useful and a degraded result beats no result — batch generation, background enrichment, anything with a human review gate after it.
When it does NOT apply¶
Interactive use. A user waiting on a response is better served by a fast, clear failure than by ninety seconds of silent retries.
Skip-on-parse-error is also wrong where message ordering carries meaning — if a dropped frame can leave the consumer's state machine inconsistent, a parse error is a real error. This ladder assumes messages are independently useful.
Do not add the fallback rung where the fallback cannot produce an acceptable answer. A single-turn call with no tool access is a genuinely different capability, and silently substituting it hides that the task was not done.
Evidence¶
A documentation pipeline began failing on a new stream event the client library did not recognise. Before the ladder, one unknown frame aborted the entire generation. After: the frame is logged and skipped, transient subprocess crashes retry once with backoff, and exhaustion drops to a single-turn call whose output goes to human review like any other.
The retry rung had to match on message text as well as exception type, because the runtime re-raised CLI crashes as plain exceptions. Matching only the typed errors caught none of the crashes actually seen in production.