One retry owner and a total deadline¶
Decided in the release after 1.23.0.
Context and problem statement¶
A call could be retried in three places at once: the provider SDK's own retries, a retrying HTTP transport, and the application's loop around the call. Two retries in each multiply to eight requests, and nobody can say how long a call may take. The only bound was the HTTP client's per-phase timeout, which a slow trickle of bytes never trips. The embedder had its own retry, the generator had none, and the reranker had none, so the same 429 meant three different things.
ADR-0020 gave every failure a kind. This ADR decides who acts on it, and for how long.
Decision drivers¶
- Retries never multiply: a test can count the wire requests a call makes.
- A spent quota is never retried; a busy queue is.
- A caller can say how long a call may take, end to end, and the call ends then.
- A caller's cancellation is never turned into a result.
Decision¶
- One retry owner: the adapter's own loop, keyed on
Failure.kind(adapters._retry, shared by the generator, the embedder and the reranker). The provider SDK stays atmax_retries=0. AWirePolicymaps each retryable kind to aRetryRule(tries, exponential wait with jitter, whether a server hint is honoured and up to what). Onlyrate_limited,overloaded,server,timeoutandtransportcan carry a rule, soquota,authand a request-side kind are never retried by construction. From the SDKs:retry-after-msbeatsRetry-After, andx-should-retry: falseends the retry whatever the kind. - Retries multiply nowhere. A wire retry re-sends one model request. pydantic-ai's output
re-ask is a new model request, with its own wire retries. Fallback (ADR-0022) starts only
after a tier's wire policy gave up. The pydantic-ai generator wraps the call's model in a
WrapperModel, so every model request the agent makes goes through the one loop. Deadline.total_sis a wall-clock bound on the whole call. The facade starts it whengenerateis entered and passes it down onGenRequest.deadline; the adapter ends the call AT it (asyncio.timeout_at), across the slot wait, every request, every backoff and every fallback tier. A backoff that would end past it ends the call at once. An expiry is atimeoutresult withphase="total". A generator that ignores the deadline is cut offDEADLINE_GRACE_Safter it by the facade.Deadline.attempt_sbounds one request and is a retriedtimeout;first_event_sbounds the wait for a streamed reply's first event.- A cancellation propagates.
asyncio.CancelledErroris never retried, never caught, and never a result; the limiter slot is released on the way out. - Concurrency is a slot taken inside the deadline and outside the retry loop (
Limiter,call.resilience.LocalLimiter). The wait spends the caller's deadline and is reported aslimiter_wait_s; a retry keeps its slot. Waiting pastmax_wait_sisrate_limitedwithprovider_code="local_limiter", not retried. - One HTTP client per generator per event loop, reused across calls and closed by
aclose(); the per-request bound is set per call. - The generator retries by default (
WirePolicy.default()). The embedder and reranker keep their 1.x default of one request;RetryPolicy(1.x) isRetryPolicy.wire().
Considered options¶
- pydantic-ai's
AsyncHTTPX2TenacityTransport. It retries below the classification, so it cannot tell a spent budget from a busy slot, and its attempts would be invisible toGenerated.attempts. Not installed; its header precedence is taken. - The SDK's own retries. Same blindness, and they stack under any loop above them.
- pydantic-ai's
request_timeoutas the per-request bound. The agent graph applies it around the wrapper's whole retry loop, not one request, so the adapter enforcesattempt_sitself. - A distributed limiter in the library. A gateway already is one (ADR-0018 keeps scheduling with the consumer); the in-process limiter only stops one process queueing behind itself.
Consequences¶
- A call's
attemptslists every retried request (kind, wait), each tagged with its tier;stamp["wire_requests"]andstamp["limiter_wait_s"]are on every result. llm_wire_retry,llm_deadline_expiredandllm_fallbackare events a consumer can count.- A caller of the generator who wants 1.x's single request passes
policy=WirePolicy.none(). - A cancelled or timed-out request may still be billed;
Role.stream_internallystreams it so closing the connection is a signal the server can act on.