A Chat port beside Generator: messages in, events and new messages out, tools run by the adapter¶
Context and problem statement¶
Generator is one prompt in, one value out. A conversation that calls tools over several model
requests, and that a person watches arrive token by token, did not fit it: each consumer that
needed one would have written its own loop over a vendor SDK, with its own tool-call parsing,
its own handling of a tool that raises, and its own idea of when the loop stops. Those loops are
where a run that never ends, a tool result pasted into user text, and a failure raised across the
port all come from.
pydantic-ai already runs this loop (Agent.iter, tool calls, parallel execution, retries of a
malformed call, streamed deltas). What it does not give is the package's contract: the named
outcome, the usage and cost result, the one retry owner, the total deadline, the limiter, and an
event stream that ends with the result in every case.
Decision¶
- A second port,
Chat, besideGenerator.stream(req)yieldsTextDelta,ReasoningDelta,ToolCallStarted,ToolCallFinished,StepFinished, andFinished(result)exactly once and last.chat(req)isstreamdrained. The port is stateless: the result carriesmessages, the new turns, and the caller appends them to its own history. - Tools are values the adapter runs.
Tool(name, description, parameters, handler, timeout_s, untrusted_output, max_output_chars)with an async handler returningToolOutput. A handler that raises, times out or returns another type is an error result the model sees, never an exception. An output over its budget is cut in the middle ([... N chars omitted]) and markedtruncated. An untrusted output reaches the model through the provider's tool-result channel inside a<tool_output ... untrusted="true">marker; it is never concatenated into user or system text. max_stepsbounds the loop. It is checked before each model request; a run past it istruncatedwithprovider_code="max_steps", and its messages keep the tool results made.- ADR-0001 holds for streams. A model, wire, tool or gate failure ends the stream with
Finishedcarrying the failure; one after output began hasin_stream=Trueand finishinterrupted. Closing the iterator or cancelling the task stops the call (the producer task is cancelled and awaited) andasyncio.CancelledErrorpropagates. - The reference adapter is pydantic-ai's loop.
PydanticAIChatdrivesAgent.iterand sharesAdapterCorewithPydanticAIGenerator: the wire policy, the deadline clocks, the limiter slot, the failure classification and the result contract are one implementation. - The call path is
call.chat/call.stream. Capability check (tools,json_schema,vision), admit gate, cache plan and pre-flight (ADR-0024), one trace observation with a child span per tool, graders on anokreply. A conversation has no fallback chain. - Conformance.
testing.StubChatplays a script ofTurns and really runs the tools;testing.conformance.check_chatholds the stub and the adapter to the same eleven cases.
Considered options¶
- Extend
Generatorwith a message list and tools. Rejected: everyGeneratorwould have to implement the loop or refuse it, andgenerate's single-value contract would gain a second meaning. - Let the consumer run the loop over
Generator. Rejected: the loop is the hard part, and two consumers would write it twice. - Validate tool arguments against the full JSON schema. Not taken: pydantic-ai's
Tool.from_schemapasses arguments through; the adapter checks the schema'srequiredkeys and re-asks the model when one is missing. A handler validates the rest.
Consequences¶
tool_choice="required"is not sent (pydantic-ai refuses it statically); the result carries aCallWarning."none"and a tool name restrict the tools offered.Message.cacheis honoured on user messages only; a cache point elsewhere has no wire form in pydantic-ai's model messages.- The wire capture skips a
text/event-streambody (reading it would buffer the stream); its usage and finish come from pydantic-ai's own parse of the stream. - Langfuse records tool names and durations in the generation's metadata, not as child spans;
OTel records an
execute_tool {name}child span per tool.