Skip to content

Add low_level public API #1599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/api/low_level.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# `pydantic_ai.low_level`

::: pydantic_ai.low_level
139 changes: 139 additions & 0 deletions docs/low_level.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# Low-Level Model Requests

The low-level module provides direct access to language models with minimal abstraction. These methods allow you to make requests to LLMs where the only abstraction is input and output schema translation, enabling you to request all models with the same API.

These methods are thin wrappers around the [`Model`][pydantic_ai.models.Model] implementations, offering a simpler interface when you don't need the full functionality of an [`Agent`][pydantic_ai.Agent].

The following functions are available:

- [`model_request`][pydantic_ai.low_level.model_request]: Make a non-streamed async request to a model
- [`model_request_sync`][pydantic_ai.low_level.model_request_sync]: Make a synchronous non-streamed request to a model
- [`model_request_stream`][pydantic_ai.low_level.model_request_stream]: Make a streamed async request to a model

## Basic Example

Here's a simple example demonstrating how to use the low-level API to make a basic request:

```python title="low_level_basic.py"
from pydantic_ai.low_level import model_request_sync
from pydantic_ai.messages import ModelRequest

# Make a synchronous request to the model
model_response = model_request_sync(
'anthropic:claude-3-5-haiku-latest',
[ModelRequest.user_text_prompt('What is the capital of France?')]
)

print(model_response.parts[0].content)
#> Paris
print(model_response.usage)
"""
Usage(requests=1, request_tokens=56, response_tokens=1, total_tokens=57, details=None)
"""
```

_(This example is complete, it can be run "as is")_

## Advanced Example with Tool Calling

You can also use the low-level API to work with function/tool calling.

Even here we can use Pydantic to generate the JSON schema for the tool:

```python
from pydantic import BaseModel
from typing_extensions import Literal

from pydantic_ai.low_level import model_request
from pydantic_ai.messages import ModelRequest
from pydantic_ai.models import ModelRequestParameters
from pydantic_ai.tools import ToolDefinition


class Divide(BaseModel):
"""Divide two numbers."""

numerator: float
denominator: float
on_inf: Literal['error', 'infinity'] = 'infinity'


async def main():
# Make a request to the model with tool access
model_response = await model_request(
'openai:gpt-4.1-nano',
[ModelRequest.user_text_prompt('What is 123 / 456?')],
model_request_parameters=ModelRequestParameters(
function_tools=[
ToolDefinition(
name=Divide.__name__.lower(),
description=Divide.__doc__ or '',
parameters_json_schema=Divide.model_json_schema(),
)
Comment on lines +65 to +72
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should aim to make this more ergonomic in the not too distant future, e.g

model_response = await model_request(
    'openai:gpt-4.1-nano',
    'What is 123 / 456?',
    function_tools=[Divide],
)

I don't think the current API prevents having this in the future, but it's worth keeping in mind.

],
allow_text_output=True, # Allow model to either use tools or respond directly
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this help with the example?

),
)
print(model_response)
"""
LowLevelModelResponse(
parts=[
ToolCallPart(
tool_name='divide',
args={'numerator': '123', 'denominator': '456'},
tool_call_id='pyd_ai_2e0e396768a14fe482df90a29a78dc7b',
part_kind='tool-call',
)
],
model_name='gpt-4.1-nano',
timestamp=datetime.datetime(...),
kind='response',
usage=Usage(
requests=1,
request_tokens=55,
response_tokens=7,
total_tokens=62,
details=None,
),
)
"""
```

_(This example is complete, it can be run "as is" — you'll need to add `asyncio.run(main())` to run `main`)_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally I find this sentence contradictory


## When to Use Low-Level API vs Agent

The low-level API is ideal when:

1. You need more direct control over model interactions
2. You want to implement custom behavior around model requests
3. You're building your own abstractions on top of model interactions

For most application use cases, the higher-level [`Agent`][pydantic_ai.Agent] API provides a more convenient interface with additional features such as built-in tool execution, structured output parsing, and more.

## OpenTelemetry Instrumentation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this heading mention logfire?


As with [agents][pydantic_ai.Agent] you can enable OpenTelemetry/logfire instrumentation with just a few extra lines
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this link to the agents page? Would the agents guide make more sense than the API page?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we no longer capitalising logfire?


```python {title="low_level_instrumented.py" hl_lines="1 6 7"}
import logfire

from pydantic_ai.low_level import model_request_sync
from pydantic_ai.messages import ModelRequest

logfire.configure()
logfire.instrument_pydantic_ai()

# Make a synchronous request to the model
model_response = model_request_sync(
'anthropic:claude-3-5-haiku-latest',
[ModelRequest.user_text_prompt('What is the capital of France?')]
)

print(model_response.parts[0].content)
#> Paris
```

_(This example is complete, it can be run "as is")_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about a screenshot?


See [Debugging and Monitoring](logfire.md) for more details.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
See [Debugging and Monitoring](logfire.md) for more details.
See [Debugging and Monitoring](logfire.md) for more details, including how to instrument with plain OpenTelemetry without logfire.

3 changes: 3 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ nav:
- graph.md
- evals.md
- input.md
- low_level.md
- MCP:
- mcp/index.md
- mcp/client.md
Expand Down Expand Up @@ -67,6 +68,7 @@ nav:
- api/usage.md
- api/mcp.md
- api/format_as_xml.md
- api/low_level.md
- api/models/base.md
- api/models/openai.md
- api/models/anthropic.md
Expand Down Expand Up @@ -208,6 +210,7 @@ plugins:
# 3 because docs are in pages with an H2 just above them
heading_level: 3
import:
- url: https://logfire.pydantic.dev/docs/objects.inv
- url: https://docs.python.org/3/objects.inv
- url: https://docs.pydantic.dev/latest/objects.inv
- url: https://dirty-equals.helpmanual.io/latest/objects.inv
Expand Down
14 changes: 4 additions & 10 deletions pydantic_ai_slim/pydantic_ai/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
result,
usage as _usage,
)
from .models.instrumented import InstrumentationSettings, InstrumentedModel
from .models.instrumented import InstrumentationSettings, InstrumentedModel, instrument_model
from .result import FinalResult, OutputDataT, StreamedRunResult, ToolOutput
from .settings import ModelSettings, merge_model_settings
from .tools import (
Expand Down Expand Up @@ -99,7 +99,7 @@ class Agent(Generic[AgentDepsT, OutputDataT]):
model: models.Model | models.KnownModelName | str | None
"""The default model configured for this agent.

We allow str here since the actual list of allowed models changes frequently.
We allow `str` here since the actual list of allowed models changes frequently.
"""

name: str | None
Expand Down Expand Up @@ -224,7 +224,7 @@ def __init__(

Args:
model: The default model to use for this agent, if not provide,
you must provide the model when calling it. We allow str here since the actual list of allowed models changes frequently.
you must provide the model when calling it. We allow `str` here since the actual list of allowed models changes frequently.
output_type: The type of the output data, used to validate the data returned by the model,
defaults to `str`.
instructions: Instructions to use for this agent, you can also register instructions via a function with
Expand Down Expand Up @@ -1512,13 +1512,7 @@ def _get_model(self, model: models.Model | models.KnownModelName | str | None) -
if instrument is None:
instrument = self._instrument_default

if instrument and not isinstance(model_, InstrumentedModel):
if instrument is True:
instrument = InstrumentationSettings()

model_ = InstrumentedModel(model_, instrument)

return model_
return instrument_model(model_, instrument)

def _get_deps(self: Agent[T, OutputDataT], deps: T) -> T:
"""Get deps for a run.
Expand Down
Loading