dlt Authentication Patterns

Authentication is the most API-specific part of any pipeline. dlt ships with implementations for the patterns you’ll encounter in real production APIs, and an extension point for the ones that don’t fit.

Built-In Auth Strategies

All auth classes live in dlt.sources.helpers.rest_client.auth. Import what you need:

from dlt.sources.helpers.rest_client.auth import (
    BearerTokenAuth,
    ApiKeyAuth,
    HttpBasicAuth,
    OAuth2ClientCredentials
)

BearerTokenAuth — The most common pattern for modern APIs. Adds an Authorization: Bearer <token> header to every request.

client = RESTClient(
    base_url="https://api.example.com",
    auth=BearerTokenAuth(token=dlt.secrets["api_token"])
)

The dlt.secrets["api_token"] pattern reads from your secrets configuration — never hardcode tokens. See dlt Secrets Management for where to store them and the priority order dlt uses to find them.

ApiKeyAuth — For APIs that expect an API key in a header or query parameter.

# Key in header
client = RESTClient(
    base_url="https://api.example.com",
    auth=ApiKeyAuth(
        name="X-API-Key",
        api_key=dlt.secrets["api_key"],
        location="header"
    )
)

# Key in query parameter
client = RESTClient(
    base_url="https://api.example.com",
    auth=ApiKeyAuth(
        name="api_key",
        api_key=dlt.secrets["api_key"],
        location="query"
    )
)

HttpBasicAuth — Username and password authentication. Less common in modern APIs, but still present in internal tools, legacy systems, and some enterprise SaaS products.

client = RESTClient(
    base_url="https://api.example.com",
    auth=HttpBasicAuth(
        username=dlt.secrets["username"],
        password=dlt.secrets["password"]
    )
)

OAuth2ClientCredentials — Server-to-server OAuth2. The client exchanges client_id and client_secret for an access token, then uses that token for API requests. dlt handles token acquisition and expiry automatically.

client = RESTClient(
    base_url="https://api.example.com",
    auth=OAuth2ClientCredentials(
        access_token_url="https://api.example.com/oauth/token",
        client_id=dlt.secrets["client_id"],
        client_secret=dlt.secrets["client_secret"]
    )
)

This is the flow used by most enterprise APIs: Google Ads, Salesforce, many CRM tools. The client credentials flow (as opposed to the authorization code flow) is appropriate for machine-to-machine access without user involvement.

Declarative Auth in REST API Source

When using REST API Source, auth is configured inside the client block of your config dictionary:

config = {
    "client": {
        "base_url": "https://api.example.com/v1",
        "auth": {
            "type": "bearer",
            "token": dlt.secrets["api_token"]
        }
    },
    ...
}

The type field maps to the built-in auth classes: "bearer", "api_key", "http_basic", "oauth2_client_credentials". REST API Source applies the auth configuration to every endpoint in the source unless overridden at the endpoint level.

Extending for Non-Standard Flows

Real APIs don’t always follow standard patterns. Refresh token rotation, custom grant types, multi-step auth flows, tokens embedded in request bodies — these exist in the wild.

When you hit a non-standard flow, extend the closest base class and override only what differs:

from dlt.sources.helpers.rest_client.auth import OAuth2ClientCredentials

class CustomOAuth(OAuth2ClientCredentials):
    def build_access_token_request(self):
        # Override token request construction for non-standard grant type
        return {
            "grant_type": "custom_grant",
            "client_id": self.client_id,
            "client_secret": self.client_secret,
            "scope": "read:all"
        }

You inherit all the token caching and expiry handling from the base class. Your override only changes the specific behavior that’s non-standard. This is significantly less work than implementing auth from scratch, and the result stays within dlt’s machinery rather than becoming a bespoke wrapper.

Auth in Practice

A few things to watch for:

Don’t hardcode credentials. Ever. Use dlt.secrets to pull from your secrets configuration. dlt will tell you exactly which key it expected when a secret is missing — the error messages are specific enough to fix immediately.

Tokens can expire mid-run. For long-running pipelines that page through millions of records, access tokens issued at startup may expire before the pipeline finishes. OAuth2ClientCredentials handles token refresh automatically. If you’re using BearerTokenAuth with a short-lived token, you may need to move to OAuth2 to support multi-hour runs.

Test auth first. Before building pagination and incremental loading on top, verify that your auth configuration returns a valid response. A single client.get("/some-endpoint") call confirms auth is working before you invest time in the rest of the pipeline.