dlt Secrets Management

dlt provides a configuration hierarchy that keeps credentials out of code and allows the same pipeline to run across local development, CI/CD, and production without code changes.

The Priority Order

dlt resolves configuration and secrets using this priority order (highest to lowest):

1. Environment variables
2. secrets.toml
3. config.toml
4. Vault integrations
5. Default argument values

Higher-priority sources override lower ones. In practice: environment variables are used in production and CI/CD (secrets injected by the runtime), secrets.toml is used locally (never committed to version control), and config.toml holds non-sensitive settings that can be committed.

Local Development: secrets.toml

For local development, create a .dlt/secrets.toml file in your project directory:

[sources.my_api]
api_key = "your-secret-key"

[destination.bigquery]
project_id = "your-project"
private_key = "-----BEGIN PRIVATE KEY-----\n..."
client_email = "service-account@project.iam.gserviceaccount.com"

Access these values in your pipeline code with dlt.secrets:

client = RESTClient(
    base_url="https://api.example.com",
    auth=BearerTokenAuth(token=dlt.secrets["sources.my_api.api_key"])
)

Or use the shorthand when the key name is unambiguous:

auth=BearerTokenAuth(token=dlt.secrets.value)  # dlt infers the path from context

Never commit secrets.toml to version control. Add .dlt/secrets.toml to your .gitignore immediately when you create a project.

CI/CD and Production: Environment Variables

In CI/CD pipelines and production environments, use environment variables. dlt maps environment variables to config keys using double underscores as separators:

# API source credentials
export SOURCES__MY_API__API_KEY="your-secret-key"

# BigQuery destination credentials
export DESTINATION__BIGQUERY__PROJECT_ID="your-project"
export DESTINATION__BIGQUERY__PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n..."

The naming convention: SOURCES__ prefix for source credentials, DESTINATION__ prefix for destination credentials, then the nested key path with double underscores.

For GitHub Actions, store these in repository secrets and reference them in your workflow:

- name: Run pipeline
  env:
    SOURCES__MY_API__API_KEY: ${{ secrets.MY_API_KEY }}
    DESTINATION__BIGQUERY__PROJECT_ID: ${{ secrets.BIGQUERY_PROJECT_ID }}
  run: python my_pipeline.py

For Google Cloud environments (Cloud Run, Cloud Functions), use Cloud Secret Manager or Workload Identity Federation instead of environment variables containing raw credentials. dlt has vault integrations that pull secrets from external managers at runtime.

Non-Sensitive Config: config.toml

Settings that aren’t secrets but should be configurable go in .dlt/config.toml:

[pipeline]
log_level = "INFO"
progress = "log"

[sources.my_api]
base_url = "https://api.example.com/v1"
page_size = 100

config.toml can be committed to version control. Keep it free of passwords, tokens, private keys, and anything you wouldn’t want in a public repository.

Accessing Secrets in Code

dlt provides multiple ways to access configuration values:

# Full key path
api_key = dlt.secrets["sources.my_api.api_key"]

# Using dlt.secrets.value with decorator context (dlt infers the path)
@dlt.source
def my_source(api_key=dlt.secrets.value):
    ...

# Using dlt.config for non-secrets
page_size = dlt.config["sources.my_api.page_size"]

When a required secret is missing, dlt raises a specific error message telling you exactly which key was expected and in which format. This makes debugging configuration issues fast — you don’t have to hunt for which environment variable name the library expects.

What This Looks Like End to End

A pipeline designed for secrets management has this structure:

my_pipeline/
├── .dlt/
│   ├── config.toml      # committed — non-sensitive settings
│   └── secrets.toml     # gitignored — local credentials only
├── my_pipeline.py       # committed — no secrets anywhere in the code
└── .gitignore           # includes .dlt/secrets.toml

The Python file references dlt.secrets["..."] throughout. When running locally, dlt reads from secrets.toml. In CI/CD, environment variables provide the same values. In production, environment variables or vault integrations do the same. No code changes across environments — only configuration changes.

This separation is the same principle behind twelve-factor app configuration: store config in the environment, never in the code. dlt’s hierarchy makes it easy to follow this principle without ceremony.