dbt profiles.yml with env_var for Multi-Client GCP

dbt’s profiles.yml supports Jinja’s env_var() function for reading environment variables at runtime. Combined with direnv, this means a single profiles.yml can serve every client project — dbt picks up the right credentials and project based on which directory you’re in when you run it.

The Core Pattern

Each client gets its own profile block in ~/.dbt/profiles.yml. Instead of hardcoding credentials, every client-specific value is read from environment variables:

acme_analytics:
  target: "{{ env_var('DBT_TARGET', 'dev') }}"
  outputs:
    dev:
      type: bigquery
      method: service-account
      project: "{{ env_var('GOOGLE_CLOUD_PROJECT') }}"
      dataset: dbt_dev
      keyfile: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"
      threads: 4
      location: EU
    prod:
      type: bigquery
      method: service-account
      project: "{{ env_var('GOOGLE_CLOUD_PROJECT') }}"
      dataset: analytics
      keyfile: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"
      threads: 8
      location: EU

globex_warehouse:
  target: "{{ env_var('DBT_TARGET', 'dev') }}"
  outputs:
    dev:
      type: bigquery
      method: service-account
      project: "{{ env_var('GOOGLE_CLOUD_PROJECT') }}"
      dataset: dbt_dev
      keyfile: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"
      threads: 4
      location: US
    prod:
      type: bigquery
      method: service-account
      project: "{{ env_var('GOOGLE_CLOUD_PROJECT') }}"
      dataset: analytics
      keyfile: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"
      threads: 8
      location: US

Every profile reads from the same environment variables: GOOGLE_CLOUD_PROJECT, GOOGLE_APPLICATION_CREDENTIALS, and DBT_TARGET. direnv sets different values for these variables in each client directory. When you switch directories, dbt automatically targets a different project with different credentials.

How the Switching Works

In Client A’s directory:

# direnv loads:
export GOOGLE_CLOUD_PROJECT=acme-analytics-prod
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.gcp-keys/acme-dbt-runner.json"
export DBT_TARGET=prod

dbt run --profile acme_analytics
# Reads from GOOGLE_CLOUD_PROJECT → acme-analytics-prod
# Uses keyfile from GOOGLE_APPLICATION_CREDENTIALS → acme key
# Targets prod output because DBT_TARGET=prod

In Client B’s directory:

# direnv loads different values:
export GOOGLE_CLOUD_PROJECT=globex-warehouse-prod
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.gcp-keys/globex-dbt-runner.json"
export DBT_TARGET=prod

dbt run --profile globex_warehouse
# Reads from GOOGLE_CLOUD_PROJECT → globex-warehouse-prod
# Uses keyfile from GOOGLE_APPLICATION_CREDENTIALS → globex key
# Same command, different client

The profile names (acme_analytics, globex_warehouse) are static — you reference them in dbt_project.yml. The credentials they use are dynamic, driven by environment variables.

Auth Method: Why service-account Over oauth

Two methods are commonly used for BigQuery connections in dbt:

method: oauth uses your personal Google identity via Application Default Credentials. It works well for single-project development and is the default recommendation in the dbt BigQuery quickstart.

method: service-account uses an explicit service account key file via the keyfile property. This is the consulting-friendly choice.

The problem with oauth in a multi-client setup is that it relies on the ADC file (~/.config/gcloud/application_default_credentials.json), which is managed by gcloud and tied to your personal OAuth session. Managing which ADC file is active across multiple clients is messy — it’s one of the things CLOUDSDK_CONFIG solves, but even with isolation, ADC sessions can expire and require browser-based re-authentication.

The oauth method also completely breaks in non-interactive environments: CI/CD pipelines, containers, and AI agent sessions that can’t perform browser flows.

service-account with env_var('GOOGLE_APPLICATION_CREDENTIALS') works in every context. The key file is explicit, durable, and doesn’t require interactive re-authentication. It inherits the path from environment variables, which direnv manages. And it works identically whether you’re running dbt run manually, from a Cloud Function, or inside a Claude Code session.

The env_var Default Value

env_var() accepts an optional second argument as a default value:

target: "{{ env_var('DBT_TARGET', 'dev') }}"

This means if DBT_TARGET isn’t set (e.g., you’re working outside a direnv-managed directory), dbt falls back to the dev target. This is a safety net: you’d rather accidentally run dev than accidentally run prod.

For GOOGLE_CLOUD_PROJECT and GOOGLE_APPLICATION_CREDENTIALS, don’t provide defaults. If those variables aren’t set, you want dbt to fail loudly with a missing variable error rather than silently using a stale global credential.

Profiles File Location

By default, dbt looks for profiles.yml in ~/.dbt/. You can override this with --profiles-dir or the DBT_PROFILES_DIR environment variable.

For consulting setups, a single ~/.dbt/profiles.yml with all client profiles is usually the cleanest approach. The profile blocks are static (only names and non-credential configuration like threads and location); the credentials themselves come from environment variables.

An alternative is keeping profiles.yml inside the project repository and including it in .gitignore. This works if you want the profile definition to live with the code. The env_var pattern applies either way — you just set DBT_PROFILES_DIR to the project directory instead.

CI/CD Compatibility

The same profiles.yml works in CI/CD without modification. Your CI pipeline sets GOOGLE_CLOUD_PROJECT, GOOGLE_APPLICATION_CREDENTIALS, and DBT_TARGET as pipeline secrets or environment variables. dbt reads them identically to how it reads them locally from direnv.

This is the main payoff of the pattern: local development and CI/CD use the same configuration mechanism. There’s no separate profiles.yml for local and production, no conditional logic, no environment-specific files to keep in sync.