Dagster Resources

Resources are Dagster’s mechanism for managing external connections. A BigQueryResource, a DbtCliResource, a GCS client, an API key — anything assets need to interact with the outside world is configured as a resource. Resources are defined centrally in the Definitions object and injected into assets at runtime.

Asset code declares what resources it needs via type-annotated function arguments; Dagster provides the right instance based on the environment. The asset does not construct its own database connection or read environment variables directly.

The Basic Pattern

Resources are configured once in Definitions and injected into any asset that requests them:

import dagster as dg
from dagster_dbt import DbtCliResource
from dagster_gcp import BigQueryResource

defs = dg.Definitions(
    assets=[mrt__finance__daily_revenue, my_dbt_assets],
    resources={
        "bigquery": BigQueryResource(project="my-gcp-project"),
        "dbt": DbtCliResource(project_dir="./transform"),
    },
)

An asset requests a resource by adding it as a type-annotated argument:

@dg.asset
def mrt__finance__daily_revenue(
    context: dg.AssetExecutionContext,
    bigquery: BigQueryResource,
) -> None:
    client = bigquery.get_client()
    # Use the BigQuery client to write results
    client.query("INSERT INTO ...").result()

Dagster matches the argument name bigquery to the key "bigquery" in the resource dictionary. The asset receives the configured BigQueryResource instance without knowing how it was configured. This indirection is the entire point: the asset’s logic doesn’t change when the connection details change.

Environment Swapping

The primary practical benefit of resources is environment management. Development, staging, and production environments typically differ in:

• Which GCP project or BigQuery dataset to target
• Which service account credentials to use
• Whether to use a test dbt profile or the production one
• Connection pool sizes, timeouts, retry policies

With resources, you swap environments by changing the Definitions configuration, not the asset code:

import os

env = os.getenv("DAGSTER_ENV", "dev")

if env == "prod":
    bq_resource = BigQueryResource(project="prod-gcp-project")
    dbt_resource = DbtCliResource(
        project_dir="./transform",
        target="prod",
    )
else:
    bq_resource = BigQueryResource(project="dev-gcp-project")
    dbt_resource = DbtCliResource(
        project_dir="./transform",
        target="dev",
    )

defs = dg.Definitions(
    assets=[...],
    resources={
        "bigquery": bq_resource,
        "dbt": dbt_resource,
    },
)

This is the same principle behind dbt’s profiles.yml with multiple targets — but applied to every external connection in your pipeline, not just the database. Your dbt assets, Python extraction assets, and downstream notification assets all get environment-appropriate connections from the same centralized configuration.

For teams running dbt on BigQuery with separate dev and prod datasets, resources align naturally. The DbtCliResource with target="dev" points dbt at the development dataset. The BigQueryResource for Python assets points at the same project. Everything stays consistent without hardcoded project IDs scattered across asset files.

Common Resources for Analytics Engineers

The resources you’re most likely to use in a dbt + BigQuery + GCP pipeline:

DbtCliResource

The bridge between Dagster and your dbt project. Configured with the project directory and optional target, profile, and CLI flags:

from dagster_dbt import DbtCliResource

dbt_resource = DbtCliResource(
    project_dir="./transform",
    target="prod",
    profiles_dir="./transform",
)

This resource is what the @dbt_assets decorator uses internally to run dbt build. When you call dbt.cli(["build"], context=context) inside your dbt assets function, you’re using this resource.

BigQueryResource

For Python assets that read from or write to BigQuery directly (not through dbt):

from dagster_gcp import BigQueryResource

bq_resource = BigQueryResource(
    project="my-gcp-project",
    location="US",
)

Useful for assets that do things SQL can’t: calling external APIs, running ML inference, generating files. The asset reads from BigQuery via the resource, processes in Python, and writes results back.

GCS Resource

For assets that interact with Cloud Storage — reading raw files, writing exports, staging data for BigQuery loads:

from dagster_gcp import GCSResource

gcs_resource = GCSResource(
    project="my-gcp-project",
)

EnvVar for Secrets

Dagster provides dg.EnvVar for reading secrets from environment variables at runtime rather than at definition time. This is important for deployment where secrets come from Secret Manager or Kubernetes secrets:

defs = dg.Definitions(
    resources={
        "bigquery": BigQueryResource(
            project=dg.EnvVar("GCP_PROJECT"),
        ),
    },
)

EnvVar reads the environment variable when the resource is instantiated at runtime, not when the Python module is imported. This prevents secrets from leaking into logs or being required during dagster dev startup.

Resources vs. Hardcoded Connections

Without resources, the temptation is to create connections inline:

# Don't do this
@dg.asset
def my_asset():
    from google.cloud import bigquery
    client = bigquery.Client(project="prod-gcp-project")  # Hardcoded!
    ...

This works but creates problems:

• Environment coupling. The asset always targets prod. Running it in development requires changing the code.
• Testing friction. Unit tests can’t easily swap in a mock client.
• Connection sprawl. Each asset creates its own client with its own configuration. Connection pooling, retries, and timeouts are configured inconsistently.
• Secret management. API keys and credentials end up in source code or scattered across environment variable reads.

Resources centralize all of this. One place to configure, one place to change, one place to audit for security.

Resources and Components

The Components abstraction uses resources heavily. When you scaffold a DbtProjectComponent, the DbtCliResource is configured in YAML rather than Python:

type: dagster_dbt.DbtProjectComponent
params:
  project_dir: ./transform
  dbt:
    target: prod

Under the hood, the component creates and registers the DbtCliResource for you. This is part of how Components reduce the Python boilerplate — resource configuration moves from Python code to declarative YAML, matching the configuration-over-code philosophy that analytics engineers are used to from dbt’s dbt_project.yml and profiles.yml.

The Mental Model

If you come from dbt, think of resources as the Dagster equivalent of profiles.yml — but generalized to every external system, not just the database. The profile defines where to connect (project, dataset, credentials). The model defines what to produce (the SQL, the dependencies). Resources maintain that same separation across your entire pipeline, not just the dbt portion.