Cloud Run Jobs for dbt

Cloud Run Jobs, combined with Cloud Scheduler for timing and Eventarc for event-driven triggers, delivers scheduling and execution capabilities comparable to managed orchestrators for under $5 monthly — often within the free tier. Execution times up to 168 hours (seven days) accommodate any realistic dbt transformation workload. Projects with hundreds of models running daily typically generate costs under $5 monthly.

Container Setup

Container flexibility gives you full control over the runtime environment: specific dbt-core and adapter versions, Python dependencies, custom packages. A multi-stage Docker build keeps images small while ensuring reproducibility:

FROM python:3.11-slim as builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.11-slim

WORKDIR /app
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin

# Copy dbt project (or clone at runtime)
COPY dbt_project/ ./dbt_project/
COPY profiles.yml ./

ENTRYPOINT ["dbt"]
CMD ["run", "--project-dir", "/app/dbt_project"]

Pin explicit versions in requirements.txt:

dbt-core==1.9.1
dbt-bigquery==1.9.0

Pin versions rather than using latest tags. Deployment variance from floating versions causes more debugging headaches than the convenience justifies. A two-repository approach separates concerns effectively: one repository holds your dbt project (models, tests, macros, documentation), the other holds the Docker image definition and deployment configuration. This separation enables independent development cycles — data analysts iterate on SQL without touching infrastructure, while platform engineers update the runtime without merge conflicts in model files.

Authentication via Workload Identity

Authentication through Workload Identity eliminates service account keys entirely. Your Cloud Run Job’s attached service account uses OAuth automatically. The dbt profiles.yml specifies method: oauth, and the system handles credential management. No keys to rotate, no secrets to leak.

# profiles.yml for Cloud Run Jobs
my_project:
  target: prod
  outputs:
    prod:
      type: bigquery
      method: oauth
      project: my-gcp-project
      dataset: analytics
      threads: 4
      location: US

This is the same ADC resolution chain at work — in Cloud Run, the attached service account’s metadata service provides credentials automatically (step 3 in the chain). No GOOGLE_APPLICATION_CREDENTIALS variable needed, no key file to manage.

For secrets beyond BigQuery authentication (API keys for external packages, GitHub tokens for private packages), store them in Secret Manager. Mount secrets as environment variables in your Cloud Run Job configuration rather than baking them into images.

Scheduling Patterns

Native integrations cover common scheduling patterns without any additional orchestration layer:

Cron-based scheduling. Cloud Scheduler triggers jobs on cron expressions. The scheduler’s service account needs roles/run.invoker on the Cloud Run Job. This handles the vast majority of dbt scheduling needs: daily runs, hourly refreshes, or any fixed cadence.

Event-driven triggers. Eventarc routes events from Cloud Storage uploads or BigQuery audit logs directly to your job. Configure an Eventarc trigger to watch for Cloud Storage object creation in your raw data bucket, filtering on specific prefixes or file patterns. The trigger invokes your Cloud Run Job, which can use the event payload to determine which models need refreshing. This pattern enables “run dbt when source data lands” without polling or intermediate services.

Retry handling. The dbt exit code (non-zero on failure) triggers Cloud Run’s built-in retry mechanism. Setting --max-retries=2 handles transient failures without custom logic. For most teams, this is sufficient error handling — no need for Airflow’s more sophisticated retry policies unless your failure modes are genuinely complex.

Monitoring

Monitoring relies on what Cloud Run provides automatically, which covers most operational needs:

• Container stdout and stderr flow to Cloud Logging
• Cloud Monitoring captures execution counts, durations, and resource utilization
• Configure log-based alerts for severity>=ERROR patterns
• Set metric-based alerts for job failures

The gap compared to Cloud Composer is visibility, not capability. Cloud Logging captures everything, but lacks Airflow’s task-level execution history, Gantt charts, and dependency visualization. For a single dbt project running on a schedule, Cloud Logging is more than adequate. The trade-off becomes real when multiple teams need shared visibility into pipeline status.

Cost Profile

The pricing model is pay-per-execution: compute time while your container runs, plus minimal storage for the container image. There is no idle cost. A typical dbt project running daily — even one with hundreds of models — generates costs under $5 monthly. Many teams stay within free tier limits entirely.

Compare this to Cloud Composer 3’s minimum of $300-400/month running idle, before a single DAG executes. The gap widens further when you account for hidden costs: Composer’s architecture involves multiple GCP services communicating across zones, and without careful configuration, data transfer costs accumulate.

Over a year, the $300-400 monthly savings compounds to $3,600-4,800 — enough to fund additional data engineering work, better tooling, or simply healthier margins. The decision framework helps determine when this cost advantage outweighs the orchestration features you’d be giving up.

When Cloud Run Jobs Falls Short

Cloud Run Jobs handles dbt execution well, but it is not a general-purpose orchestrator. The limitations that drive teams toward more capable tools:

• No backfill mechanism. Airflow’s catchup and backfill command let you rerun pipelines for specific date ranges with proper dependency handling. With Cloud Run Jobs, you’d need to implement backfill logic yourself, typically by parameterizing your dbt invocation and running it in a loop.
• No multi-step orchestration. If a single workflow spans extraction, transformation, validation, and reverse ETL, stitching together multiple Cloud Run Jobs with Cloud Workflows becomes the middle ground — but past three or four steps, Airflow’s DAG model expresses dependencies more cleanly.
• No shared UI for pipeline visibility. Cloud Logging is a search tool, not an operations dashboard. When multiple teams need to see “what ran, when, and did it succeed?” at a glance, Airflow’s native UI provides that without custom dashboarding.

For most dbt-only deployments on GCP, these limitations are theoretical rather than practical. Adding orchestration complexity is warranted only when a genuine constraint is encountered.