Cloud Run Jobs Deployment Script Pattern

Deploying dbt on Cloud Run Jobs involves multiple GCP services that need to be configured in the right order: service accounts, IAM bindings, Artifact Registry, the job itself, and a scheduler. Doing this manually through the console is error-prone and unrepeatable. A deployment script captures every step as code.

The script below deploys a complete dbt Cloud Run Jobs setup from scratch. Run it once for initial setup, then evolve individual components as your needs change. The || true guards on create commands make the script idempotent — safe to rerun without failing on resources that already exist.

The Complete Script

#!/bin/bash
set -e

export PROJECT_ID=your-project-id
export REGION=us-central1
export SA_NAME=dbt-runner
export SA_EMAIL=$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com
export IMAGE=$REGION-docker.pkg.dev/$PROJECT_ID/dbt-images/dbt-runner:v1.0.0

# --- Service Account ---
gcloud iam service-accounts create $SA_NAME \
    --display-name="dbt Cloud Run Runner" \
    --description="Service account for dbt Cloud Run Jobs" \
    --project=$PROJECT_ID || true

# --- IAM Bindings ---
# BigQuery permissions for dbt execution
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SA_EMAIL" \
    --role="roles/bigquery.dataEditor"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SA_EMAIL" \
    --role="roles/bigquery.jobUser"

# --- Artifact Registry ---
gcloud artifacts repositories create dbt-images \
    --repository-format=docker \
    --location=$REGION \
    --project=$PROJECT_ID || true

# --- Build and Push Image ---
gcloud builds submit \
    --tag $IMAGE \
    --project=$PROJECT_ID

# --- Cloud Run Job ---
gcloud run jobs create dbt-daily \
    --image=$IMAGE \
    --region=$REGION \
    --service-account=$SA_EMAIL \
    --memory=2Gi \
    --cpu=2 \
    --max-retries=2 \
    --task-timeout=3600 \
    --set-env-vars="GCP_PROJECT=$PROJECT_ID,DBT_DATASET=analytics,BQ_LOCATION=US" \
    --project=$PROJECT_ID

# --- Scheduler Service Account ---
gcloud iam service-accounts create dbt-scheduler \
    --display-name="dbt Scheduler Invoker" \
    --project=$PROJECT_ID || true

gcloud run jobs add-iam-policy-binding dbt-daily \
    --region=$REGION \
    --member="serviceAccount:dbt-scheduler@$PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/run.invoker" \
    --project=$PROJECT_ID

# --- Cloud Scheduler ---
gcloud scheduler jobs create http dbt-daily-schedule \
    --location=$REGION \
    --schedule="0 6 * * *" \
    --uri="https://$REGION-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/$PROJECT_ID/jobs/dbt-daily:run" \
    --http-method=POST \
    --oauth-service-account-email=dbt-scheduler@$PROJECT_ID.iam.gserviceaccount.com \
    --project=$PROJECT_ID

echo "Deployment complete. Test with:"
echo "gcloud run jobs execute dbt-daily --region=$REGION --wait"

Why Two Service Accounts

The script creates two distinct service accounts with different purposes:

dbt-runner is the identity your dbt code runs as. It gets bigquery.dataEditor and bigquery.jobUser — the minimum permissions needed to create tables and run queries. This is the service account attached to the Cloud Run Job. When dbt connects to BigQuery via method: oauth, it authenticates as this identity through Workload Identity.

dbt-scheduler is the identity Cloud Scheduler uses to invoke the Cloud Run Job. It gets only roles/run.invoker on the specific job — the permission to trigger execution, nothing more. It cannot read BigQuery data, cannot modify tables, cannot do anything except start the dbt job.

Separating these follows least privilege. If the scheduler service account were compromised, the attacker could trigger dbt runs (annoying, but limited impact). They couldn’t access your data warehouse. Conversely, the dbt-runner account can access BigQuery but cannot trigger its own execution — it has no self-invocation capability.

Configuration Choices

Memory and CPU. The script sets --memory=2Gi and --cpu=2. dbt’s memory usage scales with model complexity and the threads setting in profiles.yml. Each thread maintains a BigQuery connection and processes query results in memory. Start with 2GB and 2 CPUs. If you see OOM (out-of-memory) errors in Cloud Logging, increase memory. If runs are slower than expected, increase both CPUs and threads together — more threads without more CPU just creates contention.

Task timeout. --task-timeout=3600 gives dbt one hour. Set this higher than your longest expected run, with buffer for variance. A project that normally completes in 15 minutes might take 45 minutes during a full refresh or after a long period without incremental processing. The maximum is 168 hours (7 days), but if your dbt run takes that long, you have bigger problems.

Max retries. --max-retries=2 means Cloud Run retries failed executions twice. dbt exits with a non-zero code on failure, which Cloud Run interprets as a retriable error. This handles transient BigQuery failures (quota errors, brief network issues) without custom retry logic. For persistent failures, retries just delay the alert — you still need monitoring.

Environment variables. --set-env-vars passes configuration to the container. The containerized profiles.yml reads these with env_var(). Changing environment variables on the Cloud Run Job doesn’t require rebuilding the image — you can switch datasets or projects with a gcloud run jobs update command.

Testing the Deployment

After running the script, verify with a manual execution:

gcloud run jobs execute dbt-daily --region=us-central1 --wait

The --wait flag blocks your terminal until the job completes, streaming logs as they appear. Watch for:

• Authentication errors — the service account doesn’t have the right BigQuery permissions
• Dataset not found — the target dataset doesn’t exist and dbt doesn’t have bigquery.dataOwner to create it
• OOM killed — the container exceeded its memory limit (increase --memory)
• Timeout — the job hit --task-timeout before completing (increase the timeout)

If the manual execution succeeds, trigger the scheduler to verify the full chain:

gcloud scheduler jobs run dbt-daily-schedule --location=us-central1

Then check the Cloud Run Job execution list to confirm the scheduler-triggered run completed.

Evolving the Script

This script is a starting point. As your deployment matures, consider:

• Moving to Terraform for declarative infrastructure management (the script captures what to deploy; Terraform captures what should exist)
• Adding Eventarc triggers for event-driven execution alongside the scheduled fallback
• Setting up log-based alerting for job failures using gcloud logging metrics create
• Parameterizing the script further (job name, schedule, resource limits) for reuse across multiple dbt projects

The script makes the initial deployment reproducible and documented. Every gcloud command is visible, reviewable, and runnable — no console wizard steps, no undocumented IAM bindings.