Deploying dbt Core on Cloud Functions

This hub covers deploying dbt Core on Google Cloud Functions — a serverless option for running dbt on a schedule without managing containers or servers. It covers the full path from an existing dbt project to a scheduled function.

Prerequisites

Before starting, you need:

• A working dbt Core project connected to BigQuery (see dbt Project Structure and Naming for project layout conventions)
• Access to a GCP project with billing enabled
• The gcloud CLI installed and authenticated locally

The Setup Path

Step 1: Decide whether Cloud Functions is right for you.

Cloud Functions as a dbt Execution Environment compares Cloud Functions to Cloud Run Jobs and explains when each makes sense. The short version: Cloud Functions is faster to set up but has a 60-minute timeout ceiling and installs dbt packages on every execution. If your project is small and you want a working schedule today, Cloud Functions is a reasonable starting point.

Step 2: Restructure the repository.

Cloud Functions expects main.py and requirements.txt at the root of your source directory. This conflicts with standard dbt project layout, so you move the dbt files into a subdirectory. dbt Repository Structure for Cloud Function Deployment covers the target structure, the main.py entry point code, and the profiles.yml configuration with method: oauth.

Step 3: Set up the service account.

dbt needs IAM permissions to read from source projects and write to the transform project. If your data spans multiple GCP projects, the permissions need to be granted in each. dbt Service Account Setup for Multi-Project GCP Architectures has the full gcloud script to create the service account and assign the right roles across source projects, the transform project, and the Cloud Functions project.

Step 4: Deploy the Cloud Function.

From the root of your repository, deploy with:

gcloud functions deploy dbt_run \
  --region=europe-west1 \
  --service-account=dbt-transform-sa@projectid.iam.gserviceaccount.com \
  --gen2 \
  --runtime=python310 \
  --entry-point=run_dbt \
  --trigger-http \
  --timeout=3500 \
  --memory=1G

Use gen2 (2nd generation) functions — they support longer timeouts and better observability than gen1. Choose a region where gen2 is available; the GCP documentation has the current list.

After deployment, the gcloud functions describe output includes the HTTP trigger URL you’ll need for Step 5.

Step 5: Schedule with Cloud Scheduler.

Cloud Scheduler OIDC Authentication for HTTP Triggers covers how to set up a Cloud Scheduler job that authenticates to the Cloud Function using OIDC tokens. The service account needs roles/iam.serviceAccountTokenCreator on itself and roles/cloudfunctions.invoker on the function.

What This Produces

A dbt project running on a schedule, with logs in Cloud Logging, costs measured in cents per month, and no persistent infrastructure. The function runs only when triggered. The operational surface is a Python file, a requirements list, and a Cloud Scheduler job. When something fails, the logs are in Cloud Logging and the root cause is usually an IAM permission or a package version mismatch.

If the project outgrows Cloud Functions — run time exceeds 45 minutes, or packages should be baked in rather than installed at runtime — Cloud Run Jobs for dbt is the migration path. The dbt project inside dbt_transform/ stays unchanged; only the execution wrapper changes. dbt Orchestration Decision Framework for GCP maps all GCP options and their trade-offs.