GCP authentication when you have 10 clients and an AI agent

I manage about a dozen GCP projects on any given week. Different clients, different billing accounts, different datasets. On a normal morning I’ll cd into one client’s dbt project, run a build, switch to another terminal to query BigQuery for a second client, and have Claude Code working on a third. This works fine until it doesn’t. And when it doesn’t, you get cryptic auth errors, expired tokens, or an agent that just stops working mid-task because it lost access to the right project’s credentials.

If you work at a consulting agency or freelance across multiple GCP projects, this will happen to you.

The design flaw at the center of all this

The gcloud CLI stores everything in a single directory: ~/.config/gcloud/. One set of credentials, one active configuration, one Application Default Credentials file, shared across every terminal window and every process on your machine. When you run gcloud config set project client-a, that change affects every other session reading from the same directory. AWS handles this differently, defaulting to named profiles with no shared active context. Google chose global mutable state.

This gets worse because gcloud actually has two independent authentication systems that people constantly confuse.

gcloud auth login authenticates the CLI tools (gcloud, bq, gsutil). It stores OAuth tokens in credentials.db inside that global directory.

gcloud auth application-default login creates the application_default_credentials.json file used by client libraries: Python, Node.js, Terraform, dbt, anything calling google.auth.default().

Running one does not affect the other. You can have gcloud pointing at Project A while your Python script authenticates against Project B. The result is usually a permission denied error that makes no sense until you realize the credentials don’t match the project you think you’re targeting.

The typical scenario: you start a dbt build for Client A, then switch your gcloud context to Client B in another terminal. Your ADC file gets overwritten, or the active config changes, and suddenly Client A’s build fails with auth errors halfway through. Or you start Claude Code in a new terminal and it picks up stale credentials from a different client session. With a human operator doing one thing at a time, this is manageable. With AI agents running gcloud commands in parallel, auth conflicts become a constant interruption.

AI agents make it worse

The multi-project problem has existed since gcloud was created. What changed is that AI coding agents introduced a new class of constraints that turn an occasional nuisance into a daily risk.

AI agents can’t complete interactive OAuth flows. gcloud auth login opens a browser window for consent, which a terminal agent can’t handle. They can’t re-authenticate when tokens expire mid-session. And when multiple agents run simultaneously, they step on each other’s config files, causing auth failures that are tedious to diagnose.

Claude Code

Claude Code defers entirely to environment credentials when connecting to Vertex AI. It uses standard ADC, which means it needs pre-authentication via gcloud auth login and gcloud auth application-default login before you start a session.

A documented issue (GitHub #726) affects organizations that enforce GCP session length restrictions. When credentials expire, Claude Code doesn’t pick up refreshed tokens. It continues using stale cached credentials and fails with invalid_grant errors. The only fix is restarting Claude Code entirely. Headless environment authentication is still an open request (GitHub #7100), with no device code flow support yet.

There’s a security dimension too. Research has shown that malicious repos can use MCP integrations or shell hooks to exfiltrate active gcloud credentials before you’ve even granted trust to the project. This is why CLOUDSDK_CONFIG isolation matters beyond the multi-client problem: it limits the blast radius of any credential theft to a single project’s temporary config directory.

OpenAI Codex

Codex takes the opposite approach. It runs in isolated containers with no internet access by default. Secrets can be provided via environment variables during setup, but they’re removed before the agent execution phase starts. This means persistent GCP credential access is architecturally impossible without enabling network access and explicitly wiring up credentials yourself.

Codex has added device code auth (codex login --device-auth) for its own authentication, but GCP credential management remains entirely manual. The isolation model is strong by design. It just means you do all the credential plumbing upfront.

Other agents

Cursor’s agent terminal runs in a sandboxed, non-interactive subshell that doesn’t inherit user environment variables or source shell config files. Interactive cloud CLI commands are completely non-functional. The community workaround is using MCP servers for GCP interactions instead of direct CLI commands.

For all agents, service account JSON key files remain the most reliable auth method for AI agent sessions, despite Google’s strong warnings against them. There is no practical keyless alternative for local development with agents today.

The fix: environment variables and direnv

The single most useful thing you can do is set CLOUDSDK_CONFIG to a different directory per project. This one variable completely isolates all gcloud state: credentials, configurations, ADC files, access tokens, and logs. Each directory gets its own credentials.db, its own application_default_credentials.json, its own everything. Two terminal sessions using different CLOUDSDK_CONFIG paths cannot interfere with each other.

But CLOUDSDK_CONFIG alone isn’t enough. You need four variables working together to get full isolation across both CLI tools and client libraries:

Why two separate project variables? Because CLOUDSDK_CORE_PROJECT only affects the gcloud CLI and its companions. GOOGLE_CLOUD_PROJECT only affects client libraries. They’re independent systems. If you only set one, you’ll have gcloud targeting the right project while dbt targets the wrong one, or vice versa.

Automating it with direnv

Setting four environment variables every time you switch clients gets old fast. direnv solves this by loading environment variables automatically when you enter a directory and unloading them when you leave.

Here’s what an actual .envrc looks like for one of my client projects:

export CLOUDSDK_CONFIG="$HOME/.config/gcloud-acme"
export CLOUDSDK_CORE_PROJECT=acme-analytics-prod
export GOOGLE_CLOUD_PROJECT=acme-analytics-prod
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.gcp-keys/acme-dbt-runner.json"

# dbt-specific
export DBT_TARGET=prod
export DBT_PROFILES_DIR="$HOME/.dbt"

When I cd into this directory, direnv loads everything. When I leave, it unloads. Each client directory has its own .envrc with its own isolated config path, project ID, and credentials. Claude Code inherits these variables from my shell, so when I start a session inside a client directory, the agent automatically has the right context.

For a new client, the setup takes about five minutes:

# Create isolated gcloud config directory
mkdir -p ~/.config/gcloud-newclient

# Activate the service account in that isolated directory
CLOUDSDK_CONFIG=~/.config/gcloud-newclient \
  gcloud auth activate-service-account \
  --key-file=~/.gcp-keys/newclient-sa.json

# Create the .envrc
cat > ~/projects/newclient-dbt/.envrc << 'EOF'
export CLOUDSDK_CONFIG="$HOME/.config/gcloud-newclient"
export CLOUDSDK_CORE_PROJECT=newclient-warehouse
export GOOGLE_CLOUD_PROJECT=newclient-warehouse
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.gcp-keys/newclient-sa.json"
EOF

# Trust the .envrc
direnv allow ~/projects/newclient-dbt

One caveat worth knowing: the gke-gcloud-auth-plugin does not respect CLOUDSDK_CONFIG (Kubernetes Issue #554). If you do GKE work, this breaks the isolation pattern. For analytics engineering workflows that don’t touch Kubernetes, it’s not a problem.

This pattern is surprisingly unknown outside CI/CD contexts. Google’s own documentation mentions CLOUDSDK_CONFIG but doesn’t emphasize it as the multi-project solution it is. For any consulting team, it should be standard infrastructure, not an optional tooling.

Service accounts: what Google recommends vs what actually works

Google’s official position is clear: don’t use service account key files. Use Workload Identity Federation instead. (I wrote about the broader IAM least privilege approach for data teams, which covers the theory. This section is about the practice.)

WIF works by letting an external identity provider (GitHub Actions OIDC, AWS IAM, Azure AD) vouch for your workload, so Google can issue short-lived credentials without a static key file. It’s genuinely better security. The problem is that a developer laptop isn’t GitHub Actions. There’s no external identity provider to federate with. WIF solves the CI/CD problem well. It doesn’t solve the “I’m sitting at my desk running dbt for three clients” problem at all.

The middle ground is service account impersonation. You authenticate as yourself via gcloud auth login, then impersonate a service account to get a short-lived access token:

gcloud auth print-access-token \
  --impersonate-service-account=dbt-runner@acme-prod.iam.gserviceaccount.com

This generates a 60-minute token. No key file on disk. The audit trail shows both your identity and the service account’s. Security teams like this approach, and for good reason. The friction is that tokens expire and need refreshing, which adds complexity to long-running agent sessions where you can’t easily re-authenticate.

A pragmatic setup for most client work:

• One service account per client project, created inside the client’s project (so a compromised key can’t reach other clients)
• Minimum roles: BigQuery User + BigQuery Data Editor for dbt work, Storage Object Viewer if GCS access is needed
• Naming convention: dbt-runner@client-project.iam.gserviceaccount.com
• 90-day key rotation, filesystem permissions (chmod 600) on key files
• Keys stored in ~/.gcp-keys/, which are in .gitignore and excluded from cloud sync

Is this what Google recommends? No. Does it work reliably with every tool in the stack, including AI agents that can’t do interactive auth? Yes. I’d rather use impersonation tokens, and I do for sensitive production work. But for the daily reality of running dbt builds across 10 clients, service account keys with proper hygiene are the pragmatic choice.

One thing to be honest about: AI agents log their inputs and outputs. A poorly sanitized conversation log could expose a key path or even key contents. This is why short-lived impersonation tokens are genuinely better if you can absorb the operational friction. Know the tradeoff you’re making.

Plugging it into dbt

If you’re using direnv with the four-variable pattern, dbt configuration is straightforward. The profiles.yml reads from environment variables set by direnv, so the right credentials load automatically based on which directory you’re in:

acme_analytics:
  target: "{{ env_var('DBT_TARGET', 'dev') }}"
  outputs:
    dev:
      type: bigquery
      method: service-account
      project: "{{ env_var('GOOGLE_CLOUD_PROJECT') }}"
      dataset: dbt_dev
      keyfile: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"
      threads: 4
      location: EU
    prod:
      type: bigquery
      method: service-account
      project: "{{ env_var('GOOGLE_CLOUD_PROJECT') }}"
      dataset: analytics
      keyfile: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"
      threads: 8
      location: EU

# Same pattern for every client
globex_warehouse:
  target: "{{ env_var('DBT_TARGET', 'dev') }}"
  outputs:
    dev:
      type: bigquery
      method: service-account
      project: "{{ env_var('GOOGLE_CLOUD_PROJECT') }}"
      dataset: dbt_dev
      keyfile: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"
      threads: 4
      location: US

Each client has its own profile block, but they all read from the same environment variables. direnv swaps the values when you switch directories. dbt run picks up the right project and credentials automatically.

The oauth method exists and works fine for single-project development, but it breaks in non-interactive environments (including AI agent sessions). The service-account method with env_var() interpolation is the consulting-friendly choice.

When you need heavier isolation

direnv covers most consulting scenarios. But there are cases where environment variables aren’t enough.

Docker and DevContainers

If you’re running multiple AI agents simultaneously (Claude Code for one client, Codex for another), or doing production work where you want absolute separation, containers give you a hard isolation boundary:

services:
  agent-acme:
    image: analytics-agent:latest
    volumes:
      - ~/.gcp-keys/acme-sa.json:/tmp/creds.json:ro
      - ~/projects/acme-analytics:/workspace
    environment:
      - GOOGLE_APPLICATION_CREDENTIALS=/tmp/creds.json
      - GOOGLE_CLOUD_PROJECT=acme-analytics-prod

  agent-globex:
    image: analytics-agent:latest
    volumes:
      - ~/.gcp-keys/globex-sa.json:/tmp/creds.json:ro
      - ~/projects/globex-warehouse:/workspace
    environment:
      - GOOGLE_APPLICATION_CREDENTIALS=/tmp/creds.json
      - GOOGLE_CLOUD_PROJECT=globex-warehouse-prod

Each container has its own filesystem, its own environment, and no way to reach the other’s credentials. For Claude Code specifically, claudebox provides per-project isolation with separate Docker images and auth state.

Most consulting teams don’t need this. direnv handles 90% of cases. Containers add operational overhead that’s only justified for simultaneous multi-agent work or high-stakes production environments.

Terraform for standardizing SA creation

When you’re managing service accounts across 10+ client projects, manual creation gets messy. The terraform-google-modules/service-accounts module standardizes the process:

module "dbt_service_account" {
  source        = "terraform-google-modules/service-accounts/google"
  version       = "~> 4.0"
  project_id    = var.client_project_id
  prefix        = "dbt"
  names         = ["runner"]
  project_roles = [
    "${var.client_project_id}=>roles/bigquery.user",
    "${var.client_project_id}=>roles/bigquery.dataEditor"
  ]
}

One module, applied per client, gives you consistent naming, consistent roles, and a single place to update permissions. Key rotation can be automated with Terraform’s time_rotating resource combined with Cloud Scheduler.

A quick note on Terraform workspaces: they’re wrong for multi-client setups. HashiCorp’s own documentation says CLI workspaces share the same backend and are unsuitable when deployments need separate access controls. For consulting agencies, Terragrunt with per-client directory structures is the better fit.

GKE Agent Sandbox

For teams building customer-facing AI features that execute untrusted code, GKE’s Agent Sandbox uses gVisor (a user-space kernel) to provide an extra isolation layer beyond standard containers. A compromised agent can’t reach the host kernel or other workloads. This is probably overkill for consulting analytics work, but it exists if you need it.

MCP servers: where this is heading

Google launched official MCP servers for BigQuery, Cloud SQL, Spanner, and other services in late 2025 (I covered the BigQuery MCP server setup separately). The @google-cloud/gcloud-mcp package wraps the gcloud CLI for structured interaction, which directly addresses the “agents can’t do interactive auth” problem.

The shift is architectural. Instead of an agent running gcloud commands in your shell (inheriting your global state, fighting with your other sessions), the agent talks to an MCP server that manages its own credentials. If you’re new to the protocol, my MCP fundamentals overview covers the basics. The agent never touches ~/.config/gcloud/ directly. Authentication moves from the terminal layer to a protocol layer.

For multi-project work, this is promising. Community implementations like LokiMCPUniverse/gcp-mcp-server already support explicit per-project configuration where each project gets its own credential file, and the agent specifies which project context to use per request.

IDE-based agents are moving in this direction too. Cursor and Windsurf support project-level MCP configurations, so the MCP server restarts with the right credentials when you switch projects. The failure mode to watch for is context drift: your IDE open to Client A’s project while the background MCP process is still authenticated to Client B. Project-level mcp_config.json files prevent this.

My honest assessment: MCP is the right direction. Authentication managed at the protocol layer instead of the shell layer eliminates an entire category of problems. But the ecosystem is young. The servers are still maturing, documentation is thin, and for production consulting work I’m not ready to depend on them as the primary auth mechanism. Watch this space, but don’t rip out your direnv setup yet.

The layered approach

Today (30 minutes): Install direnv. Create a .envrc per client with the four variables (CLOUDSDK_CONFIG, CLOUDSDK_CORE_PROJECT, GOOGLE_CLOUD_PROJECT, GOOGLE_APPLICATION_CREDENTIALS). One service account per client project with minimum required roles.

This month: Set up a Terraform module for standardized service account creation. Automate key rotation. Get the whole team on the same direnv pattern.

This quarter: Evaluate Google’s MCP servers for your agent workflows. If they’re stable enough, they’ll make the entire shell-level isolation problem irrelevant.

Google designed gcloud for single-project developers. Consulting agencies are the opposite of that. Every solution in this article is a workaround for a design choice Google made years ago. The good news is that CLOUDSDK_CONFIG exists and works. The better news is that MCP might eventually make the whole problem disappear. Until then, four environment variables and a .envrc file are the difference between a normal work day and spending half your morning debugging auth errors across three client projects.