ServicesAboutNotesContact Get in touch →
EN FR
Note

dbt Private Packages via Git

How to distribute internal dbt packages as Git dependencies — version pinning, authentication options, and trade-offs compared to Hub packages.

Planted
dbtdata engineering

Internal packages — utility macros, shared tests, company-specific source definitions — can be distributed as Git dependencies without registration on the dbt Hub. A common progression is to share via Git within a team’s projects first, then publish to the Hub once the package is stable enough for broader use.

Basic Installation

Users reference a Git repository directly in packages.yml:

packages.yml
packages:
  - git: "https://github.com/my-org/dbt-internal-utils.git"
    revision: v0.3.0

Running dbt deps clones the repository at the specified revision. The package behaves identically to a Hub package once installed — same model resolution, same macro availability, same source definitions.

Version Pinning

Pin to a tag or commit SHA, not a branch name. This is the single most important rule for Git packages.

# Good — deterministic
packages:
  - git: "https://github.com/my-org/dbt-internal-utils.git"
    revision: v0.3.0


# Also good — even more deterministic
packages:
  - git: "https://github.com/my-org/dbt-internal-utils.git"
    revision: abc1234def5678


# Dangerous — changes with every commit
packages:
  - git: "https://github.com/my-org/dbt-internal-utils.git"
    revision: main

Pointing at main means your package resolution changes with every commit to the upstream repo. A colleague pushing a breaking change to the shared utils package at 3pm silently breaks every project that runs dbt deps after that point. Reproducibility disappears.

Tags (using semantic versioning) are the best practice. They’re human-readable, they communicate intent (is this a patch or a breaking change?), and they match how Hub packages work. Commit SHAs are useful when you need to pin to an exact state, but they’re harder to reason about.

Authentication for Private Repositories

For private repositories, dbt supports three authentication approaches.

packages:
  - private: my-org/dbt-internal-utils
    provider: github
    revision: v0.3.0

The private keyword with a provider uses the Git integration already configured in your environment — GitHub, GitLab, or Azure DevOps. This is the cleanest approach because:

  • No tokens embedded in packages.yml
  • Works with whatever auth your team already uses (SSH keys, credential helpers, GitHub CLI)
  • Supports GitHub, GitLab, and Azure DevOps natively

The downside is that it requires the running environment (developer laptop, CI runner, dbt Cloud) to have Git credentials configured. In CI, this usually means a GitHub App or deploy key.

Token via Environment Variable

packages:
  - git: "https://{{ env_var('GIT_TOKEN') }}@github.com/my-org/dbt-internal-utils.git"
    revision: v0.3.0

The env_var() function injects a token at resolution time. This works everywhere but has a drawback: the token appears in the resolved packages.yml, which means it could show up in logs if dbt prints debug output. Use a fine-grained personal access token or a GitHub App token with minimal permissions (read-only on the specific repository).

SSH

packages:
  - git: "git@github.com:my-org/dbt-internal-utils.git"
    revision: v0.3.0

SSH authentication uses the SSH key configured in the running environment. It’s common for developer laptops but can be tricky in CI environments where you need to manage deploy keys.

Trade-offs Compared to Hub Packages

Git packages lack two features that Hub packages provide:

  1. No transitive dependency resolution. If your Git package depends on dbt-utils v1.3.0 and the user’s project depends on dbt-utils v1.2.0, you get a conflict error. Hub packages resolve this automatically by finding the highest version that satisfies all constraints. With Git packages, you manually align versions.

  2. No version range syntax. Hub packages support [">=1.0.0", "<2.0.0"] which lets dbt pick the best version. Git packages pin to a single revision. When you release v0.4.0 of your internal package, every consuming project must update their packages.yml manually.

These trade-offs are manageable for internal packages where you control both sides. They become painful at scale — if 20 projects depend on your internal utils package and you release a new version, 20 PRs need to update the revision.

Monorepo vs Dedicated Repo

For internal packages, you have a choice: put the package in its own repository or include it in a monorepo alongside other dbt projects.

Dedicated repo is simpler for dbt deps:

packages:
  - git: "https://github.com/my-org/dbt-internal-utils.git"
    revision: v0.3.0

Monorepo with subdirectory requires the subdirectory key:

packages:
  - git: "https://github.com/my-org/analytics-monorepo.git"
    revision: v0.3.0
    subdirectory: "packages/dbt-internal-utils"

Dedicated repos are easier to version and tag independently. Monorepos work when multiple packages need to be versioned together, but independent tagging becomes awkward.

A Practical Internal Utils Package

Most teams start with a “utils” package that holds shared macros and tests. A typical structure:

dbt-internal-utils/
├── dbt_project.yml
├── macros/
│   ├── generate_schema_name.sql
│   ├── cents_to_dollars.sql
│   ├── limit_data_in_dev.sql
│   └── add_audit_columns.sql
├── tests/generic/
│   ├── is_positive.sql
│   ├── date_not_in_future.sql
│   └── not_empty_string.sql
└── README.md

This package has no models — just macros and custom generic tests. There are no sources to configure and no models to namespace. Moving from Git distribution to a Hub package requires adding var() configuration for anything currently hardcoded to organization-specific values.