ServicesAboutNotesContact Get in touch →
EN FR
Note

dbt-expectations Hub

Hub note for dbt-expectations — setup, test reference, conditional filtering, severity tuning, BigQuery implementation patterns, and the unit test vs data test distinction.

Planted
dbtdata qualitytesting

dbt-expectations is a community package (maintained by Metaplane) that brings 50+ pre-built data quality tests to dbt. It fills the gap between dbt’s four built-in generic tests and full anomaly detection tools like Elementary. Where native dbt handles structural validation (unique, not_null, relationships, accepted_values), dbt-expectations handles domain validation: pattern matching, value ranges, statistical checks, freshness on any model, multi-column assertions, and time series completeness.

Core Notes

  • dbt-expectations Setup and Configuration — Installation via packages.yml, the required timezone variable, transitive dependencies (dbt-date, dbt-utils), platform support (BigQuery, Snowflake, Postgres, Redshift, DuckDB, Trino), and verifying the installation.

  • dbt-expectations Test Reference — Categorized reference of the highest-value tests with BigQuery-ready YAML examples. Table-level tests (freshness, row counts), pattern validation (regex, LIKE), value range checks (row-level and statistical), multi-column validation (composite keys, cross-column comparisons), and completeness tests (time series gap detection).

  • dbt-expectations row_condition Pattern — The row_condition parameter that works across nearly every test in the package. Conditional filtering for nullable fields, status-based validation, segment-specific ranges, and partition-aware performance optimization. One parameter that eliminates the need for dozens of custom singular tests.

  • dbt Test Severity and Performance Tuning — Configuring warn vs error severity by test category, BigQuery-specific cost optimization (partition filtering, prod-only tests, tag-based scheduling), test placement by DAG layer, and a practical “first three tests” starting point.

  • dbt-expectations BigQuery Implementation Patterns — Real-world implementation with complete GA4 events and advertising performance YAML. Test placement by DAG layer (base, intermediate, mart), partition-aware patterns for BigQuery cost control, and a prioritized expansion checklist for growing test coverage.

  • Unit Tests vs Data Tests in dbt — The two-checkpoint model: unit tests gate deployments by verifying transformation logic with mocked data, data tests gate production by verifying actual data health. Why you need both, what each catches that the other misses, and practical guidance on where to start.

  • dbt Testing Taxonomy — The full taxonomy of dbt test types. dbt-expectations sits alongside generic tests, singular tests, unit tests, model contracts, and Elementary in a layered testing strategy.

  • Semantic Validation in dbt — The broader category of business rule validation that dbt-expectations enables: regex patterns, range checks, cross-column logic, and AI-powered validation.

  • Data Quality Validation Layers — The three-layer model (contracts, tests, anomaly detection) where dbt-expectations occupies the reactive validation layer alongside dbt’s native tests.

Connections

dbt-expectations occupies the “known domain violations” layer in a data quality strategy. It catches problems you can anticipate and write rules for. For problems you can’t anticipate, Elementary provides anomaly detection. For structural guarantees at the schema level, model contracts prevent breaking changes before data is materialized. For migration validation, dbt-audit-helper compares two relations to prove equivalence.