ServicesAboutNotesContact Get in touch →
EN FR
Note

ML Anomaly Detection vs Statistical Methods

When ML-powered anomaly detection earns its cost over simpler Z-score approaches — and why the answer depends on data complexity, not marketing materials.

Planted
dbtelementarydata qualitydata engineering

ML-powered anomaly detection earns its cost over simpler statistical methods only in specific conditions. The marketing claims for ML detection (“learns your data patterns automatically,” “no manual thresholds”) are accurate but not universally relevant — for many dbt projects, Z-score based detection is sufficient.

How Statistical Detection Works

Elementary OSS uses Z-score based detection. The approach is conceptually simple: compute the mean and standard deviation of a metric over a training period, then flag when the current value falls more than N standard deviations from the mean.

tests:
  - elementary.volume_anomalies:
      time_bucket:
        period: day
        count: 1
      anomaly_sensitivity: 3  # flag at 3 standard deviations
      training_period:
        period: day
        count: 14  # learn from 14 days of history

A sensitivity of 3 means alerting when a metric falls more than 3 standard deviations from its historical mean. For normally distributed data, this corresponds to roughly a 0.3% chance of a false positive on any given check. In practice, data distributions are rarely perfectly normal, so the false positive rate varies.

The strengths of this approach:

  • Transparent. You can inspect the math. When an alert fires, you can look at the historical mean, the standard deviation, and the current value, and understand exactly why it triggered.
  • Configurable. The anomaly_sensitivity and training_period parameters give you direct control over the tradeoff between sensitivity and noise.
  • No training lag. With 14 days of history, Z-score detection starts working. ML models often need months of data to learn meaningful patterns.
  • Low computational cost. Mean and standard deviation are cheap to compute, even on large tables.

The weaknesses are equally clear:

  • No seasonal awareness. If your e-commerce data has 3x volume on weekends, Z-score detection treats Monday’s drop as an anomaly every single week until you manually adjust the training window or add filters.
  • No trend adaptation. A business growing 10% month-over-month will eventually trigger volume anomaly alerts as the current count exceeds the historical mean plus three standard deviations. The detection doesn’t understand “growth.”
  • Symmetric sensitivity. A 3-sigma threshold treats a 50% drop and a 50% spike identically. For many metrics, a volume drop is an emergency while a volume spike is expected during a marketing campaign.
  • Single-metric isolation. Z-scores evaluate each metric independently. They can’t correlate a revenue drop with a corresponding volume drop to identify a common upstream cause.

How ML Detection Works

Monte Carlo, Anomalo, and Bigeye use more sophisticated ML that learns from historical patterns across multiple dimensions simultaneously.

The approaches vary by vendor, but the general architecture involves:

  • Time series models that learn daily, weekly, and seasonal patterns. Monday is expected to look different from Saturday. December is expected to look different from June.
  • Multi-metric correlation that understands relationships between metrics. If volume and revenue both drop, that’s one incident, not two.
  • Adaptive baselines that adjust to gradual changes without treating growth or seasonal shifts as anomalies.
  • Confidence scoring that differentiates between “this is definitely wrong” and “this is unusual but might be expected.”

Vimeo Engineering reported reducing incidents to 10% of their previous volume after implementing Monte Carlo. That’s a compelling number, but it requires context: Vimeo is a high-scale environment with complex seasonal patterns, multiple data sources, and the kind of data complexity where ML has enough signal to learn meaningful patterns.

Where ML Earns Its Cost

ML-powered detection earns its premium in specific conditions, not universally.

Strong seasonal patterns. If your data has predictable weekly, monthly, or annual cycles, ML learns these patterns and adjusts baselines accordingly. A retail company with Black Friday spikes, summer lulls, and weekend peaks gets significant value from a model that understands these rhythms. With Z-score detection, you’d need to manually configure different thresholds for different time periods — or accept a flood of false positives during expected variation.

High table count. When you have 500+ tables to monitor, manual threshold configuration becomes impossible. ML that automatically learns appropriate baselines for each table provides coverage that no team can replicate manually.

Complex cross-metric relationships. When a single root cause manifests as anomalies across multiple metrics and tables simultaneously, ML that can correlate these signals and identify the common cause saves hours of manual investigation. A schema change in a source table that causes null rate increases in ten downstream models is one incident, but Z-score detection fires ten independent alerts.

Gradual drift detection. If your daily user count shifts from 10,000 to 8,000 over two weeks, a row_count_between test set to 5,000-15,000 won’t flag it. ML models that track trend direction and velocity can detect this kind of slow degradation.

Where Statistical Methods Are Sufficient

For many dbt projects with relatively stable data patterns, Elementary’s Z-score approach is genuinely sufficient. The conditions where statistical methods work well:

Stable, predictable data. If your sources update on a consistent schedule, your volumes are relatively steady (or grow linearly), and you don’t have strong seasonal patterns, Z-score detection catches the anomalies that matter without the overhead of ML.

Low table count. Under 100 tables, you can manually tune sensitivity settings and training periods. The time investment is manageable, and you maintain full control over alerting behavior.

Clear failure modes. If your most common data quality issues are “source stopped sending data” (volume drops to zero) or “schema changed” (column added or removed), these are binary failures that Z-score detection catches trivially. You don’t need ML to detect that a table that normally has 10,000 rows suddenly has zero.

Budget constraints. At $0 licensing cost, Elementary OSS provides anomaly detection that catches the majority of real issues. The TCO calculation often favors OSS for teams where the alternative is no anomaly detection at all, not a choice between Z-scores and ML.

Assessment

ML-powered detection is better at handling complex patterns, but “better” and “necessary” are different questions. For a team running a single warehouse with 50–200 models and relatively stable data patterns, investing $5K–15K/month in ML-powered detection addresses a problem that Z-score detection handles adequately. Elementary’s approach catches the high-severity anomalies (data stopped arriving, volume dropped 80%, null rate spiked) that represent the majority of real incidents.

ML detection is justified in high-volume, high-complexity environments where manual threshold management becomes impractical and where the number of potential failure patterns exceeds what any team can anticipate. Below that threshold, the cost is better applied to expanding test coverage and establishing the minimum viable observability stack.

Observability tooling that goes untuned degrades over time; simpler approaches that receive regular attention tend to perform better in practice.