Self-healing pipeline capabilities range from basic retry logic to fully autonomous remediation. The term is used loosely across vendors. A five-level maturity spectrum distinguishes these capabilities by what each level actually does and what constraints apply.
Level 1: Auto-retry
The pipeline fails on a transient error, waits, tries again. Airflow’s retries parameter, Cloud Composer’s built-in retry policies, any orchestrator’s equivalent. Simple, effective for network blips and API rate limits.
A well-configured retry strategy with exponential backoff and sensible timeouts eliminates most transient incidents. This level is the foundation before any higher-level automation is considered.
Level 2: Catch-up and backfill
The pipeline detects missed or failed runs and processes data from the last successful checkpoint. Instead of requiring someone to manually trigger a backfill, the system fills gaps automatically.
For dbt incremental models, the is_incremental() logic already provides a natural catch-up mechanism. If a run fails and the next one succeeds, it picks up everything since the last materialized timestamp. The lookback window pattern extends this further: reprocessing a rolling window of recent data on every run handles both late-arriving records and recovery from partial failures.
Catch-up sounds simple, but getting it right requires thinking about idempotency. A pipeline that can safely re-run over the same time period without creating duplicates or double-counting is a pipeline that recovers gracefully. One that can’t is a pipeline where every failure demands manual investigation.
Level 3: Schema drift adaptation
The pipeline detects that upstream data changed shape (new columns, type changes) and adapts without breaking. This is where dbt’s [[on_schema_change in dbt Incremental Models|on_schema_change config]] fits. Setting it to append_new_columns means upstream schema additions don’t break your pipeline. sync_all_columns goes further, adding new columns and removing dropped ones.
The choice between these options depends on trust. For sources you control, sync_all_columns works well. For third-party data, fail might be the safer option, because a schema change you didn’t expect probably deserves investigation rather than silent adaptation.
Schema drift adaptation is level 3, not level 1 or 2, because it requires the pipeline to understand something about its own structure. It’s not just retrying the same operation. It’s modifying its own behavior based on what changed upstream.
Level 4: AI-powered diagnosis and remediation
On failure, the pipeline captures context (error logs, sample data, stack traces), sends it to an LLM, receives a structured fix, and retries with corrections. The Try-Heal-Retry pattern is the reference architecture for this level.
This is where the practical ceiling currently sits for production systems in 2026. Real implementations exist. Michael Stewart’s Datadog + Claude Code integration, Monte Carlo’s monitoring and troubleshooting agents, custom Airflow on_failure_callback setups that call Claude for structured diagnosis. They work for specific, well-defined failure classes like file ingestion errors where the fixes are predictable (encoding, delimiter, date format).
The key constraint is that level 4 requires risk tiering. Not every failure should be auto-remediated by an LLM. A file with a changed delimiter getting auto-corrected is fine. A financial calculation that starts returning different results because the LLM “fixed” a schema change is a different category of problem.
Level 5: Fully agentic
The pipeline autonomously decides what data to collect, how to transform it, and how to handle any situation without human input. This is mostly theoretical. It shows up in conference talks and vendor roadmaps, but production examples are scarce. The maturity assessment for AI agents in data quality work puts autonomous remediation firmly in the “don’t depend on this yet” category.
The gap between level 4 and level 5 is not incremental. Level 4 has a human-defined scope: these failure types, these remediation options, these safety boundaries. Level 5 has no predefined scope, which means no predefined safety boundaries either. That’s a fundamentally different engineering problem.
Where value concentrates
Most production value sits at levels 1 through 3: retry configuration in the orchestrator, dbt’s on_schema_change on incremental models, and Elementary for statistical anomaly detection. These cover the majority of incidents without LLM API calls.
Level 4 is becoming practical for specific, well-scoped failure classes where fixes are predictable and risk is contained.
ServiceNow’s 2025 Enterprise AI Maturity Index found fewer than 1% of organizations scored above 50 out of 100. Gartner reported 30% of AI initiatives abandoned, primarily due to data quality issues.