Debugging Dags
Run structured Airflow DAG failure diagnosis with af CLI when a pipeline needs root cause analysis—not just a quick log peek.
Overview
Debugging DAGs is an agent skill most often used in Operate (also Ship testing, Build integrations) that guides systematic Airflow DAG failure diagnosis and prevention recommendations via the af CLI.
Install
npx skills add https://github.com/astronomer/agents --skill debugging-dagsWhat is this skill?
- Structured multi-step flow: identify failure, pull task logs, categorize failure type, recommend prevention
- Uses Astronomer af CLI (runs diagnose, dags stats, health, dags errors, tasks logs)
- Separates simple log questions from deep RCA requests the airflow entrypoint handles lightly
- Documents astro otto and uv tool install paths for getting af on PATH
- Focuses on scrolling past Airflow boilerplate to the underlying exception
Adoption & trust: 806 installs on skills.sh; 384 GitHub stars; 3/3 security scanners passed (skills.sh audits).
What problem does it solve?
Your Airflow DAG failed and chat-only guesses hide the real exception buried under scheduler boilerplate.
Who is it for?
Builders on Astronomer or Airflow who need structured RCA after repeated or unclear pipeline failures.
Skip if: Quick 'show me the last log line' requests better handled by the airflow entrypoint skill alone.
When should I use this skill?
User requests complex DAG debugging: diagnose and fix the pipeline, full root cause analysis, why failing and how to prevent—not simple why did dag fail or show logs.
What do I get? / Deliverables
You get a categorized root cause from task logs plus actionable remediation and prevention steps tied to specific dag_id, run_id, and task_id.
- Failure classification
- Log-backed root cause summary
- Prevention and remediation recommendations
Recommended Skills
Journey fit
Spans multiple journey phases - primary shelf plus alternate fits below.
Failed scheduled pipelines are production operations problems; canonical shelf is Operate because remediation targets live DAG runs. Subphase errors matches task failures, exceptions in logs, and import errors surfaced via af health and dags errors.
Where it fits
Diagnose a failed nightly ETL DAG run after alerts fire and extract the real exception from task logs.
Investigate flaky DAG behavior in staging before promoting a scheduler change to production.
Trace import errors and DAG parse failures blocking new Airflow operators from loading.
Unblock revenue or lifecycle metrics pipelines when upstream extracts fail mid-run.
How it compares
Structured investigation workflow for Airflow—not a generic Python debugger or local unit-test runner.
Common Questions / FAQ
Who is debugging-dags for?
Solo data engineers and indie teams operating Airflow DAGs who want an agent to follow a consistent diagnose-and-fix playbook instead of random log grepping.
When should I use debugging-dags?
In Operate when pipelines fail in production, during Ship when staging DAGs break before release, or when you ask for full root cause analysis, diagnose and fix the pipeline, or prevention guidance—not for trivial 'why did dag fail' one-offs.
Is debugging-dags safe to install?
It instructs agents to run af CLI against your Airflow metadata and logs; review the Security Audits panel on this page and scope credentials and network access to trusted environments only.
SKILL.md
READMESKILL.md - Debugging Dags
# DAG Diagnosis You are a data engineer debugging a failed Airflow DAG. Follow this systematic approach to identify the root cause and provide actionable remediation. ## Running the CLI These commands assume `af` is on PATH. Run via `astro otto` to get it automatically, or install standalone with `uv tool install astro-airflow-mcp`. --- ## Step 1: Identify the Failure If a specific DAG was mentioned: - Run `af runs diagnose <dag_id> <dag_run_id>` (if run_id is provided) - If no run_id specified, run `af dags stats` to find recent failures If no DAG was specified: - Run `af health` to find recent failures across all DAGs - Check for import errors with `af dags errors` - Show DAGs with recent failures - Ask which DAG to investigate further ## Step 2: Get the Error Details Once you have identified a failed task: 1. **Get task logs** using `af tasks logs <dag_id> <dag_run_id> <task_id>` 2. **Look for the actual exception** - scroll past the Airflow boilerplate to find the real error 3. **Categorize the failure type**: - **Data issue**: Missing data, schema change, null values, constraint violation - **Code issue**: Bug, syntax error, import failure, type error - **Infrastructure issue**: Connection timeout, resource exhaustion, permission denied - **Dependency issue**: Upstream failure, external API down, rate limiting ## Step 3: Check Context Gather additional context to understand WHY this happened: 1. **Recent changes**: Was there a code deploy? Check git history if available 2. **Data volume**: Did data volume spike? Run a quick count on source tables 3. **Upstream health**: Did upstream tasks succeed but produce unexpected data? 4. **Historical pattern**: Is this a recurring failure? Check if same task failed before 5. **Timing**: Did this fail at an unusual time? (resource contention, maintenance windows) Use `af runs get <dag_id> <dag_run_id>` to compare the failed run against recent successful runs. ### On Astro If you're running on Astro, these additional tools can help with diagnosis: - **Deployment activity log**: Check the Astro UI for recent deploys — a failed deploy or recent code change is often the cause of sudden failures - **Astro alerts**: Configure alerts in the Astro UI for proactive failure monitoring (DAG failure, task duration, SLA miss) - **Observability**: Use the Astro [observability dashboard](https://www.astronomer.io/docs/astro/airflow-alerts) to track DAG health trends and spot recurring issues ### On OSS Airflow - **Airflow UI**: Use the DAGs page, Graph view, and task logs to inspect recent runs and failures ## Step 4: Provide Actionable Output Structure your diagnosis as: ### Root Cause What actually broke? Be specific - not "the task failed" but "the task failed because column X was null in 15% of rows when the code expected 0%". ### Impact Assessment - What data is affected? Which tables didn't get updated? - What downstream processes are blocked? - Is this blocking production dashboards or reports? ### Immediate Fix Specific steps to resolve RIGHT NOW: 1. If it's a data issue: SQL to fix or skip bad records 2. If it's a code issue: The exact code change needed 3. If it's infra: Who to contact or what to restart ### Prevention How to prevent this from happening again: - Add data quality checks? - Add better error handling? - Add alerting for edge cases? - Update documentation? ### Quick Commands Provide ready-to-use commands: - To clear and rerun the entire DAG run: `af runs clear <dag_id> <run_id>` - To