AgentAlley

Skills · Data & AI

Apache Airflow DAG Patterns

Unverified●30/40

Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor◐PartialPlain prose you can paste in — but no Cursor rules file
Codex◐PartialPlain prose you can paste in — but no AGENTS.md
Gemini CLI◐PartialPlain prose you can paste in
Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file
npx agentalley add airflow-dag-patterns

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

The whole source

No sign-in, no blur, nothing truncated
airflow-dag-patterns/SKILL.md116 lines3.1 KBRawView on GitHub
Frontmatter — 2 properties
nameairflow-dag-patterns
descriptionBuild production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
1---
2name: airflow-dag-patterns
3description: Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# Apache Airflow DAG Patterns
7 
8Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.
9 
10## When to Use This Skill
11 
12- Creating data pipeline orchestration with Airflow
13- Designing DAG structures and dependencies
14- Implementing custom operators and sensors
15- Testing Airflow DAGs locally
16- Setting up Airflow in production
17- Debugging failed DAG runs
18 
19## Core Concepts
20 
21### 1. DAG Design Principles
22 
23| Principle | Description |
24| --------------- | ----------------------------------- |
25| **Idempotent** | Running twice produces same result |
26| **Atomic** | Tasks succeed or fail completely |
27| **Incremental** | Process only new/changed data |
28| **Observable** | Logs, metrics, alerts at every step |
29 
30### 2. Task Dependencies
31 
32```python
33# Linear
34task1 >> task2 >> task3
35 
36# Fan-out
37task1 >> [task2, task3, task4]
38 
39# Fan-in
40[task1, task2, task3] >> task4
41 
42# Complex
43task1 >> task2 >> task4
44task1 >> task3 >> task4
45```
46 
47## Quick Start
48 
49```python
50# dags/example_dag.py
51from datetime import datetime, timedelta
52from airflow import DAG
53from airflow.operators.python import PythonOperator
54from airflow.operators.empty import EmptyOperator
55 
56default_args = {
57 'owner': 'data-team',
58 'depends_on_past': False,
59 'email_on_failure': True,
60 'email_on_retry': False,
61 'retries': 3,
62 'retry_delay': timedelta(minutes=5),
63 'retry_exponential_backoff': True,
64 'max_retry_delay': timedelta(hours=1),
65}
66 
67with DAG(
68 dag_id='example_etl',
69 default_args=default_args,
70 description='Example ETL pipeline',
71 schedule='0 6 * * *', # Daily at 6 AM
72 start_date=datetime(2024, 1, 1),
73 catchup=False,
74 tags=['etl', 'example'],
75 max_active_runs=1,
76) as dag:
77 
78 start = EmptyOperator(task_id='start')
79 
80 def extract_data(**context):
81 execution_date = context['ds']
82 # Extract logic here
83 return {'records': 1000}
84 
85 extract = PythonOperator(
86 task_id='extract',
87 python_callable=extract_data,
88 )
89 
90 end = EmptyOperator(task_id='end')
91 
92 start >> extract >> end
93```
94 
95## Detailed patterns and worked examples
96 
97Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
98 
99## Best Practices
100 
101### Do's
102 
103- **Use TaskFlow API** - Cleaner code, automatic XCom
104- **Set timeouts** - Prevent zombie tasks
105- **Use `mode='reschedule'`** - For sensors, free up workers
106- **Test DAGs** - Unit tests and integration tests
107- **Idempotent tasks** - Safe to retry
108 
109### Don'ts
110 
111- **Don't use `depends_on_past=True`** - Creates bottlenecks
112- **Don't hardcode dates** - Use `{{ ds }}` macros
113- **Don't use global state** - Tasks should be stateless
114- **Don't skip catchup blindly** - Understand implications
115- **Don't put heavy logic in DAG file** - Import from modules
116 

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40Ebay Seller ToolsData & AI · MIT·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40