DAG starter template
Airflow DAGs are written in Python and are technically just a Python module (with .py extension). DAGs are interpreted by Airflow via the DagBag facility and can then be scheduled to execute.
DAGs files are placed under the AIRFLOW__CORE__DAGS_FOLDER. The directory location can be identified as follows:
The default DAG template can help you get started creating your new DAG. The template DAG at src/flowz/dags/template.py features a set of start and end "book-end" tasks that can be used to delimit your job. You then add your own business related tasks in between.
The start and end tasks are instantiated via Airflow's EmptyOperators and act as safe landing zones for your job.
Note
More information around Airflow DAG creation and concepts is available at the Airflow tutorial.
The actual source code of the src/flowz/dags/template.py is as follows:
"""The simplest DAG template.
"""
from pathlib import PurePath
import airflow
import flowz.task
from flowz.primer import Primer
primer = Primer(dag_name=PurePath(__file__).stem.replace("_", "-"), department="ADMIN")
primer.dag_properties.update(
{"description": "Simple book-end DAG template to get you started"}
)
dag = airflow.DAG(
primer.dag_id, default_args=primer.default_args, **(primer.dag_properties)
)
task_start = flowz.task.start(dag, default_args=primer.default_args)
#
# Add your content here.
#
task_end = flowz.task.end(dag, default_args=primer.default_args)
task_start >> task_end # pylint: disable=pointless-statement
Copy src/flowz/dags/template.py into a new Python file. The filename should be descriptive enough to define your new workflow. In the following example, the Python module target will be called sample:
A more detailed description about your new DAG can be provided by editing src/flowz/dags/sample.py and replacing the description variable value to suit. description renders in the Airflow UI and helps visitors understand the intent behind your workflow.
A quick validation of your new DAG can be performed with:
dag_id | filepath | owner | paused
======================+==============+=========+=======
ADMIN_BOOTSTRAP_LOCAL | bootstrap.py | airflow | False
ADMIN_SAMPLE_LOCAL | sample.py | airflow | True
The new sample DAG is rendered under the Apache Airflow dashboard's graph view as follows:
