Reuse tasks in airflow Reuse tasks in airflow hadoop hadoop

Reuse tasks in airflow


  • Just like object is an instance of a class, an Airflow task is an instance of an Operator (strictly speaking, BaseOperator)
  • So write a "re-usable" (aka generic) operator and use it 100s of times across your pipeline(s) simply by passing different params (particularly task_id)