Skip to content

Introduction to Airflow in Python

Airflow Operators

  • The operators mostly of the times are a simple task in the DAG
  • There are a lot of possible operators. In the exercise we used a BashOperator, which is used to run commands or scripts in shell (virtualenv) on that task. This operator needs:
  • To be assigned to a variable
  • To assign a task name that will carry out it
  • To specify the command or script to be run
  • To specify the DAG to which this task belongs.
# As an example of how a BashOperator is defined:
from airflow.operators.bash_operator import BashOperator

# Definining the BashOperator:
cleanup=BashOperator(
    task_id='cleanup_task',
    bash_command='cleanup.sh',
    dag=analytics_dag
)
Hidden output

More than one Operator

You can define more than one Operator, yu can define as much operators as needed. Each operator will be a different task.

Questions:

  • An Operator is a virtualenv?
  • Which are the differences between Operators and Hooks?
  • If an Operator is a Virtualenv, you can combine for example a BashOperator with a Python Operator in the same task?
Run cancelled
# Example of two BashOperators in the DAG script:
from airflow.operators.bash_operator import BashOperator

# Defining the first Operator/Task:
consolidate= BashOperator(
    task_id='consolidate_task',
    bash_command='consolidate_data.sh'
    dag=analytics_dag
)

# Defining the second Operator/Task:
push_dat= BashOperator(
    task_id='pushdata_task',
    bash_command='push_data.sh',
    dag=analytics_dag
)