Skip to content

Course Notes

Use this workspace to take notes, store code snippets, and build your own interactive cheatsheet!

import dask.dataframe as dd

# Set the number of partitions
athlete_events_dask = dd.from_pandas(athlete_events, npartitions=4)

# Calculate the mean Age per Year
print(athlete_events_dask.groupby('Year').Age.mean().compute())

The above code is for to allocate the parallel processing

the above code is used for parallel computing using dask.dataframe key to remember is to add compute() at the end

The methods you're going to use in this exercise are:

.printSchema(): helps print the schema of a Spark DataFrame. .groupBy(): grouping statement for an aggregation. .mean(): take the mean over each group. .show(): show the results.

# Add your code snippets here
# Create the DAG object
dag = DAG(dag_id="car_factory_simulation",
          default_args={"owner": "airflow","start_date": airflow.utils.dates.days_ago(2)},
          schedule_interval="0 * * * *")

# Task definitions
assemble_frame = BashOperator(task_id="assemble_frame", bash_command='echo "Assembling frame"', dag=dag)
place_tires = BashOperator(task_id="place_tires", bash_command='echo "Placing tires"', dag=dag)
assemble_body = BashOperator(task_id="assemble_body", bash_command='echo "Assembling body"', dag=dag)
apply_paint = BashOperator(task_id="apply_paint", bash_command='echo "Applying paint"', dag=dag)

# Complete the downstream flow
assemble_frame.set_downstream(place_tires)
assemble_frame.set_downstream(assemble_body)
assemble_body.set_downstream(apply_paint)

the code is for scheduling the DAGs to carryout the workflw at regular intervals of time