Step 4: Orchestrate Your Pipelines

If you have a working ingestion + transformation setup, you are probably running dbt on a schedule — either dbt Cloud’s built-in scheduler, a cron job, or manually. That works until it doesn’t. Cron does not retry. It does not alert you when upstream data is late. It does not let you re-run a single failed model without re-running everything.

Orchestration is the layer that makes your pipelines reliable. You add it when cron becomes a liability.

What orchestration gives you

• Retries — when an API call times out, run it again automatically
• Dependencies — run dbt only after Fivetran finishes syncing
• Observability — a UI that shows you what ran, what produced what, what failed, and why
• Alerting — get paged when something breaks at 3am instead of finding out in the morning standup
• Partitioning — process data in date ranges rather than all-or-nothing full refreshes

Dagster vs Airflow

Dagster is the modern choice. Its core concept is the data asset — instead of thinking about tasks that run, you think about the data that gets produced. This maps naturally to how analysts and data engineers think.

Dagster has a first-class dbt integration: you can treat every dbt model as a Dagster asset, see its lineage in the Dagster UI, and trigger runs selectively when upstream data changes. Dagster+ is the managed cloud offering — no infrastructure to run.

Airflow is the incumbent. It has been running production data pipelines for a decade and has providers for every cloud service, database, and API. If you inherit a legacy stack, it is probably running on Airflow. The ecosystem is enormous.

The tradeoff: Airflow’s DAG model is more verbose, and the development loop is slow. Testing a DAG locally requires a running Airflow environment. Managing the scheduler, workers, and metadata database is a real operational burden if you self-host.

What most startups should do

Start with Dagster. Specifically, start with the free Dagster open-source running locally or on a single server, and move to Dagster+ Cloud when you want the managed version.

The asset-based model is easier to reason about than DAGs for most data use cases. The dbt integration is the best in the ecosystem. And you can be up and running in an afternoon.

Add Airflow only if:

• You are joining a team that already runs Airflow and migration cost is not justified
• You need a specific provider that does not exist in Dagster’s ecosystem
• You have a dedicated platform engineer who knows Airflow

A minimal Dagster + dbt setup

 1# dagster_project/assets.py
 2from dagster import Definitions
 3from dagster_dbt import DbtCliResource, dbt_assets
 4from pathlib import Path
 5
 6DBT_PROJECT_DIR = Path(__file__).parent.parent / "dbt_project"
 7
 8@dbt_assets(manifest=DBT_PROJECT_DIR / "target" / "manifest.json")
 9def my_dbt_assets(context, dbt: DbtCliResource):
10    yield from dbt.cli(["build"], context=context).stream()
11
12defs = Definitions(
13    assets=[my_dbt_assets],
14    resources={"dbt": DbtCliResource(project_dir=str(DBT_PROJECT_DIR))},
15)

This exposes every dbt model as a Dagster asset. You can materialise individual models, see the full lineage graph, and set up schedules and sensors from the Dagster UI.

When to add orchestration

You do not need orchestration on day one. The right time is when any of these become true:

• You have more than one pipeline that depends on another finishing first
• A failed run has caused a missed report or a downstream incident
• You are manually checking whether jobs have run
• You cannot tell, without looking at the warehouse, whether last night’s pipeline succeeded

At that point, the investment pays for itself within a week.

You now have a data stack

If you have followed all four steps:

Connect a BI tool (Looker, Metabase, or even a spreadsheet via a connector) and your team has self-service access to reliable, tested data. That is the modern data stack, end to end.