Dagster and Prefect: a practical Spark & Data Eng guide

After 48 days of building Spark pipelines, you've likely hit the orchestration wall: cron jobs break, manual retries pile up, and tracing data lineage becomes a nightmare. Orchestration tools like Dagster and Prefect transform how we manage these workflows, each with a fundamentally different philosophy. This practical Spark & Data Eng guide walks you through both, so you can pick the right one for your team's workflow.

Why orchestration matters now

By this point in the 100-day series, you've written transformations that work locally. But production data pipelines live in an ecosystem: upstream delays, downstream dependencies, schema changes, and partial failures. Without orchestration, you're manually checking logs, rerunning jobs at 3am, and hoping nobody notices when a report is stale.

The orchestration layer solves three problems at once:

• Visibility: who depends on whom, what failed, why
• Reliability: automatic retries, partial rerun from failure points, alerting
• Scalability: running hundreds of jobs without babysitting

Airflow was the default for years, but Dagster and Prefect represent a new generation. They push data thinking closer to the code, rather than configuration-driven DAGs. Let's see what that means in practice.

Dagster: asset-based pipelines

Dagster inverts the traditional orchestration model. Instead of thinking in tasks and dependencies, you think in assets—the actual data artifacts your team owns. An asset is a versioned, materializable unit: a table, a feature store, a model, a report.

Here's a concrete scenario: you're building a machine learning feature pipeline for a ride-hailing platform. You have raw events, cleaned events, features for driver matching, and features for pricing. Traditionally, you'd model this as five tasks with hardcoded dependencies. In Dagster, you model it as five assets, and the dependencies emerge from code.

from dagster import asset, materialize
from pyspark.sql import SparkSession

@asset
def raw_events(context) -> None:
    spark = SparkSession.builder.appName("events").getOrCreate()
    df = spark.read.parquet("s3://events/raw/2024-01-15/")
    df.write.mode("overwrite").parquet("s3://events/materialized/raw/")
    context.log.info(f"Raw events: {df.count()} rows")

@asset
def cleaned_events(context) -> None:
    spark = SparkSession.builder.appName("events").getOrCreate()
    df = spark.read.parquet("s3://events/materialized/raw/")
    cleaned = df.filter(df.event_time.isNotNull()).dropDuplicates(["event_id"])
    cleaned.write.mode("overwrite").parquet("s3://events/materialized/cleaned/")
    context.log.info(f"Cleaned events: {cleaned.count()} rows")

@asset
def driver_features(cleaned_events, context) -> None:
    spark = SparkSession.builder.appName("features").getOrCreate()
    df = spark.read.parquet("s3://events/materialized/cleaned/")
    features = df.groupBy("driver_id").agg(
        F.count("*").alias("total_rides"),
        F.avg("rating").alias("avg_rating")
    )
    features.write.mode("overwrite").parquet("s3://features/driver/")
    context.log.info(f"Driver features computed")

Notice the dependency injection: driver_features(cleaned_events, context). Dagster resolves this automatically—it knows that driver_features depends on cleaned_events existing first. You never write DAG boilerplate.

Dagster's killer features for Spark teams:

• Partitioning: Assets can be partitioned by date, geography, or custom keys. Run only the partition that failed, not the whole pipeline.
• Lineage UI: The web UI shows which assets feed into which, with data stats and last-run metadata. No guessing "did this depend on that?"
• Type hints: Optional, but typing your asset outputs catches mismatches early.
• Sensors and schedules: Trigger jobs when files land or on a cron, with built-in templating for common patterns.

The learning curve is real—Dagster has concepts like definitions, resources, and ops that take a week to internalize. But once you're past that, the asset model maps cleanly to how data teams think.

Prefect: Python-first flows

Prefect takes a different angle: flows are just Python functions. You write normal Python, add decorators, and Prefect orchestrates it. No special DSL, no complex objects. This is massive for teams that value simplicity and iteration speed.

Same ride-hailing scenario, but in Prefect:

from prefect import flow, task, get_run_logger
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

@task(retries=2, retry_delay_seconds=30)
def load_raw_events():
    spark = SparkSession.builder.appName("events").getOrCreate()
    logger = get_run_logger()
    df = spark.read.parquet("s3://events/raw/2024-01-15/")
    logger.info(f"Loaded {df.count()} raw events")
    return df

@task(retries=2, retry_delay_seconds=30)
def clean_events(df):
    logger = get_run_logger()
    cleaned = df.filter(df.event_time.isNotNull()).dropDuplicates(["event_id"])
    logger.info(f"Cleaned to {cleaned.count()} rows")
    return cleaned

@task(retries=2, retry_delay_seconds=30)
def compute_driver_features(cleaned_df):
    features = cleaned_df.groupBy("driver_id").agg(
        F.count("*").alias("total_rides"),
        F.avg("rating").alias("avg_rating")
    )
    features.write.mode("overwrite").parquet("s3://features/driver/")
    return features

@flow(name="ride_hailing_features")
def feature_pipeline():
    raw = load_raw_events()
    cleaned = clean_events(raw)
    features = compute_driver_features(cleaned)
    return features

if __name__ == "__main__":
    feature_pipeline()

You can run this file locally with `python script.py`. It works. Then you deploy it to a Prefect server or cloud, and it's orchestrated. The function signature defines the dependency graph—no configuration, no decorators specifying wiring.

Why Prefect resonates with Spark teams:

• Run locally without a server: Great for development. Your laptop runs the same flow code as production, minus the Prefect backend.
• Built-in retries and caching: Decorate a task with `@task(cache_key_fn=...)` and Prefect skips it if inputs haven't changed.
• Subflows: Compose flows into flows. Break a massive pipeline into logical chunks without losing traceability.
• Dynamic mapping: `flow.map()` over a list, and Prefect runs tasks in parallel with minimal boilerplate.
• Incremental adoption: Start with a single flow, add orchestration gradually. No need to refactor everything on day one.

Prefect 2.x (the current version) stripped away much of the abstraction compared to 1.x. That's intentional—the team wants orchestration to be boring infrastructure, not the center of your world.

Better local development

One of the biggest pain points in the Spark world is the gap between local and production. You develop on your laptop, test with a subset of data, then push to the cluster and hope.

Orchestration tools actually improve this. Both Dagster and Prefect let you:

Run the full pipeline locally (if your machine has enough RAM) or against a local Spark cluster. You see real failures before production.

Mock external dependencies. In Dagster, you use resources; in Prefect, you parametrize tasks. Write a flow that reads from S3 in production but a local folder in development:

from prefect import task
import os

@task
def load_data(env: str = "dev"):
    if env == "dev":
        path = "/tmp/data/events.parquet"
    else:
        path = "s3://events/raw/2024-01-15/"
    
    spark = SparkSession.builder.appName("events").getOrCreate()
    return spark.read.parquet(path)

@flow
def feature_pipeline(env: str = "dev"):
    raw = load_data(env=env)
    # ... rest of pipeline

Partial runs. Re-run only failed tasks or from a specific checkpoint. In Prefect, you do this in the UI. In Dagster, you materialize specific assets. This cuts debugging time from hours to minutes.

Choose by team workflow

Both tools are production-ready. The choice depends on your team's rhythm:

Choose Dagster if:

• You manage many assets (10+ tables, features, models) and care deeply about lineage.
• Your team thinks in terms of "what are our data products?" rather than "what are our jobs?"
• You have experienced engineers who can climb the learning curve and want to invest in a system they'll use for years.
• You need strong multi-tenancy (multiple teams, separate asset catalogs).

Choose Prefect if:

• You have 5-15 jobs and want to get orchestration running this week, not next month.
• Your team values simplicity and "Python is the API."
• You have developers new to orchestration who benefit from "run it like a script" semantics.
• You iterate fast and want minimal friction between local and deployed code.

Consider both if you're a large organization. Some teams use Prefect for ETL, others use Dagster for the feature platform. They coexist.

Common mistakes

Mistake 1: Treating orchestration as a backup for bad code. Neither tool will save you from a Spark job that crashes mid-execution without checkpointing. Fix the job first; orchestration is the safety net, not the foundation.

Mistake 2: Not versioning your orchestration code. Your DAG/flow definition is source code. Put it in Git. Version it. Review it. Don't treat it as configuration that lives in the Airflow UI.

Mistake 3: Ignoring resource constraints. A Prefect flow that launches 100 parallel Spark tasks will kill your cluster. Respect memory, cores, and queue limits. Use task tags or Prefect's work pool system to route jobs to appropriately-sized resources.

Mistake 4: Over-partitioning in Dagster. Partitioning is powerful, but partition by business logic (date, region) not technical artifacts (file size, object ID). You'll regret it when you need to backfill across 10,000 tiny partitions.

Key takeaways

• Orchestration isn't optional at scale. By day 48, you have the Spark expertise; now invest in the glue that runs 100+ jobs reliably.
• Dagster's asset-based model is conceptually cleaner for data platforms with rich lineage requirements. Expect a learning curve.
• Prefect's function-as-flow approach is faster to adopt and excellent for small to mid-size teams or rapid prototyping.
• Both tools dramatically improve local development. Use this to your advantage—test offline, fail faster.
• The choice between them matters less than choosing one and committing. Pick based on your team's size, experience level, and project scope.
• Neither replaces good data engineering practices: checkpointing, idempotency, schema validation, and monitoring still fall on you.