SLAs for data: Spark & Data Eng best practices explained

Introduction

Data pipelines fail silently more often than we'd like to admit. A table doesn't update, a metric drifts, a report lands three hours late—and nobody notices until a stakeholder asks why last month's numbers don't match. This is where SLAs for data: Spark & Data Eng best practices explained becomes essential. An SLA (Service Level Agreement) for data sets concrete expectations around freshness and quality, transforming vague promises into measurable commitments. When your team and your stakeholders share the same definition of "acceptable," you catch problems early and build trust.

Define freshness per table

Freshness is the time lag between when data is generated and when it's available for consumption. But "fresh" means different things to different tables. A real-time trading feed needs sub-minute latency; a weekly aggregate can tolerate a 24-hour delay. The first step is documenting exactly what fresh means for each of your tables.

Start by auditing your data catalog. Ask the business: if this table were unavailable or late by X hours, what would break? Group tables into tiers based on criticality and use patterns.

Once you've defined freshness SLAs, encode them in your data platform. Most teams store this in a metadata service or configuration file that your orchestration tool can read.

# Example: freshness definitions in Python config
FRESHNESS_SLAS = {
    "user_events": {
        "max_age_minutes": 15,
        "expected_run_time": "every 5 minutes",
        "owner": "streaming-team",
    },
    "daily_user_metrics": {
        "max_age_hours": 24,
        "expected_run_time": "06:00 UTC",
        "owner": "analytics-team",
    },
    "campaign_performance": {
        "max_age_minutes": 30,
        "expected_run_time": "every 10 minutes",
        "owner": "data-engineering",
    },
}

In production, we store this in a DynamoDB table or Postgres, then fetch it in our job startup code. When a Spark job completes, it queries the catalog to find its own SLA and compares actual runtime against it.

Monitor and alert

Defining an SLA without monitoring it is like posting a speed limit with no enforcement. You need real-time visibility into whether tables are meeting their SLAs.

The core metrics to track are: last successful run time, current lag (now minus last data timestamp), and whether the table is in breach. Most orchestration frameworks (Airflow, Dagster) offer monitoring hooks. If you're running Spark jobs directly, you'll need custom instrumentation.

# Pseudo-code: check freshness after a Spark job completes
import time
from datetime import datetime, timedelta

def check_sla_breach(table_name: str, job_start_time: float) -> bool:
    """
    Check if a table's freshness SLA has been breached.
    Return True if in breach, False otherwise.
    """
    sla_config = FRESHNESS_SLAS.get(table_name)
    if not sla_config:
        return False
    
    # Query the table's last update timestamp
    max_age = sla_config.get("max_age_minutes", 60)
    last_update_ts = query_last_update_timestamp(table_name)
    current_age_minutes = (time.time() - last_update_ts) / 60
    
    is_breach = current_age_minutes > max_age
    
    # Send metric to monitoring backend
    send_metric(
        metric_name="data_freshness_minutes",
        value=current_age_minutes,
        tags={
            "table": table_name,
            "status": "breach" if is_breach else "ok",
        },
    )
    
    return is_breach

Push these metrics to a monitoring system (Datadog, Prometheus, CloudWatch). Set up alerts that trigger when freshness breaches occur. An SLA breach is not just a warning—it's an incident. Route alerts to Slack or PagerDuty so the on-call engineer can respond within minutes.

In our experience, teams that don't alert on SLA breaches often don't realize there's a problem until a business user complains. By then, the impact is real.

Communicate delays

When a pipeline runs late, transparency matters. The worst scenario is a stakeholder discovering a delay from a stale number rather than a proactive notification. Establish a clear communication protocol.

When a job approaches its SLA deadline (say, 80% of the way through the allowed window), send a warning to relevant channels. If it breaches, escalate immediately with context: when did it fail, which job, what was the error.

# Example: Slack notification on SLA breach
def notify_sla_breach(
    table_name: str,
    sla_minutes: int,
    actual_age_minutes: float,
    last_error: str = None,
):
    """Post SLA breach notification to Slack."""
    delay_minutes = actual_age_minutes - sla_minutes
    
    message = f"""
    :warning: *Data SLA Breach Detected*
    
    Table: `{table_name}`
    SLA: {sla_minutes} minutes
    Actual age: {actual_age_minutes:.1f} minutes
    Delay: *{delay_minutes:.1f} minutes over SLA*
    
    Owner: <@{get_table_owner(table_name)}>
    Runbook: https://wiki.company.com/runbooks/{table_name}-sla-breach
    """
    
    if last_error:
        message += f"\nLast error: ```{last_error}```"
    
    post_to_slack(channel="#data-incidents", text=message)

Document SLA expectations in your wiki so everyone knows what to expect. Include a runbook for each critical table: how to investigate delays, which logs to check, who to escalate to. When a delay happens, the first 10 minutes are crucial—a runbook helps you start investigations immediately instead of figuring out where to look.

Review breaches

An SLA breach is a learning opportunity. Set aside time each week to review breaches and identify patterns.

Common causes we've seen:

• Resource contention: A long-running job starves downstream jobs for cluster resources. Solution: tighter job scheduling or resource isolation.
• Upstream dependency failures: A table depends on another table that breached. Solution: add retry logic or buffer time in SLA definitions.
• Data volume growth: A job worked fine at 1 GB but now takes 3x longer at 100 GB. Solution: optimize query or increase cluster size.
• Schema changes: A new column in a source system breaks parsing logic. Solution: stricter schema validation in ingestion.

Track breach frequency and impact. If a table breaches more than once per quarter, its SLA is probably unrealistic and needs adjustment. If a breach affects 10 downstream reports, it's high-priority to fix. Use this data to prioritize infrastructure work.

# Example: store breach history for analysis
def record_sla_breach(
    table_name: str,
    sla_minutes: int,
    actual_age_minutes: float,
    root_cause: str,
    timestamp: datetime,
):
    """Log SLA breach to analytics table for later review."""
    breach_record = {
        "table_name": table_name,
        "sla_minutes": sla_minutes,
        "actual_age_minutes": actual_age_minutes,
        "delay_minutes": actual_age_minutes - sla_minutes,
        "root_cause": root_cause,
        "timestamp": timestamp.isoformat(),
    }
    
    # Insert into a Spark-managed table
    spark.createDataFrame([breach_record]).write.mode("append").saveAsTable(
        "metadata.sla_breaches"
    )

At the end of each month, run a report: how many breaches, which tables, which root causes, and what was the impact. Use this to inform quarterly planning. If you're seeing 15 breaches a month in non-critical tables, maybe you're over-alerting and need to adjust SLAs upward.

Common mistakes

• SLA too tight: Setting a 5-minute SLA on a table that inherently takes 8 minutes to compute sets the team up for failure. SLAs should be ambitious but achievable most of the time.
• No owner assigned: If nobody owns the SLA, nobody fixes breaches. Assign a team and individual owner for each critical table.
• Monitoring lag: Checking freshness only at job completion leaves a gap. Continuously monitor—if a table hasn't been updated in 90 minutes and the SLA is 60 minutes, alert immediately.
• Alert fatigue: If you alert on every minor breach, teams learn to ignore alerts. Reserve high-severity alerts for genuine customer impact.
• Forgetting to document dependencies: A table may be fresh but its upstream source may be stale. Document the dependency chain so you can trace root causes quickly.

Key takeaways

• Define freshness SLAs per table based on business impact, not technical convenience. A real-time table and a weekly table need very different expectations.
• Implement continuous monitoring that compares actual freshness against SLA definitions and alerts when breaches occur.
• Communicate proactively: notify stakeholders and owners when delays approach or exceed SLA thresholds, with clear context and a runbook.
• Review breach patterns weekly or monthly to identify systemic issues and adjust SLAs when they're consistently unrealistic.
• Assign clear ownership so SLA breaches are treated as incidents with rapid response, not noise to ignore.