Handling nulls in Spark: real-world lessons and examples

Why nulls matter in Spark

We've all debugged a join that returned fewer rows than expected, only to discover that nulls silently dropped entire partitions of data. Handling nulls in Spark: real-world lessons and examples teaches us that null values are not just missing data—they're a silent killer of data pipelines if you don't understand how Spark treats them. In production systems processing millions of records, a mishandled null can cascade through downstream aggregations, window functions, and joins, breaking assumptions about data completeness.

This article draws from real incidents across e-commerce orders, CRM contact deduplication, and trading settlement systems. We'll walk through the practical tools Spark gives us and the patterns that actually work when nulls matter.

Understanding Spark's null semantics

Spark follows SQL semantics where any comparison or operation involving a null returns null (not true or false). This trips up engineers coming from Python or JavaScript, where null/None behaves differently.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when

spark = SparkSession.builder.appName("NullHandling").getOrCreate()

# Create a sample DataFrame
data = [
    ("alice", 1000, None),
    ("bob", None, "US"),
    ("charlie", 500, "US"),
    (None, 2000, "CA"),
]
df = spark.createDataFrame(data, ["name", "amount", "country"])
df.show()

# This filter will NOT catch nulls
result = df.filter(col("amount") > 500)
result.show()
# Output: only charlie and the row with amount=2000
# Bob's row is excluded because None > 500 evaluates to null (falsy in filter)

That last behavior is crucial: filtering on a condition where a column is null won't include the null row. It's not included or excluded explicitly—it simply vanishes from the filter result. You need explicit null checks.

fillna and dropna: first-line defense

fillna() replaces null values with a specified literal or value, and dropna() removes rows containing nulls. These are your first tools when you need to handle incomplete data.

When to use fillna: In an e-commerce pipeline, if a product category is missing, you might default it to "uncategorized" so downstream reports don't break. In a CRM system, a missing phone number might be replaced with an empty string to satisfy downstream API contracts.

from pyspark.sql.functions import col, lit

# Fill nulls with a literal
df_filled = df.fillna({
    "amount": 0,
    "country": "UNKNOWN"
})
df_filled.show()
# Output:
# | name    | amount | country |
# | alice   | 1000   | UNKNOWN |
# | bob     | 0      | US      |
# | charlie | 500    | US      |
# | null    | 2000   | CA      |

# Fill only specific columns
df_filled_selective = df.fillna({"amount": 0})
df_filled_selective.show()

# Fill with a forward-fill or backward-fill pattern (order matters)
df_filled_forward = df.fillna(0, subset=["amount"]).fillna("UNKNOWN", subset=["country"])

When to use dropna: When downstream systems absolutely cannot tolerate nulls in critical columns. For instance, a trading settlement system might drop any record missing a trade ID or settlement date rather than guess.

# Drop rows where any column is null
df_dropped = df.dropna()
df_dropped.show()
# Output: only charlie's row remains

# Drop rows where specific columns are null
df_dropped_amount = df.dropna(subset=["amount"])
df_dropped_amount.show()
# Output: alice and charlie (removes bob and the row with None name)

# Drop rows where at least N non-null values exist
df_dropped_any = df.dropna(how="any", thresh=2)
df_dropped_any.show()

In a real CRM deduplication job, we've used dropna(subset=["email", "phone"]) to filter to records where at least one contact method exists, then applied matching logic only to those.

coalesce for fallback logic

coalesce() returns the first non-null value from a list of columns or expressions. This is invaluable when you want to provide a hierarchy of fallbacks.

from pyspark.sql.functions import coalesce

# Scenario: prioritize updated email over original email
data_emails = [
    ("alice", "alice@old.com", "alice@new.com"),
    ("bob", "bob@work.com", None),
    ("charlie", None, "charlie@personal.com"),
    ("dave", None, None),
]
df_emails = spark.createDataFrame(data_emails, ["name", "original_email", "updated_email"])

df_coalesced = df_emails.select(
    col("name"),
    coalesce(col("updated_email"), col("original_email")).alias("email_to_use")
)
df_coalesced.show()
# Output:
# | name    | email_to_use             |
# | alice   | alice@new.com            |
# | bob     | bob@work.com             |
# | charlie | charlie@personal.com     |
# | dave    | null                     |

We used this pattern in a ride-hailing platform where driver ratings came from multiple sources. coalesce(in_app_rating, sms_rating, fallback_rating) ensured we always had a value to work with, falling through each source in priority order.

Null-safe equality with eqNullSafe

Standard equality (==) returns null when either side is null, which breaks joins and deduplication logic. Spark provides eqNullSafe to treat nulls as equal to each other.

# Standard equality: fails with nulls
df_left = spark.createDataFrame([("alice", 100), (None, 200)], ["name", "amount"])
df_right = spark.createDataFrame([(None, 50)], ["name", "value"])

# This join produces no match for the None row
joined_standard = df_left.join(
    df_right,
    df_left["name"] == df_right["name"],
    "inner"
)
joined_standard.show()
# Output: empty result set

# Null-safe equality: treats nulls as matching
joined_nullsafe = df_left.join(
    df_right,
    df_left["name"].eqNullSafe(df_right["name"]),
    "inner"
)
joined_nullsafe.show()
# Output:
# | name | amount | name | value |
# | null | 200    | null | 50    |

In a CRM system, we matched accounts on phone number, but some phone numbers were null (company accounts without direct lines). Without eqNullSafe, null phone numbers never matched with each other, fragmenting the data. Switching to null-safe equality consolidated those records as intended.

Document null meanings

The most overlooked part of null handling is documenting what a null actually means. A null phone number might mean "intentionally not provided," "not yet collected," or "invalid." These have different remediation strategies.

# Create explicit columns that document null semantics
from pyspark.sql.functions import when, isNull

df_documented = df.select(
    col("name"),
    col("amount"),
    col("country"),
    # Add explicit flags
    when(isNull(col("amount")), "not_provided")
        .otherwise("provided")
        .alias("amount_status"),
    when(isNull(col("country")), "not_provided")
        .otherwise("provided")
        .alias("country_status")
)
df_documented.show()

Better yet, use schema comments or a companion metadata table to codify null meanings. In a trading system, we documented that a missing settlement_date meant "awaiting counterparty confirmation," not "no settlement scheduled." This prevented the operations team from escalating false alarms.

In a travel platform processing booking data, we added a booking_stage column that explicitly captured whether nulls in payment_method indicated "user hasn't paid yet" or "payment declined, requires reentry." Downstream teams stopped confusing the two scenarios.

Common mistakes

Mistake 1: Using standard operators on nullable columns without null checks

# BAD: silently drops rows where amount is null
df.filter(col("amount") > 500)

# GOOD: explicitly handle nulls
df.filter((col("amount") > 500) | col("amount").isNull())
# or
df.filter(col("amount").isNull() | (col("amount") > 500))

Mistake 2: Assuming fillna() propagates through all operations

fillna() is not sticky. If you call it once and then add a new column via a join, that new column can introduce nulls again. Fill defensively, close to point of use.

Mistake 3: Forgetting that groupBy/aggregations silently include null as a group key

# Null is treated as a distinct group
df.groupBy("country").count().show()
# Output includes a row where country=null with its count

If your business logic doesn't want to report on null groups, filter them out explicitly.

Mistake 4: Not testing null cases in unit tests

Most test datasets are clean. Production data has nulls. Add a test case with nulls in every critical column before shipping a transformation.

Key takeaways

• Null comparisons always return null: Use explicit isNull() or isNotNull() checks in filters; standard operators silently drop null rows.
• fillna() is immediate, not permanent: Apply it close to where you need the non-null guarantee, and reapply after joins or new columns.
• coalesce() provides safe fallback logic: Chain multiple columns or expressions to implement sensible defaults without hard-coded values.
• eqNullSafe() treats nulls as equal: Use it in join conditions when you want null-to-null matching; use standard equality only if you explicitly want to exclude null matches.
• Document null semantics in code and metadata: A null phone number is not the same as a null settlement date. Write it down so future maintainers (and your future self) understand the intent.
• Test with nulls in production scenarios: Unit tests with clean data hide null-related bugs. Simulate realistic null distributions from your sources.