Built-in functions: a practical Spark & Data Eng guide

Why built-in functions matter in production

We've all been tempted to write a quick user-defined function (UDF) in Spark when a transformation felt just slightly outside the box. Six months later, your pipeline is slow, your data team is frustrated, and you're debugging serialization errors at 2 AM. The reality is that built-in functions: a practical Spark & Data Eng guide should be your first stop, not your last resort. Spark's Catalyst optimizer understands native functions deeply, but it treats your UDF as a black box. This article walks through the built-in landscape so you actually use them instead of reinventing wheels.

pyspark.sql.functions covers most needs

The pyspark.sql.functions module is vast—over 200 functions covering date, string, array, numeric, and aggregate operations. Most transformations you'll encounter in production fit neatly into this toolbox. The key insight is that these functions are written in Scala and optimized at the bytecode level, whereas a Python UDF must serialize data, call the Python interpreter for every row, and deserialize the result back. The performance difference isn't marginal; it's often 10x or more.

Let's start with a real scenario: you're building a customer event pipeline. Events arrive with timestamps in milliseconds since epoch, and you need to extract the date, hour, and day of week for aggregations.

from pyspark.sql import functions as F
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("event_pipeline").getOrCreate()

# Sample events
events = spark.createDataFrame([
    (1001, 1700000000000),
    (1002, 1700086400000),
], ["user_id", "event_ts_ms"])

# Convert milliseconds to timestamp, then extract components
result = events.select(
    "user_id",
    F.col("event_ts_ms").cast("timestamp").alias("event_time"),
    F.to_date(F.col("event_ts_ms") / 1000).alias("event_date"),
    F.hour(F.col("event_ts_ms") / 1000).alias("event_hour"),
    F.dayofweek(F.col("event_ts_ms") / 1000).alias("day_of_week")
)

result.show()

Notice we didn't write a single line of Python logic. The Catalyst optimizer sees these function calls, recognizes them as leaf operations, and pushes them down through the execution plan. If you'd written a UDF for the same logic, Catalyst would have given up optimizing that entire expression tree.

Date and time functions: handling temporal data

Temporal operations are among the most common in data pipelines. Whether you're calculating retention windows, bucketing events into hours, or computing time-to-event metrics, built-in date functions are your foundation.

from datetime import datetime, timedelta

# Sample transactions
transactions = spark.createDataFrame([
    (1, "2024-01-15", 299.99),
    (2, "2024-01-20", 50.00),
    (1, "2024-02-10", 120.00),
], ["user_id", "transaction_date", "amount"])

result = transactions.select(
    "user_id",
    "transaction_date",
    "amount",
    F.to_date("transaction_date").alias("parsed_date"),
    F.year("transaction_date").alias("year"),
    F.month("transaction_date").alias("month"),
    F.quarter("transaction_date").alias("quarter"),
    F.date_add("transaction_date", 30).alias("date_plus_30_days"),
    F.datediff(F.current_date(), "transaction_date").alias("days_ago")
)

result.show()

The datediff, date_add, date_sub, and date_trunc functions handle the bulk of window calculations. For more complex scenarios—like "start of month" or "same day last year"—combine date_trunc with date_add:

# Month-over-month comparison setup
events_with_windows = transactions.select(
    "user_id",
    F.date_trunc("month", "transaction_date").alias("month_start"),
    F.add_months(F.date_trunc("month", "transaction_date"), -1).alias("prev_month_start"),
    "amount"
)

events_with_windows.show()

String functions: cleaning and transforming text

String operations are bread and butter in ETL. Cleaning user input, normalizing categories, extracting patterns—all of it has a built-in function. A common mistake is writing regex in Python when Spark has it baked in.

# Customer records with messy strings
customers = spark.createDataFrame([
    (101, "  John Doe  ", "john.doe@EXAMPLE.COM"),
    (102, "jane_smith", "Jane.Smith@Example.com"),
    (103, "bob", "bob@domain.co.uk"),
], ["customer_id", "name", "email"])

clean = customers.select(
    "customer_id",
    F.trim(F.initcap("name")).alias("clean_name"),
    F.lower("email").alias("email_normalized"),
    F.length("name").alias("name_length"),
    F.substring("email", 1, F.instr("email", "@") - 1).alias("email_local_part"),
    F.regexp_extract("email", r"@([a-z0-9.]+)", 1).alias("email_domain"),
    F.when(F.rlike("email", r".*\.com$"), "commercial")
        .when(F.rlike("email", r".*\.co\.uk$"), "UK")
        .otherwise("other").alias("email_region")
)

clean.show()

The built-in functions here—trim, initcap, lower, substring, regexp_extract, and rlike—are compiled and vectorized. Writing equivalent logic in a UDF would serialize millions of strings to the Python worker processes, killing throughput. For array operations within strings (like splitting), use split and array_join:

# Parse and reconstruct pipe-delimited tags
records = spark.createDataFrame([
    (1, "python|data|etl"),
    (2, "go|backend|devops"),
], ["record_id", "tags"])

enriched = records.select(
    "record_id",
    F.split("tags", "\\|").alias("tags_array"),
    F.size(F.split("tags", "\\|")).alias("tag_count"),
    F.array_join(F.split("tags", "\\|"), ", ").alias("tags_readable")
)

enriched.show()

Array functions: working with nested structures

Semi-structured data—especially from APIs and event streams—often contains arrays. Rather than exploding, processing in Python, and collecting back, Spark's array functions let you transform in place.

# Event data with nested actions
events = spark.createDataFrame([
    (1, ["view", "add_to_cart", "purchase"]),
    (2, ["view", "view", "bounce"]),
    (3, ["purchase"]),
], ["user_id", "actions"])

analysis = events.select(
    "user_id",
    "actions",
    F.size("actions").alias("action_count"),
    F.array_contains("actions", "purchase").alias("purchased"),
    F.element_at("actions", 1).alias("first_action"),
    F.element_at("actions", -1).alias("last_action"),
    F.array_distinct("actions").alias("unique_actions"),
    F.array_join("actions", " -> ").alias("action_sequence")
)

analysis.show()

For more advanced needs, use transform to apply a function to each element, or filter to select matching elements:

# Filter and transform arrays inline
refined = events.select(
    "user_id",
    F.filter("actions", lambda x: F.length(x) > 4).alias("long_actions"),
    F.transform("actions", lambda x: F.upper(x)).alias("actions_upper")
)

refined.show()

Conditional logic with when/otherwise

SQL's CASE WHEN translates directly to Spark via the when and otherwise functions. These are essential for business logic: segmentation, flags, categorization, and rule-based transformations.

# Customer segmentation based on order history
orders = spark.createDataFrame([
    (1, 500.00),
    (2, 2500.00),
    (3, 150.00),
    (4, 5000.00),
], ["user_id", "lifetime_value"])

segmented = orders.select(
    "user_id",
    "lifetime_value",
    F.when(F.col("lifetime_value") >= 1000, "vip")
        .when(F.col("lifetime_value") >= 500, "premium")
        .when(F.col("lifetime_value") >= 100, "standard")
        .otherwise("prospect").alias("segment"),
    F.when(F.col("lifetime_value") >= 1000, 0.15)
        .when(F.col("lifetime_value") >= 500, 0.10)
        .otherwise(0.05).alias("discount_rate")
)

segmented.show()

Chaining when clauses works, but for many conditions, consider a coalesce with multiple case_when expressions or—for truly complex logic spanning hundreds of rules—a lookup join to a dimension table. The rule of thumb: if your when chain exceeds five levels, you're probably better served by a reference table.

Built-ins are optimized by Catalyst

Spark's Catalyst optimizer recognizes built-in functions and folds them into its logical plan. It understands predicate pushdown, constant folding, and expression simplification for native functions. Consider this example:

# Catalyst recognizes this is deterministic and optimizable
data = spark.range(1000000).select(
    F.col("id"),
    (F.col("id") * 2 + F.col("id") * 3).alias("calculated"),  # Can fold to id * 5
    F.year(F.current_timestamp()).alias("current_year")  # Evaluated once
)

# Inspect the physical plan
data.explain(mode="extended")

When you run explain, you'll see that Spark simplifies the arithmetic and evaluates the timestamp function once per task, not once per row. For UDFs, Catalyst falls back to conservative defaults: it can't push filters past a UDF, can't parallelize safely, and can't apply the sort or join optimizations it would normally use.

The optimizer also vectorizes operations. String and array functions run on columnar blocks, not row-by-row, making them orders of magnitude faster than iterative Python logic.

Common mistakes

• Writing UDFs for common tasks: Before reaching for udf(), check if the function already exists in pyspark.sql.functions. Nine times out of ten, it does.
• Chaining too many when/otherwise clauses: If you have 20+ conditions, join to a reference table instead. It's clearer and faster.
• Forgetting to cast timestamps: Spark treats timestamps as Long (milliseconds) by default in many sources. Always cast to timestamp before applying date functions, or divide milliseconds by 1000 first.
• Using concat instead of array_join: For readability and performance, prefer array_join over multiple concat calls when working with arrays.
• Not checking for null: Built-in functions often have null-handling semantics that differ from Python. Use coalesce, ifnull, or when to handle edge cases explicitly.

Key takeaways

• Lean on pyspark.sql.functions before writing UDFs. Over 200 functions cover the vast majority of real-world transformations.
• Date functions handle temporal transformations; string and array functions handle text and nested data. Master to_date, date_add, regexp_extract, split, and array_join first.
• Use when and otherwise for conditional logic, but escalate complex rule sets to reference table joins.
• Catalyst recognizes built-in functions and optimizes them aggressively: predicate pushdown, constant folding, and vectorization. UDFs block all of this.
• Performance gains from using built-ins over UDFs are often 5x to 20x. On billion-row datasets, that's the difference between a 1-minute job and a 10-minute job.