Spark on Kubernetes: a practical Spark & Data Eng guide

Introduction

If your team runs Kubernetes in production, deploying Spark jobs as containers feels natural—but it requires different thinking than traditional cluster managers. Spark on Kubernetes: a practical Spark & Data Eng guide covers the essentials for getting batch and streaming workloads into your K8s infrastructure without reinventing your data pipeline architecture.

We'll walk through direct spark-submit integration, the Spark Operator for resource management, dynamic allocation strategies, and when Kubernetes makes sense for your data team. This guide assumes you're already comfortable with Spark fundamentals and basic Kubernetes operations.

Why Kubernetes for Spark?

Kubernetes gives you infrastructure as code, resource isolation, and cost efficiency through bin-packing. If your organization already runs services on K8s, adding Spark workloads means one platform to monitor, secure, and scale. You eliminate the complexity of maintaining separate YARN or standalone Spark clusters.

The trade-off is operational overhead: you need to understand both Spark and Kubernetes semantics, manage container images, and debug across two abstraction layers. It's worth it when teams already have Kubernetes expertise and want to consolidate infrastructure.

spark-submit with Kubernetes Master

The simplest approach is submitting Spark jobs directly to your Kubernetes cluster using spark-submit with --master k8s://. Spark handles pod creation, driver lifecycle, and executor scaling without additional software.

Here's a typical workflow. First, ensure your Kubernetes cluster has a service account with permissions to create and delete pods:

# Create service account for Spark
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role \
  --clusterrole=edit \
  --serviceaccount=default:spark

Next, submit a Spark job specifying the Kubernetes master endpoint:

spark-submit \
  --master k8s://https://YOUR_CLUSTER_IP:6443 \
  --deploy-mode cluster \
  --name my-spark-job \
  --class org.example.MyApp \
  --driver-memory 2g \
  --executor-memory 4g \
  --num-executors 4 \
  --executor-cores 2 \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.kubernetes.namespace=default \
  --conf spark.kubernetes.container.image=my-spark-image:latest \
  s3://bucket/my-app.jar

Key configurations here:

• --master k8s://https://YOUR_CLUSTER_IP:6443 points to your Kubernetes API server
• --deploy-mode cluster runs the driver inside the cluster (not on your local machine)
• spark.kubernetes.container.image specifies your Spark application container image
• spark.kubernetes.authenticate.driver.serviceAccountName binds the driver pod to a service account

Spark will create a driver pod and spawn executor pods on worker nodes. Pod logs stream to stdout, and you can monitor with kubectl logs or your observability stack. Once the job completes, Spark cleans up executor pods; the driver pod remains for log inspection until manually deleted.

For development, you might keep the driver pod around to query HDFS or S3 for intermediate results. In production, automate cleanup with a TTL or external job controller.

Spark Operator for Job Resources

Direct spark-submit works, but teams managing many jobs prefer Kubernetes custom resources. The open-source Spark Operator extends Kubernetes with a SparkApplication CRD, giving you GitOps-friendly declarative job definitions.

Install the Spark Operator via Helm:

helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator/charts
helm install spark-operator spark-operator/spark-operator \
  --namespace spark-operator \
  --create-namespace

Define a SparkApplication resource:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: etl-daily-job
  namespace: data-team
spec:
  type: Scala
  mode: cluster
  image: "us-docker.pkg.dev/my-project/spark:3.5.0"
  imagePullPolicy: IfNotPresent
  mainApplicationFile: "s3://bucket/etl.jar"
  arguments:
    - "--date"
    - "2025-01-15"
  sparkVersion: "3.5.0"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 2
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    memory: "2g"
    memoryOverhead: "512m"
    serviceAccount: spark
    labels:
      team: data-engineering
  executor:
    cores: 2
    instances: 4
    memory: "4g"
    memoryOverhead: "1g"

Apply it to your cluster:

kubectl apply -f etl-job.yaml

The Spark Operator watches for SparkApplication changes and manages the full lifecycle. You get automatic retries, pod cleanup on completion, and integration with Kubernetes events. Query job status naturally:

kubectl get sparkapplications -n data-team
kubectl describe sparkapplication etl-daily-job -n data-team
kubectl logs -n data-team driver-pod-name

For scheduling recurring jobs, combine SparkApplication with Kubernetes CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-etl-cron
  namespace: data-team
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: spark
          containers:
          - name: spark-submit
            image: spark:3.5.0
            command:
            - /bin/bash
            - -c
            - |
              kubectl apply -f - <

This pattern keeps job definitions in version control and enables auditable deployments via ArgoCD or Flux.

Dynamic Allocation on Kubernetes

Kubernetes and dynamic allocation are natural partners. Instead of fixing executor counts at submit time, let Spark scale up and down based on pending tasks. This saves money and reduces cluster fragmentation.

Enable dynamic allocation with these configurations:

spark-submit \
  --master k8s://https://YOUR_CLUSTER_IP:6443 \
  --conf spark.dynamicAllocation.enabled=true \
  --conf spark.dynamicAllocation.minExecutors=1 \
  --conf spark.dynamicAllocation.maxExecutors=20 \
  --conf spark.dynamicAllocation.initialExecutors=4 \
  --conf spark.dynamicAllocation.schedulerBacklogTimeout=30s \
  --conf spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=30s \
  --conf spark.kubernetes.executor.request.cores=1 \
  --conf spark.kubernetes.executor.limit.cores=2 \
  s3://bucket/my-app.jar

Key settings:

• minExecutors and maxExecutors set the scaling bounds
• initialExecutors starts this many executors before any task scheduling (avoids cold-start latency)
• schedulerBacklogTimeout triggers scale-up when tasks queue for this duration
• sustainedSchedulerBacklogTimeout removes executors after this idle period
• spark.kubernetes.executor.request.cores and .limit.cores separate Kubernetes requests from Spark overcommit

Practical example: an ETL job starts with 4 executors. As data shuffles, the backlog grows; Spark provisions more until reaching 20. Once shuffle completes and executors idle for 30 seconds, Spark removes them down to the minimum. Kubernetes can then bin-pack other workloads into freed capacity.

For streaming jobs, use a higher minExecutors to avoid churn on micro-batches:

--conf spark.dynamicAllocation.minExecutors=5 \
--conf spark.dynamicAllocation.maxExecutors=50

Monitor executor churn via Spark metrics: dynamicAllocation.executors.numberAdded and .numberRemoved. High churn suggests backlog timeout is too aggressive.

Good for Teams Already on Kubernetes

Spark on Kubernetes shines when your team already operates Kubernetes for microservices, APIs, or workers. You consolidate observability, auth, and deployment tooling:

• Observability: One logging stack (ELK, Loki, Datadog) collects Spark pod logs alongside application logs. One metrics system (Prometheus) scrapes Spark metrics endpoints.
• Authentication: Use Kubernetes service accounts and RBAC for Spark jobs instead of separate Kerberos or LDAP infrastructure.
• Resource governance: Kubernetes resource quotas and namespace isolation enforce fair-share scheduling. Teams can't starve each other.
• CI/CD integration: Build Spark container images in your existing image pipeline. Deploy via GitOps tools (ArgoCD, Flux) alongside other workloads.
• Cost tracking: Kubernetes cost allocation tools attribute pod costs to namespaces or labels, giving you per-team or per-job cost visibility.

A real scenario: your data team runs three pipelines (ETL, analytics, ML inference). On Kubernetes, all three coexist in the same cluster, sharing worker nodes. During peak hours, auto-scaling adds nodes; off-hours, jobs bin-pack tightly. With YARN or standalone Spark, you'd manage three separate clusters or over-provision one.

The operational cost—understanding Kubernetes pod scheduling, CNI networking, persistent volumes for data locality—is justified when you already have Kubernetes expertise in-house and want consolidated operations.

Common Mistakes

1. Forgetting executor memory overhead: Spark's executor memory doesn't include JVM overhead, Python processes, or shuffle buffers. Set spark.kubernetes.executor.memoryOverhead to 25–30% of executor memory, or Kubernetes will OOM-kill pods. The same applies to the driver.

2. Using scratch storage for shuffle: If executors run on ephemeral pods with emptyDir volumes, shuffle spills to disk fail on pod eviction. For large shuffles, mount persistent volumes or use external shuffle service (complex on Kubernetes). For smaller jobs, increase memory and adjust shuffle partitions to reduce spill.

3. Ignoring pod affinity and node selectors: Executors might scatter across nodes with poor network locality, slowing data reads and shuffles. Use node affinity to co-locate executors on the same rack or availability zone. For on-premises, pin executors to nodes with local HDFS data.

4. Not setting resource requests correctly: Setting requests equal to limits (e.g., --driver-memory 2g --conf spark.kubernetes.driver.request.cores=1 with no overcommit) wastes cluster capacity. Use requests for scheduling and limits for max usage; Kubernetes can bin-pack better.

5. Debugging container image issues in production: Always test Spark images locally or in dev before deploying to prod. Missing JARs, Java version mismatches, or Python library dependencies become hard to diagnose in CI/CD. Use a multi-stage Dockerfile to keep images lean and reproducible.

Key Takeaways

• Spark on Kubernetes is viable when your team operates Kubernetes already; it consolidates infrastructure and reduces operational silos.
• Direct spark-submit with --master k8s:// works for simple one-off jobs, but SparkApplication CRDs scale better for recurring workloads and enable GitOps patterns.
• Dynamic allocation on Kubernetes reduces cost and cluster fragmentation by scaling executors based on workload demand; set realistic backlog timeouts to avoid thrashing.
• Always account for memory overhead, configure node affinity for data locality, and test container images before production deployment.
• Use Kubernetes namespaces and resource quotas to enforce fair-share scheduling and cost attribution across data teams.