Spark on AWS: Spark & Data Eng best practices explained

Most teams running Spark in production eventually hit the same wall: managing infrastructure becomes a tax on velocity. When you move Spark on AWS: Spark & Data Eng best practices explained, the cloud gives you choices—but knowing which service fits your workload, how to wire up storage, and where to cut costs without sacrificing reliability makes the difference between a smooth data pipeline and a budget hole.

We've walked this path across multiple production systems. Here's what actually works.

Options: Choosing Your Spark Service

AWS gives you four real paths to run Spark. Each trades off control, operational burden, and cost differently.

EMR: The Flexible Foundation

Amazon EMR (Elastic MapReduce) is the traditional choice. You launch a cluster with a master node, worker nodes, and a Hadoop/Spark stack. You get SSH access, can install whatever you want, and have full control over JVM tuning and library versions.

When to use it: Complex workloads with custom Scala libraries, iterative data exploration, or jobs that need Jupyter notebooks running on the cluster itself. If your team is comfortable with cluster operations, EMR is mature and battle-tested.

Reality check: You're managing node lifecycle, scaling policies, and cluster termination. We've seen teams waste cycles on cluster health rather than pipeline logic.

EMR Serverless: No Cluster Tax

EMR Serverless abstracts the cluster away entirely. You submit a Spark job and AWS handles provisioning, scaling, and shutdown. You pay for vCPU-hours and memory consumed, nothing more.

When to use it: Scheduled batch jobs with predictable resource needs. Data quality checks. ETL pipelines that run nightly. If your workload doesn't need persistent state or interactive access, this cuts operational overhead dramatically.

Example scenario: A team processes customer events nightly, aggregates metrics, and lands results in S3. EMR Serverless provisions just enough capacity for 45 minutes of work, then shuts down. No idle clusters, no on-call for cluster health.

AWS Glue: Serverless ETL with Catalog Integration

Glue is AWS's opinionated serverless ETL service. It includes Spark under the hood, plus a Data Catalog that auto-discovers schema from your data lake. You write Python or Scala jobs; Glue handles provisioning.

When to use it: Teams already deep in AWS services who want tight integration with Athena, Lambda, or Step Functions. If schema discovery and cataloging are priorities, Glue's built-in Data Catalog beats managing Hive metastores separately.

Trade-off: Less flexibility on Spark version and configuration. Fewer debugging tools. But for straightforward ETL, the integration wins.

Databricks on AWS: Managed Lakehouse

Databricks runs clusters on your AWS account but abstracts infrastructure entirely. You get notebooks, SQL, Delta Lake, and MLflow all in one platform. Databricks handles patching, optimization, and security.

When to use it: Organizations investing in a unified analytics and ML platform. Teams that want Spark without the ops burden and can justify the premium pricing. If your data science and engineering teams need shared notebooks and version control, Databricks' workspace features earn their cost.

Cost reality: You're paying for convenience and features on top of AWS infrastructure spend. Right for scale, not always right for small batch jobs.

Storage: S3 as Your Data Lake

S3 is where your Spark data lives in AWS. It's not a database—it's object storage—but when architected right, it becomes a reliable data lake.

Structure Your Buckets

Use a standard layout:

s3://my-data-lake/
  raw/                  # Ingested data, never modified
    events/
      year=2024/month=01/day=15/
    api_dumps/
  processed/            # Cleaned, deduplicated data
    customer_events/
    order_summary/
  analytics/            # Final tables, ready for BI tools
    dashboards/
    reports/

This partitioning matters. When Spark reads s3://my-data-lake/raw/events/year=2024/month=01/, it prunes partitions at query time. Your 10TB table becomes a 50GB scan.

Write Data Efficiently

Spark writes many small files by default. S3 doesn't like that. Coalesce or repartition before writing:

df.coalesce(10) \
  .write \
  .mode("overwrite") \
  .parquet("s3://my-data-lake/processed/customer_events/")

Ten Parquet files is manageable. Ten thousand is a headache for downstream readers and S3 list operations.

Use Parquet, Not CSV

CSV is human-readable. Parquet is fast. Parquet stores schema, compresses 5-10x better, and Spark reads it significantly faster. Unless you have a specific reason for CSV (legacy system integration, compliance), use Parquet for everything in S3.

Catalog: Glue Data Catalog for Table Discovery

Without a catalog, your data lake is just a pile of files. The Glue Data Catalog maintains table metadata: column names, types, partitions, and location in S3.

Why It Matters

When your Spark job reads from S3, it can use the catalog instead of scanning the file to infer schema. Other tools (Athena, QuickSight, Redshift) share the same catalog, so a table defined once is discoverable everywhere.

Populate the Catalog

Option 1: Crawlers. Glue Crawlers scan S3 paths and auto-register tables. Set them to run on a schedule. Simple, but crawlers are slow and occasionally get schema wrong on messy data.

Option 2: Write from Spark. Use Spark's Catalog API to register tables as you write:

df.write \
  .format("parquet") \
  .mode("overwrite") \
  .option("path", "s3://my-data-lake/processed/orders/") \
  .saveAsTable("orders_processed")

The table is now in the Glue Data Catalog under the default database. Athena queries see it immediately.

Option 3: Iceberg or Delta Lake. If you're using Delta tables (more on this in a future piece), they auto-register with Glue. Same with Apache Iceberg. Modern table formats handle cataloging for you.

Cross-Tool Consistency

Once a table is in Glue, Athena can query it with SQL:

SELECT order_id, total_amount
FROM orders_processed
WHERE year = 2024 AND month = 1
LIMIT 10;

Your analysts don't need Spark—they don't need to know where the file lives. They just query the table name. That's the value of a catalog.

Cost: Spot Instances for Batch Jobs

AWS Spot instances cost 70-90% less than On-Demand, but AWS can reclaim them with a two-minute warning. For batch Spark workloads, this is a straightforward trade-off.

When to Use Spot

Your nightly ETL can tolerate a two-minute interruption. It retries, no user impact. Your interactive Jupyter cluster cannot—data scientists lose their work. Use Spot for batch, On-Demand for interactive.

EMR Spot Configuration

Tell EMR to use Spot for core and task nodes:

aws emr create-cluster \
  --name my-batch-job \
  --release-label emr-6.15.0 \
  --instances \
      InstanceGroups='[
        {Name=Master,Market=ON_DEMAND,InstanceCount=1,InstanceType=m5.xlarge},
        {Name=Core,Market=SPOT,BidPrice=0.15,InstanceCount=2,InstanceType=m5.xlarge},
        {Name=Task,Market=SPOT,BidPrice=0.15,InstanceCount=5,InstanceType=m5.xlarge}
      ]' \
  --applications Name=Spark \
  --log-uri s3://my-logs-bucket/emr/

Keep the master On-Demand (loss of master terminates the cluster). Bid Spot for workers. If Spot prices spike, jobs fail—but you've saved 80% on typical runs, so occasional retries are a net win.

EMR Serverless Cost Optimization

EMR Serverless doesn't offer Spot, but you pay only for what you use. The cost optimization is automatic: run during off-peak hours if possible, and right-size your job config:

aws emr-serverless start-job-run \
  --application-id  \
  --execution-role-arn  \
  --job-driver '{
    "sparkSubmitJobDriver": {
      "entryPoint": "s3://my-bucket/jobs/process_events.py",
      "sparkSubmitParameters": "--conf spark.driver.memory=2g --conf spark.executor.memory=4g --conf spark.executor.cores=2"
    }
  }'

Smaller driver/executor memory = lower vCPU-hours billed. Start conservative, profile your job, then scale up only what's needed.

Common Mistakes

• Choosing the service for the wrong reason: EMR Serverless is great for batch, but teams buy it expecting to run Jupyter. Then they're frustrated by lack of interactive features. Pick the tool that matches your workload, not your hope for what it could be.
• Writing thousands of small S3 objects: Every write spawns a partition. Later, listing and reading becomes glacial. Coalesce before writing. Period.
• Ignoring the Data Catalog: Teams skip it because "we know where our data is." Six months in, you have 40 S3 paths, three different schemas for the same concept, and no way to find anything. Catalog from day one.
• Over-provisioning for batch jobs: Setting up a 20-node EMR cluster for a 2-minute job because "we might need it." Start small, let auto-scaling grow it if needed. EMR has aggressive scaling policies.
• Not monitoring S3 costs: Glue crawlers that run hourly, or Spark jobs that write uncompressed JSON. Audit S3 bucket sizes monthly. Parquet compression alone saves most teams 60-70% on storage.

Key Takeaways

• Choose your Spark service based on workload shape: EMR for flexibility, EMR Serverless for batch automation, Glue for AWS-native ETL, Databricks for unified analytics.
• S3 is your data lake foundation. Structure it with clear raw/processed/analytics layers, partition aggressively, and use Parquet format.
• Populate Glue Data Catalog early. It's the single source of truth for table metadata across Spark, Athena, and other AWS analytics tools.
• Use Spot instances for EMR batch jobs to cut costs 70-90%. Keep master nodes On-Demand; bid workers aggressively.
• Right-size your cluster or serverless job config. Over-provisioning burns budget; profile real workloads and scale incrementally.
• Monitor S3 storage monthly. Switch to Parquet, coalesce writes, and delete old partitions. Small disciplines compound to major savings.