AWS Glue basics: a Spark & Data Eng guide for developers

AWS Glue basics: a Spark & Data Eng guide for developers

If you've built ETL pipelines on Spark before, you know the operational burden: provisioning clusters, managing dependencies, babysitting job failures. AWS Glue removes that friction by offering serverless Spark execution without the infrastructure headache. This guide walks you through Glue's core components and shows you how to move fast without losing control.

What is AWS Glue and why it matters

AWS Glue is a fully managed ETL service that runs your Spark jobs without you provisioning or managing clusters. You write your transformation logic in PySpark or Scala, point Glue at your data sources, and it handles scaling, retries, and monitoring. The appeal for data engineers is clear: fewer operational concerns, faster time to production, and predictable costs.

The service sits at the intersection of three pain points we've all faced:

• Schema discovery across data lakes is manual and error-prone
• Infrastructure management distracts from building transformations
• Job orchestration requires separate tooling and expertise

Glue addresses each through its three main components: Glue Jobs (Spark execution), Glue Crawlers (schema discovery), and the Glue Data Catalog (centralized metadata). Understanding how these interact is key to building reliable pipelines quickly.

Glue Jobs run Spark

At its heart, a Glue Job is just managed Spark running on AWS infrastructure. You write your transformation logic in PySpark exactly as you would for any Spark cluster—the difference is Glue provisions and scales the workers automatically.

Here's a typical flow: read data from S3, apply transformations, write results back. Glue handles the cluster lifecycle; you focus on the logic.

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.sql import SparkSession

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'S3_INPUT', 'S3_OUTPUT'])

spark = SparkSession.builder.appName(args['JOB_NAME']).getOrCreate()

# Read raw data
df = spark.read.parquet(args['S3_INPUT'])

# Simple transformation
df_cleaned = df.filter(df.status == 'active') \
    .select('id', 'name', 'email', 'created_at') \
    .dropDuplicates(['email'])

# Write transformed data
df_cleaned.write.mode('overwrite').parquet(args['S3_OUTPUT'])

spark.stop()

Several things to note here. We use getResolvedOptions to inject job parameters at runtime—S3 paths, database names, whatever you need. This keeps your code portable across environments. The SparkSession is already initialized for you in Glue; you don't create it from scratch.

Glue Jobs support both Python and Scala, and come in a few flavors: Standard (G.1X and G.2X worker types for traditional Spark workloads), Ray (for distributed Python), and Streaming (for Kafka or Kinesis). For most ETL work, you'll use Standard. You specify worker count and instance type when you create the job, and Glue auto-scales within those bounds.

One practical detail: Glue has a 5-minute startup time before your code even runs. This matters for frequent small jobs; batching or scheduling longer intervals often makes sense.

Crawlers discover schemas

Manual schema management is tedious and error-prone. A Glue Crawler solves this by automatically scanning data in S3, inferring column types and structure, and registering the schema in the Data Catalog. You run a crawler once (or on a schedule), and the metadata stays current.

Say you've got raw JSON logs in S3 that you want to query. Without a crawler, you'd inspect a few files, manually define columns and types, and hope new fields don't break your pipeline. A crawler does this automatically:

# This is pseudo-code showing crawler output registration
# The actual crawler runs via AWS console or API
{
  "database": "raw_logs",
  "table": "user_events",
  "location": "s3://my-bucket/events/",
  "columns": [
    {"name": "event_id", "type": "string"},
    {"name": "user_id", "type": "bigint"},
    {"name": "event_type", "type": "string"},
    {"name": "timestamp", "type": "timestamp"},
    {"name": "properties", "type": "struct<key:string,value:string>"}
  ],
  "format": "json"
}

Crawlers support JSON, CSV, Parquet, ORC, and more. You point a crawler at an S3 path, set a schedule (hourly, daily, or on-demand), and it updates the schema whenever new data arrives. This is invaluable for data lakes where schemas evolve.

In practice, we typically use crawlers for raw ingestion layers and manually define schemas for curated data where we control the structure. Crawlers are fast and cost-effective—you pay per DPU-hour, and a typical crawl takes minutes.

Data Catalog for tables

The Glue Data Catalog is a central metadata repository. Every table Glue knows about—whether created by a crawler, defined manually, or produced by a job—lives here. Think of it as your organization's data dictionary, queryable and accessible to any tool that integrates with Glue.

The Catalog stores:

• Database and table definitions (schema, columns, types)
• S3 locations (where the actual data lives)
• Partition information (if your tables are partitioned)
• Format and serialization details (JSON, Parquet, etc.)
• Custom metadata (tags, descriptions, business glossary links)

When you write a Glue Job, you often read directly from the Catalog:

from awsglue.catalog import Database, Table
from awsglue.context import GlueContext
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("glue_catalog_read").getOrCreate()
glue_context = GlueContext(spark)

# Read table directly from Catalog
dyf = glue_context.create_dynamic_frame.from_catalog(
    database="raw_logs",
    table_name="user_events"
)

# Convert to DataFrame for standard Spark operations
df = dyf.toDF()

# Your transformation
df_filtered = df.filter(df.event_type.isin(['purchase', 'signup']))

# Write back to Catalog
glue_context.write_dynamic_frame.from_catalog(
    frame=glue_context.create_dynamic_frame.from_df(df_filtered),
    database="processed",
    table_name="user_events_filtered"
)

This code uses Glue's DynamicFrame abstraction—it's Glue's way of handling semi-structured data gracefully. DynamicFrames apply schema dynamically during processing, which saves hassle when data is messy. You can convert to standard Spark DataFrames whenever you need more control.

The Catalog also integrates with Athena (query tables with SQL), Redshift Spectrum (query external data), EMR, and third-party tools like dbt and Databricks. This centralization prevents schema drift and makes data discovery easier across your org.

Pay per job run

Glue's pricing model is straightforward: you pay for the compute resources used during job execution, measured in DPU-hours (Data Processing Units). One DPU is 4 vCPU and 16 GB of memory. There are no idle charges, no cluster management fees, and no surprise infrastructure costs.

For a typical scenario: a job using 10 G.2X workers (20 DPUs total) running for 15 minutes costs about $0.50. That's dramatically cheaper than provisioning an on-demand Spark cluster for the same work, which might run $20-50 per hour.

Here's what you actually pay for:

• Glue Jobs: $0.44 per DPU-hour (G.2X workers) or $0.34 (G.1X). Billed by the second, minimum 1 minute per run.
• Glue Crawlers: $0.44 per DPU-hour. A typical crawler uses 5-10 DPUs and completes in minutes.
• Data Catalog: Free for the first million objects; $1 per 100k objects after that. In most organizations, this is negligible.
• Development endpoints and notebooks: Charged per hour if you use them for interactive development.

The lack of cluster overhead makes Glue cost-effective for bursty, variable workloads. If you're running the same large job constantly, a permanent Spark cluster might be cheaper. But for typical data pipelines—daily, weekly, or on-demand jobs—Glue's pay-per-run model wins.

One cost optimization: use G.1X workers for I/O-heavy jobs (reading/writing large files) and G.2X for compute-heavy transformations. Matching the right worker type to your workload cuts both runtime and cost.

Common mistakes

Ignoring the 5-minute startup time. Glue Jobs have inherent latency before your code runs. Wrapping a 30-second job in Glue makes no sense; batch work or use Lambda instead for fast, lightweight tasks.

Not using the Data Catalog. Writing hardcoded S3 paths in your jobs defeats the purpose. Use the Catalog for schema discovery and portability. If paths change, update the Catalog once; all jobs adapt automatically.

Over-provisioning worker count. Glue scales well, but more workers don't always mean faster jobs. Profile your workload first. Reading a small file with 100 workers wastes money; focus on the actual bottleneck (usually I/O).

Neglecting job parameters and environment separation. Hardcoding database names and S3 buckets makes code brittle. Use getResolvedOptions to parameterize everything, and maintain separate Glue jobs or job parameters for dev, staging, and production.

Treating Crawlers as a substitute for data validation. A crawler infers schema from data, but it can't catch data quality issues or enforce business rules. Use Crawlers for discovery; add validation logic in your Glue Jobs or downstream tests.

Key takeaways

• Glue Jobs are managed Spark without cluster provisioning—write PySpark like usual, and Glue handles scaling and infrastructure.
• Crawlers automatically discover and register schemas in the Data Catalog, reducing manual metadata management.
• The Data Catalog is your central metadata hub; use it for job inputs, cross-team discovery, and integration with Athena, Redshift, and other AWS tools.
• Pricing is per DPU-hour with no idle charges, making Glue cost-effective for variable, bursty workloads.
• Use job parameters and the Catalog to keep code portable across environments and avoid hardcoding S3 paths.
• Profile before scaling; startup overhead matters for small, frequent jobs.
• Combine Glue with EventBridge, Step Functions, or Airflow for orchestration and scheduling at scale.