Skip to content
ExponaBLOG
Series · Day 100 of 100

Apache Spark & Data Engineering

  1. 01
    Why data engineering matters: what developers should know
    Reliable data pipelines power analytics, AI and products.
  2. 02
    What Apache Spark is: a practical Spark & Data Eng guide
    A distributed engine for large-scale data processing.
  3. 03
    Spark architecture explained: what developers should know
    Driver, executors and the cluster manager.
  4. 04
    PySpark tutorial: getting started — a step-by-step guide
    Run Spark from Python.
  5. 05
    RDDs vs DataFrames vs Datasets: best practices explained
    Which Spark API to use.
  6. 06
    Spark DataFrame examples for Spark & Data Eng developers
    The operations you will use daily.
  7. 07
    Lazy evaluation in Spark for Spark & Data Eng developers
    Nothing runs until an action.
  8. 08
    Transformations vs actions: patterns, pitfalls and fixes
    Know what triggers work.
  9. 09
    Spark SQL: everything engineers need to know in practice
    Query DataFrames with SQL.
  10. 10
    Built-in functions: a practical Spark & Data Eng guide
    Avoid UDFs with Spark functions.
  11. 11
    Python UDFs and their cost: patterns, pitfalls and fixes
    User-defined functions, used carefully.
  12. 12
    Reading and writing data for Spark & Data Eng developers
    Sources and formats.
  13. 13
    File formats for data engineering: a step-by-step guide
    Parquet, Avro, ORC and more.
  14. 14
    Defining schemas explicitly: what developers should know
    Faster, safer reads.
  15. 15
    Partitioning data on disk: practical tips and trade-offs
    Organize files for fast queries.
  16. 16
    Spark partitions and parallelism: lessons from production
    How work is split.
  17. 17
    Shuffles explained: a practical Spark & Data Eng guide
    The most expensive operation in Spark.
  18. 18
    Spark performance tuning for Spark & Data Eng developers
    Make slow jobs fast.
  19. 19
    Broadcast joins: a Spark & Data Eng guide for developers
    Join a small table without shuffling.
  20. 20
    Data skew: everything engineers need to know in practice
    When one task takes forever.
  21. 21
    Adaptive Query Execution for Spark & Data Eng developers
    Spark tunes plans at runtime.
  22. 22
    Caching and persistence: real-world lessons and examples
    Reuse data across actions.
  23. 23
    Reading the Spark UI: a practical Spark & Data Eng guide
    Find where time goes.
  24. 24
    explain() and query plans: practical tips and trade-offs
    Understand what Spark will do.
  25. 25
    Spark memory management: real-world lessons and examples
    Executors and memory settings.
  26. 26
    ETL pipeline design: a practical Spark & Data Eng guide
    Extract, transform, load done well.
  27. 27
    ETL vs ELT: everything engineers need to know in practice
    Transform before or after loading?
  28. 28
    Medallion architecture: a practical guide for developers
    Bronze, silver and gold layers.
  29. 29
    Data lakes, warehouses and lakehouses: a practical guide
    Where analytics data lives.
  30. 30
    Delta Lake basics: a Spark & Data Eng guide for developers
    ACID tables on data lakes.
  31. 31
    Apache Iceberg basics in Apache Spark & Data Engineering
    An open table format for big data.
  32. 32
    Upserts and slowly changing dimensions, explained simply
    Track changes over time.
  33. 33
    Incremental processing: a practical guide for developers
    Process only what changed.
  34. 34
    Change data capture: a practical Spark & Data Eng guide
    Stream database changes.
  35. 35
    Spark Structured Streaming: patterns, pitfalls and fixes
    Stream processing with DataFrames.
  36. 36
    Watermarks and late data for Spark & Data Eng developers
    Handle events that arrive late.
  37. 37
    Exactly-once in streaming: practical tips and trade-offs
    Avoid duplicates and data loss.
  38. 38
    Spark with Kafka: a Spark & Data Eng guide for developers
    Real-time pipelines.
  39. 39
    Data quality checks: a practical Spark & Data Eng guide
    Catch bad data before it spreads.
  40. 40
    Deduplication strategies for Spark & Data Eng developers
    Remove duplicate records.
  41. 41
    Window functions in Spark: practical tips and trade-offs
    Rankings, running totals and latest records.
  42. 42
    Handling nulls in Spark: real-world lessons and examples
    Nulls break joins and aggregates.
  43. 43
    Working with dates and timestamps: a step-by-step guide
    Time is tricky in data pipelines.
  44. 44
    Nested data and JSON: a practical Spark & Data Eng guide
    Work with complex columns.
  45. 45
    Joins in Spark: Spark & Data Eng lessons from production
    Join types and strategies.
  46. 46
    Orchestrating pipelines with Airflow in Spark & Data Eng
    Schedule and monitor data workflows.
  47. 47
    Airflow best practices: a practical guide for developers
    Reliable DAGs.
  48. 48
    Dagster and Prefect: a practical Spark & Data Eng guide
    Modern orchestration alternatives.
  49. 49
    dbt for transformations: real-world lessons and examples
    SQL-based modeling with software practices.
  50. 50
    Data modeling for analytics: what developers should know
    Star schemas still matter.