Spark and Python dependencies: best practices explained

Introduction

Distributing Python code across a Spark cluster is straightforward; distributing the libraries that code depends on is where teams get stuck. When your PySpark job needs NumPy, Pandas, or a custom internal library, the executors on worker nodes won't have those dependencies unless you explicitly ship them. Spark and Python dependencies: best practices explained covers the strategies that work in production—from packaging wheels to freezing versions—so your jobs run consistently whether you're testing locally or scaling to hundreds of executors.

Package wheels for executors

The most reliable way to ship dependencies is as pre-built wheels. A wheel is a binary package format that Python tools like pip understand natively; it avoids the compilation overhead and inconsistencies that come with source distributions.

Here's why wheels matter: when Spark distributes your PySpark job to an executor, that executor doesn't have access to your development machine's pip cache or any globally installed packages. The executor has a fresh Python environment—typically the system Python or a minimal environment—and nothing else. If your code imports pandas and pandas isn't present, the job fails immediately. Building wheels ahead of time and bundling them ensures every executor gets the exact binary artifact you tested.

To create a wheel for your own code, use pip wheel or Poetry:

# With pip
pip wheel --no-cache-dir --wheel-dir ./dist ./my_package/

# With Poetry
poetry build --format wheel

For third-party dependencies, download wheels for your target Python version and platform:

pip download pandas numpy scikit-learn \
  --python-version 39 \
  --platform manylinux2014_x86_64 \
  --only-binary=:all: \
  --dest ./deps/

When submitting the job, pass all wheels via --py-files:

spark-submit \
  --py-files deps/pandas-2.0.0-py3-none-any.whl,deps/numpy-1.24.0-py3-none-any.whl \
  job.py

Spark extracts these wheels to each executor's Python path before running your code. The executor can then import the modules as if they were installed locally. This approach is portable: the same wheel works across identical platforms, and there's no compilation happening on the cluster.

Use virtual environment archives

For more complex dependency trees, bundling a pre-built virtual environment as a zip or tar archive is faster and more reliable than shipping individual wheels. This is especially useful when you have many nested dependencies or platform-specific packages like psycopg2.

Create a virtual environment on a machine that matches your cluster's OS and Python version:

python3.9 -m venv spark_env
source spark_env/bin/activate
pip install -r requirements.txt
deactivate

Then archive it:

cd spark_env
tar czf spark_env.tar.gz .
# or zip
zip -r spark_env.zip .

Submit the job and tell Spark to extract and use that environment:

spark-submit \
  --conf spark.pyspark.python=./spark_env/bin/python \
  --archives spark_env.tar.gz \
  job.py

Spark unpacks the archive on each executor, and the spark.pyspark.python config points the executor to the Python executable inside it. Every executor then runs with the same isolated environment, including the same versions of every dependency.

The trade-off: a virtual environment archive can be 100–500 MB depending on your dependency footprint. If your job processes small batches across many partitions, the upload and extraction overhead might outweigh the benefit. For long-running or compute-heavy jobs, it's worth it.

Pin versions

Unpinned dependencies are a classic source of silent failures in production. When you specify pip install pandas without a version, pip pulls the latest major version available at install time. If you build wheels on your development machine one month and the cluster installs from a fresh requirements.txt the next month, you could end up with different versions of Pandas—and subtle API or behavior changes can break your Spark job without any obvious error.

Always freeze your dependencies:

pip freeze > requirements.txt

This generates a file like:

pandas==2.0.3
numpy==1.24.4
pyarrow==12.0.1
requests==2.31.0

Now every build and deployment installs the exact same versions. When you need to upgrade, do it deliberately in a separate branch, test it end-to-end, and then merge the updated requirements.txt.

If you use Poetry, it automatically maintains a locked dependency file (poetry.lock) that serves the same purpose:

poetry export --format requirements.txt --output requirements.txt --with-credentials

For Spark jobs especially, pinning is non-negotiable. Pandas 1.x and 2.x have different internal layouts for DataFrames. NumPy versions can change integer behavior. Arrow serialization formats have evolved. A patch-level version bump in a transitive dependency can silently alter the output of your job or cause it to hang on a specific partition. Frozen versions let you control exactly when and how those changes reach production.

Managed platforms simplify this

If you're running Spark on Databricks, AWS Glue, or another managed service, much of this complexity is handled for you—though understanding the mechanics still helps when things go wrong.

Databricks lets you install packages at the cluster level via the UI or API, and it manages the download, storage, and distribution to all nodes. You can also attach wheel files directly to a job run. For custom code, you upload a wheel and reference it in --py-files just like on-premises, but Databricks handles the infrastructure.

AWS Glue supports a similar model: you can specify Python dependencies in the job configuration, and Glue downloads them and prepares the environment. For more control, you package everything into a zip and pass it as a job artifact.

Google Cloud Dataproc and Azure Synapse follow similar patterns. All of these platforms abstract away the manual wheel-building and version-pinning steps to some degree, but the underlying principle is the same: dependencies must be explicit and immutable.

Even on managed platforms, we recommend freezing your own application code as a wheel and uploading it. This separates your custom logic from third-party dependencies and makes it easier to iterate on your code without rebuilding the entire dependency stack.

Common mistakes

Forgetting to test on a single-machine cluster first. A job that works fine in local mode can fail spectacularly on a distributed cluster if the executors lack a dependency. Always test with --master local[*] or on a small cluster before scaling.

Using system packages instead of wheels. It's tempting to SSH into worker nodes and run apt install python3-pandas or pip install pandas directly. This creates manual, fragile state that drifts over time and is impossible to reproduce. Ship everything as code.

Mixing local and cluster paths. If your job references a local file path like /home/user/my_data.csv, it won't exist on the executors. Use HDFS, S3, or another shared filesystem, or embed small config files into your wheel.

Not accounting for Python version mismatches. If you build wheels with Python 3.9 but your cluster runs Python 3.10, they may not be compatible. Always check the platform tag in the wheel filename and match it to your cluster's OS and Python version.

Assuming pip install on the driver will reach the executors. Running pip install pandas on the driver machine before submitting a job does not install it on the executors. The driver and executors are separate Python processes, often on different machines. Dependencies must be shipped explicitly.

Key takeaways

• Package your dependencies as wheels and pass them via --py-files for portable, reproducible Spark jobs.
• For complex dependency trees, create and archive a full virtual environment to avoid compilation overhead on the cluster.
• Always pin versions in requirements.txt or poetry.lock; unpinned dependencies silently break jobs between environments.
• Test locally and on a small cluster before scaling; missing dependencies often aren't caught until the job runs on actual executors.
• Managed platforms like Databricks and Glue handle some of the distribution mechanics, but the principle remains: make dependencies explicit and immutable.
• Keep your own application code separate from third-party libraries by building a wheel for your code and freezing third-party versions independently.