Python packaging basics: real-world lessons and examples

Introduction

By day 70, you've built enough backend services that sharing code between them isn't optional anymore. When your authentication module lives in one service, your domain models in another, and your shared utilities scattered across three repositories, you're either duplicating work or creating a maintenance nightmare. Python packaging basics: real-world lessons and examples shows you how to properly structure, version, and distribute internal packages so your team can depend on them reliably without reinventing the wheel.

We've learned these lessons the hard way—through failed deployments, version conflicts, and late-night debugging sessions. This guide distills what actually works in production environments where multiple services depend on shared code.

Why Python packaging matters at scale

When you're working alone or with one microservice, keeping everything in a single repository makes sense. But once you have three or more services that need to share domain logic, validation rules, or API client libraries, packaging becomes infrastructure. Without it, you're choosing between:

• Copying code into every service (maintenance nightmare)
• Using git submodules (fragile and painful)
• Creating a proper package that multiple services can import (the right choice)

Proper packaging also forces you to think about interfaces and stability. A package with a documented API is harder to break accidentally than shared code buried in a monorepo.

Starting with pyproject.toml and a build backend

The days of setup.py and setup.cfg are behind us. A modern Python package uses pyproject.toml as the single source of truth for project metadata, dependencies, and build configuration. This file is so important because it's standardized—tools like pip, poetry, and build understand it without custom configuration.

Here's a realistic example for an internal authentication library:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "acme-auth"
version = "0.2.1"
description = "Shared authentication utilities for Acme services"
readme = "README.md"
requires-python = ">=3.11"
license = {text = "MIT"}
authors = [
    {name = "Platform Team"}
]
dependencies = [
    "pydantic>=2.0",
    "python-jose[cryptography]>=3.3.0",
    "python-dotenv>=1.0.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "pytest-cov>=4.0",
    "black>=23.0",
    "ruff>=0.1.0",
]

[tool.hatch.build.targets.wheel]
packages = ["src/acme_auth"]

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "--cov=src/acme_auth --cov-report=term-only"

[tool.black]
line-length = 100
target-version = ['py311']

[tool.ruff]
target-version = "py311"
line-length = 100

Notice the key decisions here: we're using hatchling as the build backend (lightweight and reliable), we've pinned requires-python to 3.11 minimum, and we've separated development dependencies into an optional group. The package points to a src/acme_auth directory structure, which prevents accidental imports of the uninstalled version during development.

When you build this package with python -m build, it creates both a wheel and a source distribution. The wheel is what gets installed; it's pre-built and much faster than installing from source.

Semantic versioning for internal packages

Internal packages live in a different world than open-source libraries. Your downstream consumers are your own services, and you control the upgrade schedule. But semantic versioning still matters because it lets you signal what kind of change you're making.

Semantic versioning uses three numbers: MAJOR.MINOR.PATCH.

• PATCH (0.2.1 → 0.2.2): Bug fixes only. Always safe to upgrade.
• MINOR (0.2.0 → 0.3.0): New features that are backward compatible. Your code won't break, but you can use new functionality.
• MAJOR (1.0.0 → 2.0.0): Breaking changes. You must update your code to stay compatible.

In practice at a growing team, you'll bump versions like this:

• Your auth library adds a new optional parameter to verify_token() → minor version bump (0.2.0 → 0.3.0)
• You refactor the internal structure of token caching but the public API stays the same → patch bump (0.3.0 → 0.3.1)
• You remove the deprecated legacy_auth() function that was marked for removal → major bump (0.3.5 → 1.0.0)

When you make a breaking change, communicate it explicitly. Add a CHANGELOG entry and give teams notice before the major version lands in your private index. Some teams require pull requests that acknowledge the dependency change before the bot allows the merge.

Automation helps here. Use a git tag strategy: tag every release with v0.2.1, and have your CI pipeline automatically build and publish when a tag is pushed. This removes the manual "publish step" and keeps versions in sync with git history.

Private index on GitHub Packages or CodeArtifact

Internal packages don't belong on PyPI. You need a private index that's accessible only to your team. The two most practical options are GitHub Packages (if you're already using GitHub) and AWS CodeArtifact (if you're in the AWS ecosystem).

GitHub Packages setup

GitHub Packages is free if your repository is public or if you're on a paid GitHub plan. Publishing is straightforward: create a GitHub Actions workflow that builds and publishes on every release tag.

name: Publish package

on:
  push:
    tags:
      - 'v*'

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: python -m pip install build
      - run: python -m build
      - run: |
          python -m pip install twine
          twine upload dist/* \
            --repository-url https://npm.pkg.github.com/your-org \
            --username __token__ \
            --password ${{ secrets.GITHUB_TOKEN }}

On the consuming side, create a .pypirc file or set environment variables:

[distutils]
index-servers =
    github

[github]
repository: https://npm.pkg.github.com/your-org
username: __token__
password: ghp_xxxxxxxxxxxxxxxxxxxx

Then in your consuming service's requirements, reference it by full URL or configure pip to use the custom index. For a service that needs both public and private packages, configure pip with multiple indexes:

pip install --index-url https://npm.pkg.github.com/your-org --extra-index-url https://pypi.org/simple/ acme-auth

AWS CodeArtifact approach

If you're on AWS, CodeArtifact integrates cleanly with IAM and is often cheaper at scale. Setup is similar but uses AWS credentials:

aws codeartifact login \
  --tool pip \
  --repository your-repo \
  --domain your-domain \
  --domain-owner 123456789012 \
  --region us-east-1

This updates your pip configuration automatically. Publishing happens the same way as GitHub Packages—your CI pipeline uses AWS credentials to authenticate to CodeArtifact.

Whichever you choose, the key principle is: your internal packages live in a private, authenticated index. You don't have to worry about naming collisions with PyPI, and you can control who accesses what.

Pin dependencies in applications, not libraries

This is where many teams get it wrong. The rule is simple: libraries declare minimum versions; applications pin exact versions.

Your internal auth library should have loose constraints:

# acme-auth's pyproject.toml
dependencies = [
    "pydantic>=2.0",
    "python-jose[cryptography]>=3.3.0",
]

This says: "I need at least pydantic 2.0 and python-jose 3.3.0, but I work with newer versions too." If another package needs pydantic 2.5, there's no conflict.

Your consuming application (e.g., your order service) should have exact pins:

# order-service's requirements.txt or pyproject.toml
acme-auth==0.2.1
pydantic==2.6.1
python-jose==3.3.0
fastapi==0.109.0
sqlalchemy==2.0.23

Why? Because you've tested this exact set of versions together. When the auth library says it works with pydantic>=2.0, that doesn't mean every version works perfectly. Pinning in applications ensures reproducible deployments and makes debugging easier—you know exactly what's running in production.

Use pip-compile (from pip-tools) to generate pinned requirements from a looser specification:

# requirements.in
acme-auth>=0.2.0
fastapi>=0.100
sqlalchemy>=2.0

# Run: pip-compile requirements.in
# Generates: requirements.txt with exact pins

This workflow gives you the best of both worlds: flexibility for library maintenance and reproducibility for application deployment.

Structuring your package for long-term maintenance

Once your package is published and other services depend on it, changing things gets harder. Plan your structure accordingly.

Use a src/ layout:

acme-auth/
├── src/
│   └── acme_auth/
│       ├── __init__.py
│       ├── exceptions.py
│       ├── models.py
│       ├── auth.py
│       └── jwt.py
├── tests/
├── README.md
├── pyproject.toml
└── CHANGELOG.md

This prevents the installed package from being shadowed by a local directory during development. It also makes it clear which code is part of the distributed package.

Keep your public API small and documented. Only export what you intend for external use in __init__.py:

# src/acme_auth/__init__.py
from .auth import create_token, verify_token
from .exceptions import TokenExpired, InvalidToken

__version__ = "0.2.1"
__all__ = ["create_token", "verify_token", "TokenExpired", "InvalidToken"]

Internal functions (prefixed with _) can change freely. Public functions are promises to your consumers.

Common mistakes

• Publishing with version 0.1.0 and never bumping it: Version 0.x.x signals instability. Pick a real version scheme from day one, even if your API might change.
• Pinning dependencies in libraries: If your auth library requires exactly pydantic==2.0.0, and another package needs pydantic==2.5.0, installation fails. Use ranges instead.
• Not documenting breaking changes: When you remove a function or change a signature, write it prominently in CHANGELOG.md and bump the major version. Your teammates will thank you.
• Forgetting to update the version in pyproject.toml before publishing: Automate this with tools like bump2version or rely entirely on git tags and CI to manage versions.
• Storing credentials in code or .pypirc files checked into git: Use GitHub Secrets, AWS Secrets Manager, or environment variables. Never commit credentials.

Key takeaways

• Use pyproject.toml with a modern build backend (hatchling or flit) as your single source of truth for package metadata.
• Adopt semantic versioning (MAJOR.MINOR.PATCH) and use git tags to mark releases automatically in CI.
• Publish to a private index (GitHub Packages or CodeArtifact) where only your team can access internal packages.
• Libraries declare loose dependency constraints (>=X.Y.Z); applications pin exact versions for reproducibility.
• Use a src/ layout and keep your public API small and intentional.
• Document breaking changes and give teams notice before major version bumps.
• Automate publishing via CI so versions stay in sync with git history and credentials are never exposed.