Python project structure for large codebases in practice
As your Python backend grows from a few thousand lines to hundreds of thousands, folder chaos becomes a real bottleneck. We've seen teams waste weeks hunting down where business logic lives, debugging circular imports, and arguing about where new code belongs. Python project structure for large codebases in practice isn't glamorous, but it's the difference between a codebase that scales and one that collapses under its own weight.
Why structure matters more than you think
Most guides tell you to organize by technical layers: models/, views/, services/, utils/. That approach works until you have fifty models and no idea which belong to billing versus user management. We've found that splitting by feature—not by layer—keeps large codebases navigable and lets teams own their domains.
The real pain comes later: when a junior developer imports billing logic into the catalog module, when three teams accidentally create conflicting utilities, when onboarding takes days because conventions aren't documented. A solid structure with enforced boundaries solves these problems before they metastasize.
Package by feature — users/, billing/, catalog/ with their own models and services
Start by identifying your core business domains. In e-commerce, that might be users, billing, catalog, orders, and shipping. In a SaaS platform, it could be tenants, subscriptions, auth, and integrations. Each domain gets its own top-level package with everything it needs.
src/
├── users/
│ ├── __init__.py
│ ├── models.py
│ ├── schemas.py
│ ├── service.py
│ ├── repository.py
│ └── routes.py
├── billing/
│ ├── __init__.py
│ ├── models.py
│ ├── schemas.py
│ ├── service.py
│ ├── repository.py
│ └── routes.py
├── catalog/
│ ├── __init__.py
│ ├── models.py
│ ├── schemas.py
│ ├── service.py
│ ├── repository.py
│ └── routes.py
├── shared/
│ ├── __init__.py
│ ├── exceptions.py
│ ├── decorators.py
│ └── database.py
└── main.py
Each feature package contains its own models (ORM definitions), schemas (Pydantic or similar), service layer (business logic), repository layer (data access), and routes (HTTP endpoints). This makes it immediately clear where to add new functionality and which files to modify when a domain changes.
Here's what a user service might look like:
# src/users/service.py
from .repository import UserRepository
from .models import User
from ..shared.exceptions import UserNotFound
class UserService:
def __init__(self, repo: UserRepository):
self.repo = repo
def get_user(self, user_id: int) -> User:
user = self.repo.find_by_id(user_id)
if not user:
raise UserNotFound(f"User {user_id} not found")
return user
def create_user(self, email: str, name: str) -> User:
if self.repo.find_by_email(email):
raise ValueError(f"Email {email} already exists")
return self.repo.create(email=email, name=name)
And the billing service stays isolated:
# src/billing/service.py
from .repository import InvoiceRepository
from .models import Invoice
from ..users.repository import UserRepository
class BillingService:
def __init__(self, invoice_repo: InvoiceRepository, user_repo: UserRepository):
self.invoice_repo = invoice_repo
self.user_repo = user_repo
def create_invoice(self, user_id: int, amount: float) -> Invoice:
user = self.user_repo.find_by_id(user_id)
if not user:
raise ValueError(f"User {user_id} not found")
return self.invoice_repo.create(user_id=user_id, amount=amount)
Notice: billing imports from users only where necessary (the user lookup). It doesn't import user services, routes, or schemas. The boundary is clear.
Shared kernel — Common utilities in one small package
Every codebase needs shared code: database connections, exception classes, decorators, middleware. The temptation is to scatter these across a huge utils/ folder. Instead, keep a lean shared/ package for things that truly belong everywhere.
# src/shared/exceptions.py
class AppError(Exception):
"""Base exception for the application."""
status_code = 500
class UserNotFound(AppError):
status_code = 404
class InsufficientBalance(AppError):
status_code = 422
class UnauthorizedAccess(AppError):
status_code = 403
# src/shared/decorators.py
import functools
from typing import Callable
import time
def log_execution(func: Callable) -> Callable:
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.time()
result = func(*args, **kwargs)
duration = time.time() - start
print(f"{func.__name__} took {duration:.3f}s")
return result
return wrapper
def require_auth(func: Callable) -> Callable:
@functools.wraps(func)
def wrapper(request, *args, **kwargs):
if not request.user:
raise UnauthorizedAccess()
return func(request, *args, **kwargs)
return wrapper
# src/shared/database.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
DATABASE_URL = "postgresql://user:password@localhost/dbname"
engine = create_engine(DATABASE_URL)
SessionLocal = sessionmaker(bind=engine)
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
The shared/ package should be small and stable. If it grows beyond 500 lines of code, you're probably mixing concerns. Any utility that's specific to one domain (say, invoice formatting logic) belongs in that domain's package, not in shared.
Enforce boundaries — import-linter fails the build on forbidden imports
Structure alone doesn't prevent bad imports. We've all seen from billing.service import BillingService pop up in catalog code months later. That's when you need import-linter, a tool that catches forbidden imports before they reach production.
Install it:
pip install import-linter
Create a .importlinter configuration file:
[importlinter]
root_package = src
[importlinter:contract:1]
name = Billing module isolation
type = forbidden
source_modules =
src.billing
forbidden_modules =
src.catalog
src.orders
[importlinter:contract:2]
name = Catalog should not import users service
type = forbidden
source_modules =
src.catalog
forbidden_modules =
src.users.service
src.billing.service
[importlinter:contract:3]
name = Only routes import routes
type = forbidden
source_modules =
src.users.service
src.users.repository
src.billing.service
forbidden_modules =
src.*.routes
Run it in your CI/CD pipeline:
lint-imports
If someone commits a cross-domain import that violates your contracts, the build fails immediately. This stops bad patterns before code review becomes a debate.
Document conventions — A short CONTRIBUTING guide saves hours of review
A two-page CONTRIBUTING.md file prevents endless back-and-forth during code review. Here's what we include:
# Contributing to the Backend
## Project Structure
We organize code by business domain, not technical layer:
- `src/users/` — User accounts, profiles, authentication
- `src/billing/` — Invoices, payments, subscriptions
- `src/catalog/` — Products, categories, inventory
- `src/shared/` — Exceptions, database setup, decorators
Each domain package contains: models, schemas, service, repository, routes.
## Adding a new feature
1. Identify which domain it belongs to. If it spans domains, it belongs in the
smallest domain that covers it. If it's truly cross-cutting, add it to shared.
2. Create or modify files in that domain:
- Add ORM models to `models.py`
- Add Pydantic schemas to `schemas.py`
- Add business logic to `service.py`
- Add data access to `repository.py`
- Add HTTP routes to `routes.py`
3. Inject dependencies via the service layer. Don't create global instances.
4. Use exceptions from `shared.exceptions` or create domain-specific ones.
## Import rules
- Domains can only import from their own package and `shared/`.
- Routes can import services; services can import repositories.
- Never import routes from services or repositories.
- Never import one domain's service from another domain's service.
If you need data from another domain, use its repository or public API.
## Testing
Each feature package includes a `tests/` subfolder mirroring the structure:
```
src/users/
├── tests/
│ ├── test_service.py
│ ├── test_repository.py
│ └── test_routes.py
```
Use fixtures to inject dependencies. Run tests with `pytest src/`.
## Code review
We check for:
- Circular imports (will fail import-linter)
- Business logic in routes (belongs in service)
- Hardcoded values (use config or constants)
- Missing error handling (catch and log or raise AppError subclasses)
When this is documented and enforced, new team members know exactly where to put code and why reviews are faster.
Common mistakes
- Splitting domains too finely: A package-per-endpoint creates more overhead than benefit. Group related endpoints into one domain.
- Bloated shared folder: If shared has 50 files, you're putting domain logic there. Move it back to where it belongs.
- Skipping import-linter: Manual code review for imports is tedious and inconsistent. Automate it.
- Ignoring the CONTRIBUTING guide: If it's not written down, it's not a rule. Every team member has a different mental model until you codify it.
- Over-engineering layers: You don't need a DTOMapper layer, a QueryBuilder layer, and a Transformer layer. Service, repository, schema. Done.
Key takeaways
- Organize by business feature (users, billing, catalog), not by technical layer.
- Each feature owns its models, schemas, service, repository, and routes.
- Keep shared code small and truly cross-cutting; domain-specific logic stays in its domain.
- Use import-linter in CI/CD to prevent cross-domain imports and enforce boundaries automatically.
- Document your structure conventions in CONTRIBUTING.md and refer to it during code review.
- This approach scales: we've seen it work well for teams shipping 10,000+ lines per sprint without constant refactoring.
Comments
Loading comments…