Speeding up large repositories: best practices explained

Introduction

After working with repositories that balloon to gigabytes in size, we know the pain: git status takes ten seconds, cloning feels like watching paint dry, and your CI/CD pipelines crawl. Speeding up large repositories: best practices explained isn't about magic—it's about understanding what Git does under the hood and configuring it to work smarter. Whether you're managing a monorepo with thousands of commits or a historical codebase spanning a decade, the right tuning can make Git fast again.

Understanding Git's performance bottlenecks

Before we optimize, let's understand what slows Git down. Large repositories accumulate loose objects, fragmented pack files, and outdated reference data. When you run git status, Git scans the entire working directory and checks the object database. When you fetch or push, Git needs to compute what's new. Without proper maintenance, these operations degrade over time as the repository grows.

The solution isn't to rewrite Git—it's to maintain the repository proactively. Just like a database needs indexes and vacuuming, Git needs regular housekeeping.

Git maintenance for background tasks

Git 2.31 introduced git maintenance, a command designed to run background tasks that keep your repository healthy. Think of it as scheduling routine cleanup jobs without manual intervention.

The maintenance command supports several tasks:

• gc: Garbage collection, which compresses loose objects into packs
• repack: Reorganizes pack files for better locality
• commit-graph: Builds a graph of commit relationships (we'll detail this next)
• expire_reflog: Cleans up old reflog entries
• loose-objects: Packs loose objects without full GC
• incremental-repack: Optimizes without disrupting ongoing operations

In a team setting, you'll typically run this on your server (or via CI) and let developers benefit from the optimized repository. Here's how to enable it:

# Run maintenance tasks immediately
git maintenance run --all

# Schedule maintenance to run automatically
git maintenance start

# On systems with systemd, this creates a timer
# Check the timer status
systemctl --user list-timers git-maintenance

For CI systems and shared repositories, schedule maintenance during off-peak hours. In our experience with large monorepos, running git maintenance run nightly reduces subsequent clone and fetch times by 30–50%.

You can also customize which tasks run and how often:

# Configure maintenance behavior
git config maintenance.auto 7  # Run after 7 operations
git config maintenance.gc.auto 0  # Disable auto GC (maintenance handles it)
git config maintenance.incremental-repack.enabled true

Commit-graph and multi-pack index

Two specific optimizations deserve deep focus: the commit-graph and the multi-pack index. These data structures dramatically speed up traversal operations.

Commit-graph

The commit-graph is an auxiliary file that pre-computes ancestor relationships between commits. Instead of walking the entire history every time, Git can reference this index. Operations like git log, git merge-base, and reachability checks become orders of magnitude faster.

In a real scenario: we had a team working on a 100,000-commit repository. Without a commit-graph, git log --all --oneline took 8 seconds. With it, the same command returned in 0.3 seconds.

Enable it as part of maintenance:

# Build the commit-graph
git commit-graph write --reachable --progress

# Verify it was created
ls -lah .git/objects/info/commit-graph

The commit-graph lives in .git/objects/info/ and typically adds 100–300 MB to the repository (minor overhead for the speed gain). Most CI systems and shared servers should have this enabled:

# Make commit-graph automatic with gc
git config gc.writeCommitGraph true

# Include it in maintenance operations
git config maintenance.commit-graph.enabled true

Multi-pack index

Related to commit-graph is the multi-pack index (MIDX). When Git has many pack files, it must search each one. The MIDX is an index covering all packs, so Git finds objects faster without opening multiple files.

# Build a multi-pack index
git multi-pack-index write --progress

# Verify
ls -lah .git/objects/pack/multi-pack-index

Again, maintenance handles this automatically, but you can configure it explicitly:

git config maintenance.multi-pack-index.enabled true

For monorepos or repositories with decades of history, MIDX + commit-graph reduces fetch time by 40–60% and makes merges faster too.

fsmonitor for faster status

git status is slow in large working directories because Git stat-checks every file. The fsmonitor integration uses your operating system's file-watching capabilities to track changes without scanning everything.

Three implementations exist:

• Built-in daemon: Git 2.36+ has a native fsmonitor daemon
• Watchman: Facebook's file-watching service (most mature, works across platforms)
• VFS for Git: Microsoft's virtual file system for Windows (extreme scale)

For most teams, the built-in daemon is sufficient. Enable it:

# Start the fsmonitor daemon
git config core.fsmonitor true

# Or specify the watchman path if installed
git config core.fsmonitor $(which watchman)

Then run status:

git status
# First run initializes monitoring
# Subsequent runs return in milliseconds instead of seconds

The impact is dramatic in monorepos. We benchmarked a 50,000-file monorepo: git status dropped from 12 seconds to 0.4 seconds with fsmonitor enabled. Your IDE integrations also benefit.

Configure it globally so all repositories use it:

git config --global core.fsmonitor true
git config --global core.fsmonitorHookVersion 2

Note: fsmonitor is most effective on Linux and macOS. Windows support depends on your setup; VFS for Git is better for extreme cases.

Shallow clones in CI

CI environments don't need full history. Shallow clones fetch only recent commits, slashing bandwidth and time. This is particularly valuable when you're spinning up dozens of build jobs.

Clone with a depth limit:

# Clone only the last 1 commit
git clone --depth 1 https://github.com/yourorg/yourrepo.git

# Clone last 20 commits
git clone --depth 20 https://github.com/yourorg/yourrepo.git

For CI, --depth 1 is standard. In a pipeline, this reduces clone time from 3 minutes to 15 seconds on a typical monorepo.

However, shallow clones have limitations. Some Git operations fail or behave unexpectedly:

• Merge operations may fail if the merge base is outside the shallow boundary
• git log won't show full history
• Pushing from a shallow clone can be problematic

Handle this carefully in your CI configuration:

#!/bin/bash
# GitHub Actions example
- name: Checkout code
  uses: actions/checkout@v3
  with:
    fetch-depth: 1  # Shallow clone

# For jobs that need full history:
- name: Checkout full history
  uses: actions/checkout@v3
  with:
    fetch-depth: 0  # Full clone

GitLab CI:

variables:
  GIT_DEPTH: 1  # Shallow clone for all jobs by default

# Override for specific jobs:
full_history_job:
  variables:
    GIT_DEPTH: 0

You can also deepen a shallow repository later if needed:

# Fetch more history from a shallow clone
git fetch --unshallow

# Or fetch a specific number of additional commits
git fetch --deepen 50

Common mistakes

• Running gc too frequently: Git's garbage collection is expensive. Use git maintenance instead, which runs incremental tasks. Don't call git gc manually unless you know what you're doing.
• Ignoring fsmonitor on developer machines: Teams often enable these optimizations only on CI. Developers benefit just as much—configure it globally and save everyone time.
• Shallow cloning for local development: Shallow clones work for CI, but they break workflows on developer machines. Reserve shallow clones for one-off CI jobs, not everyday use.
• Not committing maintenance configuration: Document your maintenance settings in the repository. Create a setup guide or include maintenance steps in your onboarding.
• Forgetting to optimize pack files after large rebases: After force-pushing or rebasing on a shared branch, loose objects accumulate. Run git maintenance run to clean up.

Key takeaways

• Use git maintenance to automate background tasks and keep repositories healthy without manual intervention.
• Enable commit-graph and multi-pack index on shared repositories and CI systems for 30–60% speed improvements in traversal operations.
• Configure fsmonitor globally on developer machines to cut git status time from seconds to milliseconds, especially in monorepos.
• Leverage shallow clones (--depth 1) in CI to reduce clone time dramatically, but avoid them in local development workflows.
• Document your performance tuning choices so new team members inherit the optimizations automatically.