Building a reverse proxy in Rust: lessons from production

Building a reverse proxy in Rust: lessons from production

A reverse proxy sits between your clients and backend services, forwarding requests, managing connections, and handling failures gracefully. Building a reverse proxy in Rust: lessons from production shows us that the language's memory safety and async ecosystem make it an excellent choice for this workload, but there are critical decisions around framework selection, connection management, and error handling that separate a prototype from something you'd trust with production traffic.

We've built and operated reverse proxies across multiple SaaS platforms and e-commerce systems. This article walks through the technical choices you'll face, the pitfalls we've hit, and practical patterns that work at scale.

Choosing your HTTP foundation with hyper

Most Rust reverse proxies start with hyper, a low-level HTTP client and server library built on Tokio. Hyper gives you fine-grained control over connections, request/response handling, and streaming, which is essential for a proxy that must efficiently forward large payloads without buffering everything in memory.

At its core, hyper lets you build a simple forwarding layer:

use hyper::{Body, Request, Response, Server, StatusCode};
use hyper::service::{make_service_fn, service_fn};
use std::net::SocketAddr;

async fn handle_request(req: Request<Body>) -> Result<Response<Body>, hyper::Error> {
    let uri = req.uri().clone();
    let method = req.method().clone();
    let headers = req.headers().clone();
    let body = req.into_body();

    // Forward the request to an upstream
    let upstream_uri = format!("http://backend-server{}", uri);
    let mut upstream_req = Request::builder()
        .method(method)
        .uri(upstream_uri)
        .body(body)?;

    // Copy headers (filter as needed)
    *upstream_req.headers_mut() = headers;

    let client = hyper::Client::new();
    match client.request(upstream_req).await {
        Ok(resp) => Ok(resp),
        Err(_) => Ok(Response::builder()
            .status(StatusCode::BAD_GATEWAY)
            .body(Body::from("Upstream error"))
            .unwrap()),
    }
}

#[tokio::main]
async fn main() {
    let addr = SocketAddr::from(([127, 0, 0, 1], 3000));
    let make_svc = make_service_fn(|_conn| async {
        Ok::<_, hyper::Error>(service_fn(handle_request))
    });

    let server = Server::bind(&addr).serve(make_svc);
    println!("Reverse proxy listening on {}", addr);
    if let Err(e) = server.await {
        eprintln!("Server error: {}", e);
    }
}

This skeleton works, but production reverse proxies need far more: request routing, header manipulation, timeouts, retries, and observability. Hyper handles HTTP mechanics well, but you'll build a lot of orchestration logic on top.

Learning from Pingora and similar frameworks

Cloudflare open-sourced Pingora, a reverse proxy framework written in Rust, and studying its architecture reveals important patterns. Pingora separates concerns into phases: connection, request parsing, request filter, upstream selection, upstream connection, response filter, and logging. This phase-based design lets you plug business logic at each stage without tangling it with HTTP handling.

You don't need to adopt Pingora wholesale, but the design philosophy matters: define clear hooks where you can inject retry logic, circuit breaking, rate limiting, and request transformation without modifying core forwarding code. Here's a simplified phase-based approach:

pub trait ProxyPhase: Send + Sync {
    async fn on_request(&self, req: &mut Request<Body>) -> Result<(), ProxyError>;
    async fn on_upstream_selection(&self, req: &Request<Body>) -> Result<UpstreamAddr, ProxyError>;
    async fn on_response(&self, resp: &mut Response<Body>) -> Result<(), ProxyError>;
}

pub struct ProxyHandler {
    phase: Arc<dyn ProxyPhase>,
    upstream_pool: Arc<UpstreamPool>,
}

impl ProxyHandler {
    pub async fn handle(&self, mut req: Request<Body>) -> Result<Response<Body>, ProxyError> {
        // Phase 1: Request filtering
        self.phase.on_request(&mut req).await?;

        // Phase 2: Upstream selection
        let upstream = self.phase.on_upstream_selection(&req).await?;

        // Phase 3: Forward
        let mut resp = self.upstream_pool.forward(upstream, req).await?;

        // Phase 4: Response filtering
        self.phase.on_response(&mut resp).await?;

        Ok(resp)
    }
}

This abstraction lets you swap implementations, test phases independently, and reuse logic across multiple proxy instances.

Async I/O with Tokio: avoiding blocking in forwarding

Tokio is the async runtime Rust developers reach for, and for reverse proxies, it's non-negotiable. A proxy must handle thousands of concurrent connections; traditional thread-per-request models collapse under that load. Tokio's work-stealing scheduler and lightweight tasks make this feasible.

The critical discipline is: never block a Tokio task. Reading a file from disk, performing CPU-intensive work, or calling a synchronous library function will stall the entire executor. In a proxy, blocking means a single slow upstream response or a blocked DNS lookup can degrade latency for every client.

Here's where mistakes happen. If you need to look up an upstream address dynamically, use async DNS resolution:

use tokio::net::lookup_host;

pub async fn resolve_upstream(host: &str, port: u16) -> Result<SocketAddr, ProxyError> {
    let query = format!("{}:{}", host, port);
    lookup_host(&query)
        .await
        .map_err(|e| ProxyError::Dns(e.to_string()))?
        .next()
        .ok_or_else(|| ProxyError::NoAddresses)
}

Similarly, timeouts must be enforced with tokio::time::timeout, not by hope:

use tokio::time::{timeout, Duration};

pub async fn forward_with_timeout(
    upstream: SocketAddr,
    req: Request<Body>,
) -> Result<Response<Body>, ProxyError> {
    let client = hyper::Client::new();
    timeout(Duration::from_secs(30), client.request(req))
        .await
        .map_err(|_| ProxyError::Timeout)?
        .map_err(ProxyError::Hyper)
}

Tokio also forces you to think about cancellation. If a client disconnects, you want to cancel the upstream request immediately. Tokio's select! macro handles this elegantly:

use tokio::select;

async fn bidirectional_forward(
    mut client_rx: Body,
    mut upstream_tx: &mut Body,
) -> Result<(), ProxyError> {
    loop {
        select! {
            chunk = client_rx.data() => {
                match chunk {
                    Some(Ok(data)) => upstream_tx.write_all(&data).await?,
                    Some(Err(e)) => return Err(ProxyError::ClientError(e)),
                    None => break,
                }
            }
            _ = client_rx.trailers() => {
                // Client closed
                break;
            }
        }
    }
    Ok(())
}

Connection pooling to upstreams: reusing TCP

Opening a new TCP connection for every request is wasteful. Establishing a connection incurs a three-way handshake, TLS negotiation if HTTPS, and increased latency. Production reverse proxies maintain pools of connections to each upstream, reusing them across requests.

Hyper's HttpConnector includes a built-in connection pool, but for more control, you can manage pools explicitly:

use std::sync::Arc;
use tokio::sync::Semaphore;
use std::collections::HashMap;

pub struct UpstreamPool {
    pools: Arc<tokio::sync::Mutex<HashMap<SocketAddr, Vec<PooledConnection>>>>,
    max_connections: usize,
}

pub struct PooledConnection {
    stream: tokio::net::TcpStream,
    idle_since: std::time::Instant,
}

impl UpstreamPool {
    pub async fn acquire(&self, upstream: SocketAddr) -> Result<PooledConnection, ProxyError> {
        let mut pools = self.pools.lock().await;

        if let Some(pool) = pools.get_mut(&upstream) {
            if let Some(conn) = pool.pop() {
                // Check if connection is stale (e.g., idle > 30 seconds)
                if conn.idle_since.elapsed().as_secs() < 30 {
                    return Ok(conn);
                }
            }
        }

        // No pooled connection; create new one
        let stream = tokio::time::timeout(
            std::time::Duration::from_secs(5),
            tokio::net::TcpStream::connect(upstream),
        )
        .await
        .map_err(|_| ProxyError::ConnectTimeout)?
        .map_err(|e| ProxyError::ConnectError(e))?;

        Ok(PooledConnection {
            stream,
            idle_since: std::time::Instant::now(),
        })
    }

    pub async fn release(&self, upstream: SocketAddr, conn: PooledConnection) {
        let mut pools = self.pools.lock().await;
        pools.entry(upstream)
            .or_insert_with(Vec::new)
            .push(conn);
    }
}

In practice, you'll also need to detect broken connections. A connection that's been idle in the pool might have been closed by the upstream server. Attempting to use it causes an error; detecting this early and removing stale connections keeps your pool healthy:

pub async fn forward_on_pooled(
    pool: &UpstreamPool,
    upstream: SocketAddr,
    req: Request<Body>,
) -> Result<Response<Body>, ProxyError> {
    let conn = pool.acquire(upstream).await?;

    // Use connection; if it fails, don't return it to the pool
    match send_on_conn(conn.stream, req).await {
        Ok(resp) => {
            pool.release(upstream, conn).await;
            Ok(resp)
        }
        Err(e) => {
            // Connection is broken; discard it
            Err(ProxyError::UpstreamError(e))
        }
    }
}

Common mistakes

• Unbounded concurrency: Accepting connections without limiting concurrent tasks means a sudden traffic spike exhausts memory. Use a semaphore or accept limit to cap concurrent connections.
• Silent timeouts: Forwarding a request without a timeout can hang indefinitely if the upstream is unresponsive. Always wrap async operations in tokio::time::timeout.
• Header leakage: Blindly forwarding all headers can expose internal state (e.g., X-Internal-Server-ID). Maintain an allowlist of headers to forward and strip sensitive ones.
• Not handling streaming bodies: Buffering large request or response bodies defeats the purpose of a proxy. Use streaming APIs and avoid .to_bytes() or similar unless absolutely necessary.
• Pool exhaustion: If upstream connections are held too long or not released on error, the pool empties and new requests fail. Test pool behavior under failure scenarios.
• No observability: Without logs, metrics, and traces, debugging production issues is nearly impossible. Instrument request routing, upstream selection, errors, and latencies from day one.

Key takeaways

• Hyper is the foundation for HTTP handling; layer business logic on top using a phase-based design inspired by frameworks like Pingora.
• Tokio's async runtime is non-negotiable for concurrency, but requires discipline: never block tasks, always enforce timeouts, and gracefully handle cancellation.
• Connection pooling to upstreams is essential for latency; manage pool health by detecting and discarding stale connections.
• Production reverse proxies must handle streaming bodies, limit concurrency, validate upstream connectivity, and provide rich observability.
• Test failure modes explicitly: upstream timeouts, connection pool exhaustion, upstream crashes, and rapid reconnects.