A tunnel platform taught me networking, explained simply

A tunnel platform taught me networking in ways no course ever could. I spent weeks building an ngrok alternative to solve internal development velocity—and ended up learning more about infrastructure, scaling, and production realities than I had in my previous five years of CRUD work combined.

Why I Built a Tunnel Platform

Our team faced a recurring problem: local webhook testing against third-party APIs required exposing localhost to the internet. Engineers were either using ngrok's free tier with its session limits and unstable subdomains, or paying enterprise rates. I thought building an internal tunnel service would be a quick weekend project. It wasn't. But that friction turned into education.

The architecture seemed simple at first: accept inbound requests from the internet, route them through a persistent WebSocket connection to a client running on an engineer's machine, and tunnel the traffic back. Reality introduced me to reverse proxies, certificate management, connection pooling, and infrastructure patterns I'd only read about.

Reverse Proxies and Wildcard SSL in Production

I started naive. I thought I could just accept incoming HTTP requests, deserialize them into JSON, send them over a WebSocket, and deserialize them back on the client. That works for a proof of concept. Production demands more.

The first wall I hit: SSL/TLS termination. When an external service like Stripe or GitHub tries to send a webhook to https://myapp-dev.tunnel.internal, it expects valid TLS. I needed wildcard certificates because the subdomain part varies per engineer and per service.

I used Let's Encrypt with DNS validation (not HTTP—too fragile for dynamic subdomains). The challenge: renewal automation. I built a renewal loop that ran every six hours, checked certificate expiry, and renewed anything expiring within 30 days. This lives in a background worker, not on the reverse proxy itself. Coupling certificate renewal to your HTTP handler is how you get cascading failures during renewal windows.

interface TunnelCertificate {
  domain: string;
  certificatePem: string;
  keyPem: string;
  expiresAt: Date;
  renewalAttempts: number;
}

async function renewExpiredCertificates(
  certsRepository: CertificatesRepository,
  letsEncryptClient: LetsEncryptClient
): Promise<void> {
  const expiringSoon = await certsRepository.findExpiringBefore(
    addDays(new Date(), 30)
  );

  for (const cert of expiringSoon) {
    try {
      const renewed = await letsEncryptClient.renew(cert.domain);
      await certsRepository.update(cert.id, {
        certificatePem: renewed.cert,
        keyPem: renewed.key,
        expiresAt: renewed.expiresAt,
        renewalAttempts: 0,
      });
    } catch (error) {
      await certsRepository.incrementRenewalAttempts(cert.id);
      // Alert if too many failures
      if (cert.renewalAttempts > 3) {
        await alerting.notify('Certificate renewal failing', { domain: cert.domain });
      }
    }
  }
}

The reverse proxy itself—I used Nginx initially, then moved to Node.js with Express for tighter integration with my tunnel registry. The proxy needs to:

• Match incoming Host header to a registered tunnel
• Look up the active WebSocket connection for that tunnel
• Forward the request over the tunnel
• Wait for response (with timeout)
• Return response to the client

Long response times became a problem. If an engineer's machine was slow or the WebSocket buffered, the proxy would hold connections open, eating file descriptors. I learned to set aggressive timeouts and implement a circuit breaker: if a tunnel fails to respond within five seconds three times in a row, close the tunnel and notify the client. This prevents cascading hangs.

Multi-Region Routing is Mostly Edge Cases

I initially planned to support multiple regions: engineers in Colombo connecting to a tunnel endpoint in Singapore, while webhooks come from US-based services. The latency trade-offs seemed elegant: local tunnel client connects to nearest region, webhooks routed intelligently.

In practice, I spent 60% of my time on edge cases and 15% of it actually helped users:

• Failover complexity: If the Singapore endpoint goes down, does the tunnel automatically reconnect to Mumbai? What about in-flight requests? I built a reconnection strategy that was overkill for our team size.
• Sticky routing: A webhook comes in from the US, but the tunnel client is in Colombo. Which region handles the routing? I had to implement request affinity and learned that "nearest" is not always "best"—sometimes stable is better than fast.
• Certificate management across regions: Now I'm renewing certificates in three regions. State consistency became a problem. I centralized certificate storage in PostgreSQL with replication.

The lesson: I over-engineered for a problem I didn't have. Our team works across two time zones. A single region (AWS in Singapore) handles 99% of our load with acceptable latency. Multi-region is a premature optimization I'd cut in week two if I built it again.

Long-Lived Connections Change How You Scale

Here's where I learned something genuinely valuable: traditional HTTP services scale horizontally by load-balancing stateless requests. My tunnel platform violates both assumptions.

Each WebSocket connection represents a registered tunnel—a stateful relationship between a client machine and the proxy. If I spin up a second proxy instance, incoming webhooks might hit instance A, but the tunnel connection lives on instance B. Now what?

I explored three approaches:

1. Sticky load balancing: Hash the tunnel ID to a single proxy instance. Works until that instance crashes and you lose all tunnels.
2. Distributed tunnel registry: Use Redis to track which instance owns which tunnel. On request, look up the owner, connect to that instance, forward the request. This works but adds latency and Redis dependency.
3. Connection synchronization: Every proxy instance maintains every tunnel connection. Wastes bandwidth but eliminates lookups and single points of failure.

I chose option two because I already had Redis for caching. But I under-estimated the operational cost: Redis failovers mean tunnel lookups fail. I added a fallback: if the registry misses, broadcast to all instances. It's not elegant, but it's resilient.

The real learning: long-lived connections force you to think about connection ownership. You can't treat them like HTTP requests. Sessions, sockets, subscriptions—they all have the same constraint. This fundamentally changes your scaling strategy.

Side Projects Teach What Tutorials Can't

Courses teach you how nginx works. Building a tunnel platform teaches you what it's like when nginx fails and 30 engineers can't test their integrations.

Real-world lessons I'd never encounter in a tutorial:

• Connection leaks: A bug in my tunneling code left WebSocket connections open even after the client disconnected. Memory grew linearly. I spent a day with memory profilers learning to spot this.
• Chaos engineering: I discovered my failover logic only worked if I simulated the exact failure conditions. Random crashes are different from graceful shutdowns.
• Observability debt: Six weeks in, I couldn't answer "why are tunnel X requests slow?" because I hadn't instrumented request timing. I retrofitted logging and it took another week.
• The human side: A feature I thought was obvious (automatic reconnection) confused users. I learned that infrastructure is only good if people understand it.

None of this appears in a networking fundamentals course. You only learn it by shipping something, having it break in production (internal production counts), and fixing it under real constraints.

Common Mistakes

• Treating tunnels as stateless: I initially tried to distribute tunnel load across instances without accounting for connection affinity. Tunnels are stateful by definition.
• Ignoring certificate lifecycle: I automated renewal but didn't monitor renewal success rates. Three weeks in, a renewal failed silently and the service went down during expiry.
• Over-engineering for scale: Multi-region support was premature optimization. Start single-region and move to multi-region only when you have concrete latency problems and the team size to support it.
• Assuming WebSocket libraries handle disconnects gracefully: They don't. You need explicit heartbeats, reconnection logic, and timeout handlers.
• Skipping observability: Build logging and metrics from day one. Retrofitting them is miserable.

Key Takeaways

• Reverse proxies need careful certificate management, timeouts, and circuit breakers to stay reliable in production.
• Wildcard SSL and DNS validation are practical for dynamic subdomains, but automation requires robust retry and monitoring logic.
• Multi-region routing is valuable only when you have latency problems and the team to support it. Start simple.
• Long-lived connections are fundamentally different from stateless HTTP. They require connection ownership, affinity awareness, and distributed registry patterns.
• Building infrastructure teaches lessons no course can: failure modes, operational complexity, and the real cost of features.
• Start with ruthless simplicity (single region, sticky load balancing, basic logging). Complexity should solve real problems, not anticipated ones.