gRPC with Go: concepts, trade-offs and production lessons

gRPC has quietly become the backbone of service-to-service communication in production systems, offering binary serialization and HTTP/2 multiplexing where REST would stumble. If you're building microservices that need sub-millisecond latency or handling high-concurrency workloads, gRPC with Go: concepts, trade-offs and production lessons is the mental framework that separates "it works" from "it scales." In this post, we'll walk through what we've learned shipping gRPC-based APIs across e-commerce, SaaS, and trading platforms.

Define the Service — Protocol Buffers Describe Messages and RPCs

Before we write a single line of Go, we define our contract in Protocol Buffers (protobuf). This is non-negotiable in production: the schema is your source of truth, versioning is built in, and you get multi-language code generation for free.

Let's say we're building a product catalog service. Here's a minimal proto definition:

syntax = "proto3";

package catalog.v1;

option go_package = "github.com/myteam/catalog/v1";

message Product {
  string id = 1;
  string name = 2;
  double price = 3;
  int32 stock = 4;
}

message GetProductRequest {
  string product_id = 1;
}

message GetProductResponse {
  Product product = 1;
}

service CatalogService {
  rpc GetProduct(GetProductRequest) returns (GetProductResponse);
  rpc ListProducts(Empty) returns (stream Product);
}

message Empty {}

Each field gets a unique number (1, 2, 3…) because protobuf encodes them numerically. This is how backward compatibility works: add a new field with a higher number, old clients ignore it silently. Remove a field? Reserve its number so nobody reuses it accidentally. In real systems, we've seen teams accidentally reuse field numbers and corrupt production data; reserve them religiously.

Notice stream Product in the ListProducts RPC. This is one of gRPC's superpowers: server-push and bidirectional streaming, all multiplexed over a single HTTP/2 connection. No polling, no long-lived connections with heartbeat complexity. Clean.

Generate Code — protoc with Go Plugins or buf

Raw protoc works, but we recommend buf for anything team-sized. It enforces linting rules, manages dependencies, and generates code consistently.

With protoc directly:

protoc \
  --go_out=. \
  --go-grpc_out=. \
  catalog/v1/catalog.proto

This generates two files: catalog_pb.go (message types) and catalog_grpc.pb.go (service stubs). The GRPC file contains an interface you'll implement on the server side and a client stub for calling the service.

With buf, you define a buf.yaml:

version: v1
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT
    - COMMENTS

Then run:

buf generate

It's cleaner for monorepos and multi-team setups because it handles vendoring and version pinning. We've used buf in e-commerce platforms where frontend teams, backend teams, and mobile teams all depend on the same proto definitions; the single source of truth saved us from protocol desync bugs.

Key decision: store generated code in version control or regenerate on every build? We store it. Regenerating adds latency to builds and makes diffs noisier; checking it in means the generated interface is stable for code review.

Implement the Server — Embed the Unimplemented Server and Add Methods

The generated catalog_grpc.pb.go file contains an interface and an "unimplemented" server struct. The trick is to embed the unimplemented server, then add your methods. This pattern gives you forward compatibility: if the service definition adds a new RPC, you get a compile error reminding you to implement it (if you don't embed unimplemented, old code silently ignores new RPCs).

package main

import (
  "context"
  "github.com/myteam/catalog/v1"
  "google.golang.org/grpc"
  "google.golang.org/grpc/codes"
  "google.golang.org/grpc/status"
  "net"
)

type CatalogServer struct {
  catalog.UnimplementedCatalogServiceServer
  db ProductDB
}

func (s *CatalogServer) GetProduct(ctx context.Context, req *catalog.GetProductRequest) (*catalog.GetProductResponse, error) {
  if req.ProductId == "" {
    return nil, status.Error(codes.InvalidArgument, "product_id is required")
  }

  product, err := s.db.GetByID(ctx, req.ProductId)
  if err != nil {
    if err == ErrNotFound {
      return nil, status.Error(codes.NotFound, "product not found")
    }
    return nil, status.Error(codes.Internal, "database error")
  }

  return &catalog.GetProductResponse{
    Product: &catalog.Product{
      Id:    product.ID,
      Name:  product.Name,
      Price: product.Price,
      Stock: int32(product.Stock),
    },
  }, nil
}

func (s *CatalogServer) ListProducts(req *catalog.Empty, stream catalog.CatalogService_ListProductsServer) error {
  products, err := s.db.ListAll(stream.Context())
  if err != nil {
    return status.Error(codes.Internal, "failed to fetch products")
  }

  for _, p := range products {
    if err := stream.Send(&catalog.Product{
      Id:    p.ID,
      Name:  p.Name,
      Price: p.Price,
      Stock: int32(p.Stock),
    }); err != nil {
      return err
    }
  }

  return nil
}

func main() {
  listener, _ := net.Listen("tcp", ":50051")
  grpcServer := grpc.NewServer()
  
  catalog.RegisterCatalogServiceServer(grpcServer, &CatalogServer{
    db: NewDB(),
  })

  grpcServer.Serve(listener)
}

Always use status.Error with gRPC error codes (NotFound, InvalidArgument, Internal, etc.). Clients rely on these codes to decide whether to retry or fail fast. Return a 503 Internal equivalent and let your infrastructure retry; return 400 InvalidArgument and the client should give up immediately.

Deadlines and Interceptors — Set Deadlines on Every Call; Use Interceptors for Auth and Logging

This is where production-grade gRPC separates from toy implementations.

Deadlines first. Every gRPC call should have a deadline. On the client side:

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

product, err := client.GetProduct(ctx, &catalog.GetProductRequest{ProductId: "123"})
if err != nil {
  if status.Code(err) == codes.DeadlineExceeded {
    // Handle timeout
  }
}

On the server side, you inherit the deadline from the incoming request. If a call arrives with a 5-second deadline and you burn 3 seconds in middleware, your handler only has 2 seconds left. Check it:

func (s *CatalogServer) GetProduct(ctx context.Context, req *catalog.GetProductRequest) (*catalog.GetProductResponse, error) {
  deadline, ok := ctx.Deadline()
  if ok && time.Now().After(deadline) {
    return nil, status.Error(codes.DeadlineExceeded, "context already expired")
  }
  
  // Continue with logic
}

Interceptors are middleware. Use unary interceptors for request-response calls, stream interceptors for streaming:

func loggingInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
  start := time.Now()
  
  resp, err := handler(ctx, req)
  
  duration := time.Since(start)
  if err != nil {
    fmt.Printf("RPC %s failed in %v: %v\n", info.FullMethod, duration, err)
  } else {
    fmt.Printf("RPC %s succeeded in %v\n", info.FullMethod, duration)
  }
  
  return resp, err
}

func authInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
  token := extractTokenFromContext(ctx)
  if !isValid(token) {
    return nil, status.Error(codes.Unauthenticated, "invalid token")
  }
  
  return handler(ctx, req)
}

grpcServer := grpc.NewServer(
  grpc.ChainUnaryInterceptor(authInterceptor, loggingInterceptor),
)
catalog.RegisterCatalogServiceServer(grpcServer, &CatalogServer{db: NewDB()})
grpcServer.Serve(listener)

Interceptors run in order, so auth before logging. In production, we chain five or six: auth, request ID injection, logging, metrics collection, timeout enforcement. Each one is a thin slice of responsibility.

Common Mistakes

• Ignoring error codes. Returning status.Error(codes.Internal, "...") for every failure. Clients can't tell if they should retry. Use NotFound, InvalidArgument, FailedPrecondition, Unavailable appropriately.
• No deadlines on client calls. A hung dependency will hang your entire request. Always set a timeout context.
• Reusing field numbers in proto changes. Corruption waiting to happen. Reserve removed numbers immediately.
• Streaming without backpressure handling. If you send 10,000 messages down a stream without pausing, the client buffer fills and you'll hit resource limits. Batch and monitor.
• Forgetting reflection in development. Turn on gRPC reflection so tools like grpcurl can inspect your service without .proto files.

Key Takeaways

• Protocol Buffers are the contract; proto definitions in version control are non-negotiable for team alignment and versioning.
• Use buf for code generation in multi-team codebases; it handles consistency and linting at scale.
• Embed UnimplementedServiceServer so new RPC additions force you to add implementations; ignore this and you'll ship incomplete services silently.
• Deadlines on every client call are insurance against cascading failures; they're not optional in production.
• Interceptors handle cross-cutting concerns cleanly; chain them intentionally, with auth before logging.
• Use gRPC error codes semantically; clients key off codes to decide retry logic, so codes.Internal and codes.InvalidArgument have different meanings to callers.
• Binary serialization and HTTP/2 multiplexing make gRPC faster than REST for service-to-service traffic, but the real win is the type safety and streaming semantics.