gRPC with Python: Python Backend lessons from production

Introduction

When you're coordinating dozens of internal services across your infrastructure, HTTP REST calls start to feel sluggish. Protocol Buffers and gRPC offer a different path: strongly typed, binary serialization, and HTTP/2 multiplexing all working together. gRPC with Python: Python Backend lessons from production covers what we've learned running gRPC in real systems—where deadlines matter, where network latency compounds, and where a poorly defined service contract can break your entire request flow.

In this article, we walk through building gRPC services with Python from the ground up, then focus on the production practices that keep them reliable.

Define services in .proto files

Every gRPC service starts with a .proto file. This is your contract—it defines the messages your service sends and receives, plus the RPC methods that tie them together. Unlike REST endpoints that live scattered across decorators and docstrings, your gRPC API lives in one place, version-controlled, and language-agnostic.

Let's build a real example: an order service that other internal services query for order details and updates.

syntax = "proto3";

package order.v1;

service OrderService {
  rpc GetOrder(GetOrderRequest) returns (OrderReply) {};
  rpc ListOrders(ListOrdersRequest) returns (stream OrderReply) {};
  rpc CreateOrder(CreateOrderRequest) returns (OrderReply) {};
}

message GetOrderRequest {
  string order_id = 1;
}

message ListOrdersRequest {
  string user_id = 1;
  int32 limit = 2;
  int32 offset = 3;
}

message OrderReply {
  string order_id = 1;
  string user_id = 2;
  float total = 3;
  string status = 4;
  int64 created_at = 5;
}

message CreateOrderRequest {
  string user_id = 1;
  repeated OrderItem items = 2;
}

message OrderItem {
  string product_id = 1;
  int32 quantity = 2;
  float price = 3;
}

A few things to notice: every field has a number (used for backward compatibility), we use repeated for lists, and we define nested messages for structure. The stream keyword on ListOrders means the server sends back multiple OrderReply messages—useful for pagination without round-trips.

In production, your team owns these contracts as much as they own database schemas. Changes need review. Breaking changes need versioning (see the v1 package). This discipline prevents the chaos of REST endpoints that change behavior without warning.

grpcio-tools generates stubs

Once you have your .proto file, the grpcio-tools compiler generates Python code you actually use. This isn't manual work; it's a build step.

Install the tools:

pip install grpcio grpcio-tools

Then compile your proto:

python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. order.proto

This generates two files: order_pb2.py (message definitions) and order_pb2_grpc.py (service stubs). You never edit these—they're generated, versioned, and regenerated when your proto changes.

On the server side, you implement the service:

import grpc
from concurrent import futures
import order_pb2
import order_pb2_grpc

class OrderServicer(order_pb2_grpc.OrderServiceServicer):
    def __init__(self, db):
        self.db = db

    def GetOrder(self, request, context):
        order = self.db.get_order(request.order_id)
        if not order:
            context.set_code(grpc.StatusCode.NOT_FOUND)
            context.set_details("Order not found")
            return order_pb2.OrderReply()
        
        return order_pb2.OrderReply(
            order_id=order['id'],
            user_id=order['user_id'],
            total=order['total'],
            status=order['status'],
            created_at=order['created_at']
        )

    def ListOrders(self, request, context):
        orders = self.db.list_orders(
            user_id=request.user_id,
            limit=request.limit,
            offset=request.offset
        )
        for order in orders:
            yield order_pb2.OrderReply(
                order_id=order['id'],
                user_id=order['user_id'],
                total=order['total'],
                status=order['status'],
                created_at=order['created_at']
            )

def serve():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    servicer = OrderServicer(db=MyDatabase())
    order_pb2_grpc.add_OrderServiceServicer_to_server(servicer, server)
    server.add_insecure_port('[::]:50051')
    server.start()
    server.wait_for_termination()

if __name__ == '__main__':
    serve()

On the client side, calling this service is straightforward:

import grpc
import order_pb2
import order_pb2_grpc

channel = grpc.aio.secure_channel('order-service:50051', grpc.ssl_channel_credentials())
stub = order_pb2_grpc.OrderServiceStub(channel)

request = order_pb2.GetOrderRequest(order_id='12345')
response = stub.GetOrder(request)
print(f"Order: {response.order_id}, Status: {response.status}")

The generated stubs handle serialization, deserialization, and the HTTP/2 transport. You never touch those details—you just call methods on the stub and get back typed message objects.

Use deadlines on every call

This is where production gRPC separates from hobby projects. In a microservices environment, if one service hangs, it can cascade and bring down your entire system. Deadlines are your circuit breaker.

Every gRPC call should have a deadline—a hard timeout. If the call doesn't complete by then, gRPC cancels it and returns a DEADLINE_EXCEEDED error. You set deadlines on the client side:

import grpc
import time

channel = grpc.aio.secure_channel('order-service:50051', grpc.ssl_channel_credentials())
stub = order_pb2_grpc.OrderServiceStub(channel)

# Set deadline to 500ms from now
deadline = time.time() + 0.5

try:
    request = order_pb2.GetOrderRequest(order_id='12345')
    response = stub.GetOrder(request, timeout=deadline)
except grpc.RpcError as e:
    if e.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
        print("Order service timed out—circuit breaker triggered")
        # Handle gracefully: return cached data, default response, etc.
    else:
        print(f"Other error: {e.code()}")

In async code, which we recommend for I/O-heavy services:

import grpc
import asyncio
import time

async def get_order_with_timeout(stub, order_id):
    request = order_pb2.GetOrderRequest(order_id=order_id)
    try:
        # Create a deadline 500ms in the future
        deadline = time.time() + 0.5
        response = await stub.GetOrder(request, timeout=deadline)
        return response
    except grpc.RpcError as e:
        if e.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
            print(f"Timeout fetching order {order_id}")
            return None
        raise

We've seen services where missing deadlines caused thread pool exhaustion—requests piled up waiting for a slow downstream service, and soon the entire server hung. Deadlines prevent this. Set them conservatively; use monitoring to find the right values for your network topology.

Great for internal low-latency calls

gRPC shines where HTTP REST struggles: internal service-to-service communication. The binary format (Protocol Buffers), HTTP/2 multiplexing, and connection pooling all reduce latency and bandwidth.

Consider a typical payment flow: your API gateway calls the order service, which calls the payment service, which calls the ledger service. With REST and JSON, each hop serializes and deserializes. With gRPC, each message is compact binary, and HTTP/2 lets all those requests share one connection.

In production, we've measured 10-50ms latency improvements per call compared to REST on the same network. For user-facing requests that chain 3-5 internal calls, that compounds. Multiply it across millions of requests, and you're looking at real cost savings in compute and better p99 latencies.

One more pattern: use gRPC for internal APIs, REST for external APIs. Your public API still benefits from REST's simplicity and browser friendliness. But everything internal—worker services, analytics pipelines, event processors—runs gRPC.

Common mistakes

• Forgetting to handle gRPC errors properly. gRPC errors aren't HTTP status codes. Always catch grpc.RpcError and check the status code. Treat UNAVAILABLE and DEADLINE_EXCEEDED as retryable; others as permanent failures.
• Not setting deadlines. This leads to cascading timeouts. Every call needs a timeout, period.
• Mutating proto messages after sending them. Once you pass a message to a gRPC call, don't modify it. The serialization might still be in flight. Create new message objects instead.
• Ignoring backward compatibility in proto evolution. Never reuse field numbers. Never change field types. Use optional fields for new columns. Proto versioning looks easy until it breaks production.
• Using synchronous stubs in async code. If your service is async (which it should be), use async stubs. Mixing them causes thread pool deadlocks.

Key takeaways

• .proto files define your service contract. Treat them as seriously as database schemas.
• grpcio-tools generates type-safe stubs from your proto. Never write them by hand; always regenerate from the canonical definition.
• Set deadlines on every gRPC call. Without them, slow services cascade failures across your infrastructure.
• gRPC is optimized for internal, low-latency service-to-service communication. Binary serialization and HTTP/2 multiplexing reduce latency and bandwidth compared to REST.
• Use async stubs and connection pooling to maximize throughput. Reuse channels across requests rather than creating new ones.
• Handle errors by status code, not exception type. gRPC separates transient failures from permanent ones for you.