Protocol Buffers basics: real-world lessons and examples

Why Protocol Buffers matter in production

When you're building microservices that talk to each other, or shipping APIs that multiple teams consume, message format becomes a critical choice. Protocol Buffers basics: real-world lessons and examples shows us how to avoid silent failures, coordinate schema changes across teams, and generate type-safe clients in Go, Python, Node, Java—without coordinating a massive rollout. We've learned the hard way that getting this right early saves months of refactoring down the line.

Protocol Buffers (protobuf) give you three things: a human-readable schema language, compact binary serialization, and backward/forward compatibility built in. Unlike JSON, which is flexible but verbose and unversioned, protobuf forces you to be explicit about change management. Over the past 47 days of building production APIs in Go, we've seen teams ship features faster and break fewer services when they use protobuf thoughtfully.

Fields have numbers—never reuse them

Every field in a protobuf message has a number. That number is what actually gets written to the wire; the field name is just for readability in the .proto file. This is the most critical rule to understand, because breaking it creates silent data corruption that can take weeks to debug.

Let's say we define an order message:

syntax = "proto3";

package orders;

message Order {
  int32 id = 1;
  string customer_name = 2;
  double total = 3;
  string email = 4;
}

Six months later, you realize email is a poor field name. You want it to be customer_email. That's safe—just rename it, recompile, done. But what if someone on the team thinks, "We don't use the email field anymore, let me delete it and reuse field number 4 for something new"?

// DO NOT DO THIS
message Order {
  int32 id = 1;
  string customer_name = 2;
  double total = 3;
  // email was field 4, now we're reusing 4 for shipping_address
  string shipping_address = 4;
}

If an old client sends a message with email in field 4, a new service reading it will interpret those bytes as shipping_address. The data corruption is silent—no error, no warning. The old client's email string ends up in the shipping address field, and you've corrupted your database.

Instead, mark the field as reserved or deprecated:

message Order {
  int32 id = 1;
  string customer_name = 2;
  double total = 3;
  reserved 4;
  string shipping_address = 5;
}

Now field 4 is explicitly forbidden, and your protobuf compiler will reject any attempt to use it. The rule is simple: field numbers are permanent. Delete the field, mark it reserved, and move to the next number. In production systems where you can't coordinate a single deployment across all clients and servers, this discipline prevents weeks of debugging.

Add fields safely

One of protobuf's best features is that adding new fields is safe, as long as you follow the rules. New code can ignore fields it doesn't recognize, and old code can ignore fields it doesn't know about.

Imagine we have a service that creates orders, and we want to add a discount_code field:

message Order {
  int32 id = 1;
  string customer_name = 2;
  double total = 3;
  reserved 4;
  string shipping_address = 5;
  string discount_code = 6;
}

Old clients that don't know about discount_code will simply ignore field 6 when they deserialize a message. If the service sending the message includes discount_code, those old clients won't crash—they'll just skip it. Meanwhile, new clients that do support discounts will see the field and use it. This is the contract protobuf gives you: services can roll out at different times without coordination.

The caveats:

• Always provide sensible defaults. If a field is missing, it gets the zero value (0 for numbers, empty string for strings, false for bools).
• Don't change a field's type. If total starts as an int32, don't later change it to a string—that's a wire-format break.
• Required fields (proto2 syntax) are dangerous in production because old readers can't deserialize messages that include them. Proto3 doesn't have required, which is safer but requires more discipline to handle missing fields.

In our experience, the safest approach is to add new optional fields with a new field number and test thoroughly before removing support for old fields in any service.

Use buf for linting and breaking-change checks

Buf is a tool that treats your .proto files like code. It lints them, enforces style rules, and most importantly, it detects breaking changes before they reach your repository.

Set up a buf.yaml in your project root:

version: v1
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT

Now when you add a field and try to commit, run:

buf breaking ./proto --against '.git#branch=main'

Buf will compare your schema against the version on main and flag any breaking changes:

• Field number reuse (exactly what we just warned about).
• Type changes on existing fields.
• Removing enum values that are still in use.
• Removing services or RPC methods.

In a team setting, we add this check to CI. Any PR that tries to introduce a breaking change gets blocked, and the team has to explicitly discuss and agree that the break is necessary. This friction is a feature—it keeps you honest about compatibility.

Buf also lints for style:

buf lint ./proto

It enforces naming conventions (message names in PascalCase, field names in snake_case), file organization, and import patterns. Having a single source of truth for these rules means new team members write proto files that look exactly like old ones, with no bikeshedding.

Generate clients for many languages

One .proto file, one source of truth. Your Go services, Python data pipelines, Node.js frontends, and Java legacy systems all consume the same contract, with no manual translation.

Let's say we have a simple service definition:

syntax = "proto3";

package orders;

option go_package = "github.com/mycompany/orders-api/gen/go/orders";

import "google/protobuf/timestamp.proto";

message CreateOrderRequest {
  string customer_name = 1;
  double total = 2;
  string discount_code = 3;
}

message CreateOrderResponse {
  int32 order_id = 1;
  google.protobuf.Timestamp created_at = 2;
}

service OrderService {
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);
}

With a single buf build command, we generate Go, Python, TypeScript, and more. In Go, we get:

// Generated code
type CreateOrderRequest struct {
  CustomerName string
  Total float64
  DiscountCode string
}

type CreateOrderResponse struct {
  OrderId int32
  CreatedAt *timestamppb.Timestamp
}

// And a gRPC client
type OrderServiceClient interface {
  CreateOrder(ctx context.Context, req *CreateOrderRequest) (*CreateOrderResponse, error)
}

The Go code is idiomatic Go—CamelCase fields, proper error handling. Python gets snake_case attributes. TypeScript gets readonly properties. Each language gets code that feels native to it, all from the same schema. This is where protobuf saves enormous amounts of effort: no hand-writing data classes, no mismatches between client and server contracts, no version skew between your Go API and your Python consumer.

Common mistakes

Reusing field numbers: We've mentioned this, but it bears repeating. Field numbers are permanent. A junior engineer once deleted a field and reused its number without realizing the impact. The data corruption was silent for weeks until a customer report surfaced the issue.

Changing field types: Moving price from int32 (cents) to double (dollars) looks like a simple refactor, but it breaks the wire format. Old clients encode a 4-byte integer; new code tries to read 8 bytes and gets garbage. Use a new field number and deprecate the old one.

Ignoring buf breaking checks: Some teams run buf lint but skip the breaking checks because they "slow down development." Then a junior engineer ships a breaking change that breaks production for an hour. Buf is fast; the discipline is worth it.

Not versioning your proto definitions: Store your .proto files in version control, tag releases, and keep a changelog. When a consumer needs to regenerate clients, they should be able to check out a specific version and get exactly what they expect.

Over-nesting messages: It's tempting to create deeply nested message hierarchies. But this makes code generation unwieldy and makes backward compatibility harder. Prefer flat structures with clear field names.

Key takeaways

• Field numbers are permanent identifiers in the wire format. Never reuse them; mark unused fields as reserved instead.
• Protobuf's backward and forward compatibility is powerful, but only if you follow the rules: add fields, don't remove them; never change types; always provide defaults.
• Buf enforces these rules through linting and breaking-change detection. Run it in CI to catch mistakes before they reach main.
• Generate type-safe clients in multiple languages from a single .proto definition, eliminating hand-written data classes and version skew.
• In distributed systems where clients and servers roll out at different times, protobuf's strict versioning discipline prevents silent failures and data corruption.