On this page

Serialization & Wire Formats

Inside one process, data is a graph of objects in memory: pointers, padding, language-specific layouts, garbage-collector headers. None of that means anything to another process, let alone another machine running another language. Serialization (also called encoding or marshalling) turns in-memory data into a sequence of bytes that can be stored or sent; deserialization turns bytes back into in-memory data on the other side.

Every RPC, every message on a queue, every row in a cache, every event in a log is serialized somewhere. The choice of format determines how big your messages are, how much CPU you spend on encoding, which languages can talk to each other, and — most importantly over a system's lifetime — whether you can change your data model without breaking running services.

This page explains what serialization must solve, walks through how the major formats encode the same record byte by byte, gives safe/unsafe change rules for each, and ends with guidance for choosing. For RPC framing see RPC Fundamentals and gRPC Deep Dive; for evolving event schemas over time in an event-driven system see Event Schema Evolution.


What Serialization Must Solve

The requirements

RequirementQuestion it answersWhy it matters
Type fidelityDoes a 64-bit integer, a timestamp, raw bytes or a decimal survive the round trip?Silent precision loss is a correctness bug
SizeHow many bytes on the wire / on disk?Network cost, storage cost, cache efficiency
SpeedHow much CPU to encode and decode?At high throughput, serialization can dominate CPU profiles
Schema evolutionCan old code read new data and new code read old data?You can never upgrade every producer and consumer at once
Language supportIs there a mature library for every language you use?A format with a weak library in one language becomes that team's problem
Self-descriptionCan you read the data without a separate schema?Debuggability, ad-hoc tools, long-term archives
Human readabilityCan you curl it and read it?Developer experience, public APIs
Random access / zero-copyCan you read one field without parsing the whole message?Huge messages, memory-mapped files, latency-critical paths
SafetyCan malicious input crash or compromise the decoder?Every decoder that touches network input is an attack surface

No format wins on every row. JSON is readable and universal but large and loosely typed. Protobuf is compact and evolvable but needs a schema to read. Cap'n Proto and FlatBuffers make reads nearly free but produce larger messages and have stricter evolution rules.

Why evolution is the hardest requirement

In a distributed system, producers and consumers are deployed independently. During any rolling deploy, some instances run version N and some run N+1; data written months ago is still sitting in Kafka topics, caches and databases. So you need two properties:

  • Backward compatibility — new code can read data written by old code.
  • Forward compatibility — old code can read data written by new code (and, ideally, preserve fields it doesn't understand when it re-writes the data).
flowchart LR
    subgraph Old["Old code (v1)"]
        P1["Producer v1"]
        C1["Consumer v1"]
    end
    subgraph New["New code (v2)"]
        P2["Producer v2"]
        C2["Consumer v2"]
    end
    P1 -->|"backward compat: new reads old"| C2
    P2 -->|"forward compat: old reads new"| C1
    P1 --> C1
    P2 --> C2

In a rolling deploy, consumers are upgraded before or after producers depending on the change, so you usually need both. Event logs make it worse: a consumer replaying from the beginning of a topic must read every historical version. Event Schema Evolution treats the compatibility modes in depth; here we focus on what each format allows.

Text vs. binary

Text formats (JSON, XML, CSV, YAML)Binary formats (Protobuf, Avro, Thrift, MessagePack, CBOR, …)
Readable without toolsYesNo (need a decoder, often a schema)
SizeLarger: field names repeated, numbers as digitsSmaller: tags/positions instead of names, compact integers
Encode/decode speedSlower: number parsing, escaping, tokenizingFaster: mostly copying and varint decoding
Binary dataNeeds base64 (+33% size)Native
Number precisionFormat-dependent and often ambiguousExact integer widths
SchemaOptional (JSON Schema, XSD)Usually required (except self-describing ones like MessagePack/CBOR)

The running example on this page is one small record:

{"id":150,"name":"testing"}

We'll encode it in each format and count bytes.


JSON

JSON (RFC 8259) is the lingua franca of web APIs. Minified, our record is 27 bytes: every field name is spelled out, the number is three ASCII digits, and the punctuation adds the rest.

Where JSON bites

  • Numbers are underspecified. The JSON grammar allows arbitrary-precision numbers, but most parsers (including JavaScript's JSON.parse) decode them as IEEE-754 doubles, which represent integers exactly only up to 253. A 64-bit ID like 1234567890123456789 silently becomes 1234567890123456800. Twitter's API famously ships IDs twice — id as a number and id_str as a string — for exactly this reason. Send 64-bit integers as strings in JSON APIs.
  • No binary type. Bytes must be base64-encoded: +33% size and an encode/decode step.
  • No dates, decimals or enums. Everything is a string by convention; every service picks its own convention.
  • Duplicate keys are allowed by the grammar with unspecified semantics; different parsers keep the first, the last, or error. That inconsistency has been the root of real security bugs when two components disagree about a payload.
  • Size and speed. Field names are repeated in every record. For large arrays of small records, names can be most of the payload. Compression (gzip, zstd) recovers much of this on the wire, at a CPU cost.

Evolution in JSON

JSON has no schema, so compatibility is a property of your code, not the format. The robust convention is the tolerant reader: ignore unknown fields, treat missing fields as defaults, never reuse a field name for a different meaning. JSON Schema can validate payloads and be checked for compatibility in CI, but nothing in the wire format enforces it.

ChangeSafe?Notes
Add an optional field✅ if readers ignore unknown fieldsStrict deserializers (e.g. "fail on unknown properties") break forward compatibility
Remove a field⚠️Safe only if all readers treat it as optional
Rename a field❌It's a remove plus an add
Change a type (number → string)❌Unless every reader accepts both
Change meaning of a field❌Never; add a new field instead

MessagePack (and CBOR)

MessagePack is "binary JSON": the same data model (maps, arrays, strings, numbers, booleans, null, plus binary and extension types), but with compact type-tagged binary encoding. It's self-describing — no schema needed — and maps 1:1 onto JSON-like data. CBOR (RFC 8949) is a standardised format in the same family, used in WebAuthn and COSE.

Worked encoding

82                         fixmap with 2 entries
a2 69 64                   fixstr len 2: "id"
cc 96                      uint8: 150   (values 0–127 fit in one byte; 150 needs the 0xcc prefix)
a4 6e 61 6d 65             fixstr len 4: "name"
a7 74 65 73 74 69 6e 67    fixstr len 7: "testing"

19 bytes, versus JSON's 27. The savings come from length-prefixed strings (no quotes or escaping), single-byte type tags, and binary numbers. Field names are still present, so evolution behaves like JSON: tolerant readers, no enforced schema.

Use MessagePack/CBOR when you want JSON's flexibility with less size and parsing cost — cache values, internal RPC between dynamically typed services, embedded devices — and don't need schema-enforced evolution.


Protocol Buffers

Protocol Buffers (Protobuf) was developed at Google and open-sourced in 2008. You describe messages in a .proto file; a compiler (protoc) generates code for each language. It is the default payload format for gRPC.

syntax = "proto3";

package demo.v1;

message Person {
  int32  id    = 1;   // the number is the field's identity on the wire
  string name  = 2;
  optional string email = 3;   // explicit presence: can tell "unset" from ""
  repeated string tags  = 4;
  reserved 5, 6;               // never reuse numbers of deleted fields
  reserved "nickname";
}

The wire format

A Protobuf message is a sequence of key–value records. Each key packs the field number and a wire type:

Wire typeIDUsed for
VARINT0int32, int64, uint32, uint64, sint32, sint64, bool, enum
I641fixed64, sfixed64, double
LEN2string, bytes, embedded messages, packed repeated fields
SGROUP / EGROUP3 / 4Groups (deprecated)
I325fixed32, sfixed32, float

Field names never appear on the wire. That is why Protobuf is compact, and why the field number — not the name — is the contract.

Varints, worked

A varint stores an integer in 7-bit groups, least significant group first; the top bit of each byte says "more bytes follow".

Encoding 150:

150 in binary:              1001 0110
split into 7-bit groups:    0000001  0010110      (high group, low group)
emit low group first:       0010110  → set continuation bit → 1001 0110 = 0x96
then high group:            0000001  → no continuation      → 0000 0001 = 0x01
varint(150) = 96 01

Decoding reverses it: strip the top bits (0010110, 0000001), concatenate high-to-low (0000001 0010110 = 150).

Now the whole message Person { id: 150, name: "testing" }:

08            key: field 1, wire type 0  → (1 << 3) | 0 = 0x08
96 01         varint 150
12            key: field 2, wire type 2  → (2 << 3) | 2 = 0x12
07            length 7
74 65 73 74 69 6e 67   "testing"

12 bytes — less than half of JSON. Consequences that follow directly from the wire format:

  • Field numbers 1–15 need a one-byte key; 16–2047 need two bytes. Put frequent fields in 1–15.
  • The maximum field number is 229 − 1; numbers 19000–19999 are reserved for the implementation.
  • Negative int32/int64 values take 10 bytes, because they are sign-extended to 64 bits before varint encoding. For fields that are often negative, use sint32/sint64, which apply ZigZag encoding first: (n << 1) ^ (n >> 63) maps 0, −1, 1, −2, 2 … to 0, 1, 2, 3, 4 …, so small magnitudes stay small.
  • Repeated scalar fields are packed by default in proto3: one LEN record containing all the varints back to back.
  • In proto3, a scalar field with its default value (0, "", false) isn't written at all, unless declared optional. Readers can't distinguish "set to 0" from "not set" for non-optional scalars — a common source of bugs (e.g. a discount of 0% vs. "no discount specified").

Code: encoding by hand (Go)

Doing it by hand once is the fastest way to internalise the format. This program reproduces the bytes above and checks varint and ZigZag round-trips.

// wire.go — go run wire.go
package main

import (
	"encoding/hex"
	"errors"
	"fmt"
)

// AppendVarint encodes v as a base-128 varint: 7 bits per byte, low bits first,
// high bit set on every byte except the last.
func AppendVarint(b []byte, v uint64) []byte {
	for v >= 0x80 {
		b = append(b, byte(v)|0x80)
		v >>= 7
	}
	return append(b, byte(v))
}

func ReadVarint(b []byte) (v uint64, n int, err error) {
	for shift := uint(0); n < len(b) && shift < 64; shift += 7 {
		c := b[n]
		n++
		v |= uint64(c&0x7f) << shift
		if c < 0x80 {
			return v, n, nil
		}
	}
	return 0, 0, errors.New("truncated or overlong varint")
}

// ZigZag maps signed to unsigned so small magnitudes stay small: 0,-1,1,-2 -> 0,1,2,3
func ZigZag(n int64) uint64   { return uint64((n << 1) ^ (n >> 63)) }
func UnZigZag(u uint64) int64 { return int64(u>>1) ^ -int64(u&1) }

const (
	wtVarint = 0
	wtLen    = 2
)

func tag(field int, wt int) uint64 { return uint64(field)<<3 | uint64(wt) }

// EncodePerson hand-encodes: message Person { int32 id = 1; string name = 2; }
func EncodePerson(id int32, name string) []byte {
	var b []byte
	b = AppendVarint(b, tag(1, wtVarint))
	b = AppendVarint(b, uint64(int64(id))) // int32 negatives sign-extend to 10 bytes
	b = AppendVarint(b, tag(2, wtLen))
	b = AppendVarint(b, uint64(len(name)))
	return append(b, name...)
}

func main() {
	fmt.Println("varint(150)      =", hex.EncodeToString(AppendVarint(nil, 150)))
	fmt.Println("varint(300)      =", hex.EncodeToString(AppendVarint(nil, 300)))
	msg := EncodePerson(150, "testing")
	fmt.Printf("Person(150,testing) = %s (%d bytes)\n", hex.EncodeToString(msg), len(msg))
	minusOne := int32(-1)
	fmt.Printf("int32 -1 as varint  = %d bytes\n", len(AppendVarint(nil, uint64(int64(minusOne)))))
	fmt.Printf("sint32 -1 zigzag    = %d -> %d byte\n", ZigZag(-1), len(AppendVarint(nil, ZigZag(-1))))

	// self-checks
	if hex.EncodeToString(msg) != "089601120774657374696e67" {
		panic("unexpected encoding")
	}
	for _, v := range []uint64{0, 1, 127, 128, 150, 300, 1 << 35, ^uint64(0)} {
		got, _, err := ReadVarint(AppendVarint(nil, v))
		if err != nil || got != v {
			panic(fmt.Sprint("varint roundtrip failed for ", v))
		}
	}
	for _, s := range []int64{0, -1, 1, -64, 63, -1 << 40} {
		if UnZigZag(ZigZag(s)) != s {
			panic("zigzag roundtrip failed")
		}
	}
	fmt.Println("self-check passed")
}

Output:

varint(150)      = 9601
varint(300)      = ac02
Person(150,testing) = 089601120774657374696e67 (12 bytes)
int32 -1 as varint  = 10 bytes
sint32 -1 zigzag    = 1 -> 1 byte
self-check passed

Code: decoding without a schema (TypeScript)

Because every record carries its field number and wire type, you can walk any Protobuf message without its .proto — this is how protoc --decode_raw works, and it's how old code skips fields it doesn't know.

// protodump.ts — decode any protobuf message without its schema
type Field = { field: number; wireType: number; value: bigint | Uint8Array };

function readVarint(buf: Uint8Array, pos: number): [bigint, number] {
  let result = 0n;
  for (let shift = 0n; pos < buf.length && shift < 70n; shift += 7n) {
    const byte = buf[pos++];
    result |= BigInt(byte & 0x7f) << shift;
    if ((byte & 0x80) === 0) return [result, pos];
  }
  throw new Error("truncated or overlong varint");
}

export function decode(buf: Uint8Array): Field[] {
  const out: Field[] = [];
  let pos = 0;
  while (pos < buf.length) {
    let key: bigint;
    [key, pos] = readVarint(buf, pos);
    const field = Number(key >> 3n);
    const wireType = Number(key & 7n);
    if (field === 0) throw new Error("field number 0 is invalid");
    switch (wireType) {
      case 0: { // VARINT
        let v: bigint;
        [v, pos] = readVarint(buf, pos);
        out.push({ field, wireType, value: v });
        break;
      }
      case 1: // I64 (fixed64, sfixed64, double)
        out.push({ field, wireType, value: buf.subarray(pos, pos + 8) });
        pos += 8;
        break;
      case 2: { // LEN (string, bytes, embedded message, packed repeated)
        let len: bigint;
        [len, pos] = readVarint(buf, pos);
        const end = pos + Number(len);
        if (end > buf.length) throw new Error("length exceeds buffer");
        out.push({ field, wireType, value: buf.subarray(pos, end) });
        pos = end;
        break;
      }
      case 5: // I32 (fixed32, sfixed32, float)
        out.push({ field, wireType, value: buf.subarray(pos, pos + 4) });
        pos += 4;
        break;
      default:
        throw new Error(`unsupported wire type ${wireType}`); // 3/4 = deprecated groups
    }
  }
  return out;
}

// Usage: the Person{id:150, name:"testing"} bytes from above
const bytes = Uint8Array.from([0x08, 0x96, 0x01, 0x12, 0x07, ...new TextEncoder().encode("testing")]);
for (const f of decode(bytes)) {
  const shown = typeof f.value === "bigint" ? f.value.toString() : new TextDecoder().decode(f.value);
  console.log(`field ${f.field} (wire type ${f.wireType}):`, shown);
}
// field 1 (wire type 0): 150
// field 2 (wire type 2): testing

Note the bounds checks: a decoder facing untrusted input must reject lengths that run past the buffer and varints longer than 10 bytes, or it can be crashed by a few malicious bytes.

Protobuf evolution rules

Old readers skip fields with numbers they don't recognise (the wire type tells them how many bytes to skip), and modern proto3 runtimes retain unknown fields when re-serializing a message (restored in protobuf 3.5 after early proto3 dropped them). New readers see absent fields as defaults. From those mechanics the rules follow:

ChangeSafe?Why
Add a new field with a new number✅Old readers skip it; new readers see default when absent
Remove a field and reserved its number and name✅Old data's field is skipped; reservation prevents accidental reuse
Rename a field✅ on the binary wireNames aren't on the wire — but breaks JSON mapping and text format, and generated-code callers
Reuse a deleted field's number for a new field❌Old data will be misinterpreted as the new field
Change a field's number❌It's a delete plus an add; existing data is lost to new readers
int32 ↔ int64 ↔ uint32 ↔ uint64 ↔ bool⚠️Wire-compatible (all varints), but values are truncated/cast like C when the target is narrower
sint32 ↔ sint64⚠️Compatible with each other, not with int32/int64 (ZigZag vs plain)
string ↔ bytes⚠️Compatible if the bytes are valid UTF-8
fixed32 ↔ sfixed32, fixed64 ↔ sfixed64✅Same wire type and width
Embedded message ↔ bytes⚠️Compatible if the bytes contain an encoded instance of the message
int32 → string or any change of wire type❌Readers can't parse the value
Singular → repeated for scalars⚠️Parsers accept both forms for many types, but packed vs non-packed and "last value wins" semantics make this risky — avoid
Move a single field into a new oneof✅Wire-compatible
Move multiple existing fields into a oneof❌Old data with several set would lose all but one
Add a value to an enum⚠️Wire-safe; old readers see an unknown value (proto3 keeps the number). Always define a 0 = UNSPECIFIED value
Remove an enum value⚠️reserved the number; old data may still contain it
Change default values (proto2)❌Defaults aren't on the wire; readers with different schemas disagree
Add/remove required (proto2)❌Old readers reject messages missing a required field — which is why proto3 removed required entirely

Lint your schemas

Tools like buf breaking compare a .proto change against the previous version and flag wire- and source-incompatible changes in CI. That turns the table above into a build failure instead of an incident.

Protobuf has also introduced Editions (starting with "edition 2023"), which replace the proto2/proto3 syntax split with per-feature settings such as field presence. The wire format is unchanged; the evolution rules above still hold.


Avro

Apache Avro came out of the Hadoop project (Doug Cutting, 2009) with a different philosophy: the schema is always available at read time, so the data doesn't need to carry field tags at all. Schemas are written in JSON:

{
  "type": "record",
  "name": "Person",
  "namespace": "demo",
  "fields": [
    { "name": "id",   "type": "int" },
    { "name": "name", "type": "string" }
  ]
}

Worked encoding

Avro writes fields in schema order with no keys and no field numbers. Integers are ZigZag varints; strings are a ZigZag-varint length followed by UTF-8 bytes.

ac 02                      id: zigzag(150) = 300 → varint(300) = ac 02
0e                         name length: zigzag(7) = 14 = 0x0e
74 65 73 74 69 6e 67       "testing"

10 bytes — the smallest so far, because there's nothing but values. The cost: these bytes are meaningless without the exact schema that wrote them. You can't even tell where id ends without knowing it's an int.

Writer's schema and reader's schema

Avro's key idea is schema resolution. The decoder takes two schemas:

  • the writer's schema — exactly what the producer used, needed to parse the bytes;
  • the reader's schema — what the consumer's code expects.

The library parses with the writer's schema and then maps fields by name to the reader's schema: fields in both are copied (with allowed type promotions such as int → long, float → double, string ↔ bytes); fields only in the writer are skipped; fields only in the reader are filled from the reader's default — and if there's no default, resolution fails.

flowchart LR
    B["Bytes (no field tags)"] --> D["Decoder"]
    WS["Writer schema v1: id int, name string"] --> D
    RS["Reader schema v2: id long, name string, email null-or-string default null"] --> D
    D --> R["Record: id=150, name=testing, email=null"]

Where does the reader get the writer's schema?

ContextMechanism
Object Container Files (Avro files in HDFS/S3)The writer's schema is embedded once in the file header; many records follow
Kafka with a schema registryEach message starts with a small header containing a schema ID; the consumer fetches the schema from the registry once and caches it
RPCSchemas are exchanged in the handshake
Database rowsStore a schema version alongside the data

The Confluent Schema Registry wire format, widely used with Kafka, prefixes each message with one magic byte (0) and a 4-byte schema ID, followed by the Avro payload. The registry also enforces a compatibility mode per subject before accepting a new schema: BACKWARD (the default: new schema can read data written with the previous one), FORWARD, FULL (both), their _TRANSITIVE variants (checked against all previous versions, not just the last), or NONE. See Event Schema Evolution for registry workflows.

sequenceDiagram
    participant P as Producer
    participant SR as Schema Registry
    participant K as Kafka
    participant C as Consumer
    P->>SR: register schema v2 for subject orders-value
    SR-->>P: compatible with v1, id 42
    P->>K: magic 0, schema id 42, avro bytes
    K->>C: message
    C->>SR: fetch schema 42 (cached afterwards)
    SR-->>C: writer schema v2
    Note over C: resolve writer v2 against the consumer's reader schema

Code: schema resolution in practice (Python)

# avro_demo.py — pip install fastavro msgpack
import io, json
import msgpack
from fastavro import parse_schema, schemaless_writer, schemaless_reader

record = {"id": 150, "name": "testing"}

# --- Size of the same record in three encodings ---
print("JSON       :", len(json.dumps(record, separators=(",", ":")).encode()), "bytes")
packed = msgpack.packb(record)
print("MessagePack:", len(packed), "bytes", packed.hex(" "))

writer_v1 = parse_schema({
    "type": "record", "name": "Person", "namespace": "demo",
    "fields": [{"name": "id", "type": "int"}, {"name": "name", "type": "string"}],
})
buf = io.BytesIO()
schemaless_writer(buf, writer_v1, record)
v1_bytes = buf.getvalue()
print("Avro       :", len(v1_bytes), "bytes", v1_bytes.hex(" "))

# --- Schema evolution: reader v2 adds a field WITH a default (backward compatible) ---
reader_v2 = parse_schema({
    "type": "record", "name": "Person", "namespace": "demo",
    "fields": [
        {"name": "id", "type": "long"},                       # int -> long promotion is allowed
        {"name": "name", "type": "string"},
        {"name": "email", "type": ["null", "string"], "default": None},
    ],
})
old_data_new_reader = schemaless_reader(io.BytesIO(v1_bytes), writer_v1, reader_v2)
print("v1 data read with v2 schema:", old_data_new_reader)

# --- A reader field with NO default and absent from the writer schema fails ---
reader_bad = parse_schema({
    "type": "record", "name": "Person", "namespace": "demo",
    "fields": [{"name": "id", "type": "int"}, {"name": "name", "type": "string"},
               {"name": "country", "type": "string"}],
})
try:
    schemaless_reader(io.BytesIO(v1_bytes), writer_v1, reader_bad)
    raise AssertionError("expected schema resolution failure")
except Exception as e:  # fastavro raises SchemaResolutionError
    print("v1 data read with incompatible schema ->", type(e).__name__)

assert v1_bytes.hex() == "ac020e74657374696e67"
assert old_data_new_reader == {"id": 150, "name": "testing", "email": None}
print("self-check passed")

Output:

JSON       : 27 bytes
MessagePack: 19 bytes 82 a2 69 64 cc 96 a4 6e 61 6d 65 a7 74 65 73 74 69 6e 67
Avro       : 10 bytes ac 02 0e 74 65 73 74 69 6e 67
v1 data read with v2 schema: {'id': 150, 'name': 'testing', 'email': None}
v1 data read with incompatible schema -> SchemaResolutionError
self-check passed

Avro evolution rules

ChangeBackward (new reads old)Forward (old reads new)Notes
Add field with default✅✅The standard safe change
Add field without default❌✅New readers can't fill it from old data
Remove field that had a default✅✅Old readers fill it from their default
Remove field without default✅❌Old readers require it
Rename field⚠️⚠️Use aliases in the reader schema; plain rename is remove + add
Promote int → long / float / double, long → float/double, float → double✅❌Old readers can't narrow
string ↔ bytes✅✅Allowed promotion both ways
Add a branch to a union✅❌Old readers fail on the new branch
Add an enum symbol✅⚠️Old readers fail unless the enum declares a default symbol
Change field order✅✅Resolution is by name
Change a field's type incompatibly❌❌—

Two Avro-specific traps:

  • Union defaults must match the first branch. A field {"type": ["null", "string"], "default": null} is valid; {"type": ["string", "null"], "default": null} is not. Put "null" first for optional fields.
  • Names and namespaces matter. Records resolve by full name; renaming a record type or changing its namespace breaks resolution unless aliased.

Apache Thrift

Thrift was created at Facebook (open-sourced in 2007, now an Apache project). Like Protobuf, it uses an IDL with numbered fields and generates code; unlike Protobuf, it bundles a full RPC framework (transports, servers) and offers several wire protocols for the same IDL.

struct Person {
  1: required i32    id,
  2: optional string name,
  3: optional string email,
}

Encoding the example

TBinaryProtocol is simple and fixed-width: each field has a 1-byte type, a 2-byte field ID, then the value (i32 as 4 bytes, strings as 4-byte length + bytes), and a stop byte ends the struct:

08 00 01 00 00 00 96                   type i32 (8), id 1, value 150
0b 00 02 00 00 00 07 74 65 73 74 69 6e 67   type string (11), id 2, length 7, "testing"
00                                     stop

22 bytes. TCompactProtocol shrinks this: the field header packs a delta from the previous field ID with a 4-bit type into one byte, and integers are ZigZag varints:

15             delta 1, compact type i32 (5)
ac 02          zigzag varint 150
18             delta 1, compact type binary/string (8)
07             varint length 7
74 65 73 74 69 6e 67   "testing"
00             stop

13 bytes — essentially Protobuf-sized.

Thrift evolution rules

Thrift's rules mirror Protobuf's because the mechanics are the same (field IDs on the wire, unknown fields skipped by type):

ChangeSafe?
Add an optional field with a new ID✅
Remove an optional field (never reuse its ID)✅
Rename a field✅ on binary/compact protocols (names aren't on the wire); ❌ for the JSON protocol
Change a field ID or reuse one❌
Change a field's type❌ (no varint-family compatibility as in Protobuf)
Add a required field❌ — old writers won't send it, new readers reject
Make a required field optional or remove it❌ — old readers reject messages without it

The Thrift community's advice is "required is forever": once a field is required, you can never remove it without coordinated upgrades. Most teams simply never use required, the same conclusion that led proto3 to drop it.


Cap'n Proto and FlatBuffers: Zero-Copy Formats

Protobuf, Avro and Thrift all have a parse step: the decoder reads the bytes and builds language objects. For large messages, or when you only need a few fields, that step is wasted work. Zero-copy formats lay out the bytes so that the encoded buffer is the data structure — you read fields directly from the buffer (or a memory-mapped file) with pointer arithmetic.

flowchart LR
    subgraph Parse["Parse-based (Protobuf, Avro, Thrift)"]
        B1["Bytes"] -->|"decode every field"| O1["Language objects"]
        O1 --> A1["Read field"]
    end
    subgraph ZeroCopy["Zero-copy (Cap'n Proto, FlatBuffers)"]
        B2["Bytes / mmap"] -->|"offset arithmetic"| A2["Read field"]
    end

Cap'n Proto

Created by Kenton Varda, who previously led Protocol Buffers v2 at Google. Messages are made of 8-byte words; each struct has a fixed-size data section (scalars at fixed offsets) and a pointer section (relative pointers to text, lists and sub-structs). Fields are numbered with ordinals (@0, @1), and new fields must take the next ordinal.

struct Person {
  id   @0 :UInt32;
  name @1 :Text;
  email @2 :Text;   # added later: appended, never inserted
}

For our example the unpacked encoding is roughly: a segment table (8 bytes), a root pointer (8), one data word holding id (8), one pointer word for name (8), and the NUL-terminated text padded to a word (8) — about 40 bytes. Cap'n Proto offers an optional packing scheme that compresses runs of zero bytes, bringing the size much closer to Protobuf's while still being cheap to unpack. Its RPC system adds promise pipelining (calling methods on results that haven't arrived yet).

FlatBuffers

Created at Google by Wouter van Oortmerssen, originally for games, and later used for TensorFlow Lite's model format. A FlatBuffer table starts with an offset to a vtable that lists, for each field, its offset within the table (or 0 if absent). Reading a field is: look up the vtable slot, add the offset, read. Absent fields cost nothing in the table and read as their default.

table Person {
  id:   uint (id: 0);
  name: string (id: 1);
  email: string (id: 2);   // add at the end; deprecate, never delete
}
root_type Person;

Our record comes to a few tens of bytes, depending on alignment and vtable sharing.

Zero-copy trade-offs

BenefitCost
Read latencyField access without a parse pass; ideal for mmap'd files—
Partial readsRead 1 field of a 10 MB message cheaply—
Size—Alignment padding and offsets make messages larger than Protobuf (unless packed/compressed)
Mutation—Building messages uses builder APIs; in-place modification is limited
EvolutionSimilar to Protobuf: add fields, never reuseStricter: fields must be added at the end of the ordinal/ID sequence; types fixed
Validation—Readers of untrusted data must run the verifier (FlatBuffers) or rely on bounds-checked readers (Cap'n Proto), because offsets in malicious buffers could point anywhere
ChangeCap'n ProtoFlatBuffers
Add field at the end of the ordinal/ID sequence✅✅
Insert a field in the middle❌ (ordinals must be sequential)❌ without explicit id: attributes
Remove a field❌ (keep it; rename to mark obsolete)⚠️ mark deprecated — the slot stays reserved
Rename a field✅ (binary)✅ (binary); breaks JSON conversion
Change a field's type❌❌
Change a default value❌ (defaults are XOR-encoded into data)❌ (readers with different defaults disagree)

Comparison

Size of the running example

FormatBytes for {id:150, name:"testing"}Verified
JSON (minified)27Computed
MessagePack19Computed (script above)
Thrift Binary22Hand-encoded
Thrift Compact13Hand-encoded
Protobuf12Computed (Go program above)
Avro (no header)10Computed (script above)
Cap'n Proto (unpacked, with framing)~40Approximate
FlatBuffersa few tensApproximate

Tiny records exaggerate the differences in fixed overhead. For realistic payloads, the gap between JSON and binary formats is typically a few times in size, and compression narrows it further.

Representative performance characteristics

These are representative relative characteristics, not measurements: real numbers depend heavily on the library, language, message shape and hardware. Benchmark your own payloads before deciding.

FormatRelative sizeEncode CPUDecode CPUPartial readSchema required to read
JSONLargestHighHigh (number/string parsing)NoNo
MessagePack / CBORMediumMediumMediumNoNo
ProtobufSmallLowLowNo (parse whole message)Yes, for meaning (structure is walkable)
Thrift CompactSmallLowLowNoYes, for meaning
AvroSmallestLowLowNoYes — can't parse without the writer's schema
Cap'n Proto / FlatBuffersMedium–large (smaller if packed)Low~None (access in place)YesYes

Evolution model at a glance

FormatField identity on the wireUnknown fieldsDefaultsTypical guard rail
JSONNameIgnored if reader is tolerantReader codeJSON Schema checks in CI
MessagePackNameSame as JSONReader codeConventions
ProtobufField numberSkipped and preservedType defaults; optional for presencereserved, buf breaking
ThriftField IDSkippedPer-field defaults in IDLNever use required
AvroNone (position + writer schema)Skipped via schema resolutionReader-schema defaultsSchema registry compatibility mode
Cap'n ProtoOrdinal / offsetIgnoredDefaults in schemaAppend-only ordinals
FlatBuffersvtable slotIgnoredDefaults in schemaAppend-only, deprecated

Failure Modes

The silent reinterpretation

The worst serialization bug isn't a crash — it's data that decodes successfully into the wrong meaning. Reusing a Protobuf field number, reordering an Avro schema's fields without the writer's schema (e.g. a consumer hardcoding the schema), or changing a JSON field from cents to dollars all produce plausible-looking garbage.

War Story (illustrative)

A team deleted a deprecated int64 legacy_discount = 7 field from an order message and, a few months later, added int64 loyalty_points = 7 without a reserved statement. Orders still in a retry queue from before the change were decoded by the new consumers with their old discount value now appearing as loyalty points. Nothing errored. The discrepancy surfaced weeks later in a finance reconciliation. The fix was mechanical — reserved 7; and a CI breaking-change check — but cleaning up the corrupted ledger was not.

Unsafe deserialization

Formats that can instantiate arbitrary types from the input — Java's native ObjectInputStream, Python's pickle, YAML loaders that construct arbitrary objects, .NET's BinaryFormatter — can execute code chosen by whoever crafted the bytes. They must never be used on untrusted input. Prefer data-only formats (JSON, Protobuf, Avro …), use safe loaders (yaml.safe_load), and treat any "deserialize to an arbitrary class" feature as a code-execution feature. The OWASP pages under Security cover this class of vulnerability.

Resource exhaustion

Even data-only decoders must defend against hostile input: a length prefix claiming 4 GB, deeply nested messages that blow the stack, or a JSON body with millions of keys. Set maximum message sizes (gRPC, for example, limits received messages to 4 MB by default), cap nesting depth, and stream large payloads instead of buffering.

Precision and type mismatches across languages

64-bit integers in JavaScript, unsigned integers in Java (which has none), decimal money values in floating point, timestamps with and without time zones — these are cross-language problems that the format exposes rather than causes. Use explicit types: Protobuf's google.protobuf.Timestamp, Avro's logical types (timestamp-millis, decimal), strings for 64-bit IDs in JSON.


Choosing a Format

flowchart TD
    Q1{"Public API or browser clients?"} -->|"yes"| J["JSON (maybe with JSON Schema / OpenAPI)"]
    Q1 -->|"no"| Q2{"Data at rest in big-data files or Kafka with a registry?"}
    Q2 -->|"yes"| A["Avro (or Protobuf with a registry)"]
    Q2 -->|"no"| Q3{"Need to read fields without parsing, e.g. mmap or huge messages?"}
    Q3 -->|"yes"| Z["FlatBuffers or Cap'n Proto"]
    Q3 -->|"no"| Q4{"Service-to-service RPC with codegen?"}
    Q4 -->|"yes"| P["Protobuf with gRPC (or Thrift if already invested)"]
    Q4 -->|"no"| M["MessagePack or CBOR for schemaless binary"]
Use caseGood defaultWhy
Public REST APIJSONUniversal tooling, readable, browser-native
Internal RPCProtobuf + gRPCCompact, fast, strong evolution story, codegen everywhere
Kafka events, data lakeAvro + schema registry (or Protobuf + registry)Smallest payloads, registry-enforced compatibility, schema-on-read
Mobile/game assets, ML models, mmap'd indexesFlatBuffersZero-copy reads, small runtime
Low-latency RPC with very large messagesCap'n ProtoNo parse step, promise pipelining
Cache values in dynamic languagesMessagePack / CBORSchemaless, smaller and faster than JSON
Config filesJSON / YAML / TOMLHuman editing matters more than size
Long-term archivesAvro container files or ParquetSchema travels with the data

Rules that hold regardless of format:

  1. Treat schemas as code: version them, review them, and run breaking-change checks in CI.
  2. Never reuse identifiers — field numbers, IDs, ordinals or names with a changed meaning.
  3. Add, don't change. New meaning → new field. Deprecate old fields; delete only when no data or code uses them, and reserve the identifier.
  4. Default to optional. Required fields are a compatibility trap in every format that has them.
  5. Know your deploy order. Backward compatibility lets you upgrade consumers first; forward compatibility lets you upgrade producers first. Event logs need both, transitively.

Key Takeaways

  1. Serialization must balance type fidelity, size, speed and evolution; evolution is the requirement that bites hardest over time.
  2. JSON is universal and readable but loosely typed (beware 64-bit integers) and verbose.
  3. Protobuf identifies fields by number; varints and ZigZag keep integers small; unknown fields are skipped and preserved. Never reuse numbers.
  4. Avro writes no field tags at all; decoding needs the writer's schema, resolved against the reader's by name, with defaults filling gaps. A schema registry makes that practical.
  5. Thrift is Protobuf-like with multiple protocols; avoid required.
  6. Cap'n Proto and FlatBuffers skip parsing entirely, trading size and stricter evolution for near-free reads.
  7. Defend decoders: size limits, depth limits, bounds checks, and no arbitrary-type deserialization of untrusted input.

Interview Questions

1. Walk through how Protobuf encodes `id = 150` in field 1.

The key is (1 << 3) | 0 = 0x08 (field 1, wire type VARINT). 150 is 10010110 in binary; split into 7-bit groups low-first: 0010110 with the continuation bit → 0x96, then 0000001 → 0x01. So the bytes are 08 96 01.

2. Why do negative int32 values take 10 bytes in Protobuf, and what's the fix?

int32 values are sign-extended to 64 bits before varint encoding, so −1 becomes 64 one-bits, which needs 10 seven-bit groups. Use sint32/sint64, which ZigZag-encode first (−1 → 1, 1 → 2, −2 → 3), so small negative numbers are 1 byte.

3. How can Avro be smaller than Protobuf, and what does that cost?

Avro writes no field tags or numbers — just values in schema order. The cost is that bytes can't be parsed (not even skipped) without the exact writer's schema. Systems solve this by embedding the schema in file headers or putting a schema ID in each message and fetching the schema from a registry.

4. Which changes are safe in a Protobuf schema? Which aren't?

Safe: adding fields with new numbers; removing fields while reserving their numbers and names; renaming on the binary wire (though it breaks JSON); some type changes within the varint family with truncation caveats. Unsafe: reusing or changing a field number, changing wire type (e.g. int32 → string), moving multiple existing fields into a oneof, and in proto2 adding/removing required fields.

5. Explain backward vs forward compatibility and which you need for a rolling deploy.

Backward: new readers can read old data. Forward: old readers can read new data. During a rolling deploy both versions run simultaneously, so data written by either can be read by either — you need both (full compatibility), unless you strictly sequence upgrades (consumers first requires only backward; producers first requires only forward). Replaying long-lived logs requires transitive compatibility across all historical versions.

6. Why did proto3 remove required fields?

A required field can never be safely removed or added: old readers reject messages missing it, and new readers reject old messages without it. In large systems with independent deploys this turned routine schema cleanups into outages. Making everything optional at the wire level and validating in application code is safer.

7. When would you choose FlatBuffers or Cap'n Proto over Protobuf?

When parse cost or partial reads dominate: memory-mapped data files, large messages where you read a few fields, latency-critical paths (games, ML model loading, high-frequency systems). You accept larger messages (unless packed), builder-style APIs, append-only evolution, and the need to verify untrusted buffers.

8. A JavaScript client shows the wrong user ID. The API returns `{"id": 1234567890123456789}`. Why?

JSON numbers are decoded as IEEE-754 doubles in JavaScript, which represent integers exactly only up to 2^53. The 64-bit ID is rounded. Return IDs as strings (or add an id_str field), or use a parser with BigInt support.


Further Reading

  • Protocol Buffers documentation: Encoding and Updating A Message Type (protobuf.dev)
  • Apache Avro specification: Schema Resolution and Binary Encoding
  • Confluent documentation: Schema Evolution and Compatibility, Wire Format
  • Apache Thrift: Thrift: Scalable Cross-Language Services Implementation (Slee, Agarwal, Kwiatkowski, Facebook, 2007)
  • Cap'n Proto Encoding spec (capnproto.org) and FlatBuffers Internals (flatbuffers.dev)
  • Martin Kleppmann, Designing Data-Intensive Applications, chapter 4 (Encoding and Evolution)
  • Related pages: RPC Fundamentals, gRPC Deep Dive, gRPC Internals, Event Schema Evolution, API Versioning