Serialization & Wire Formats
Inside one process, data is a graph of objects in memory: pointers, padding, language-specific layouts, garbage-collector headers. None of that means anything to another process, let alone another machine running another language. Serialization (also called encoding or marshalling) turns in-memory data into a sequence of bytes that can be stored or sent; deserialization turns bytes back into in-memory data on the other side.
Every RPC, every message on a queue, every row in a cache, every event in a log is serialized somewhere. The choice of format determines how big your messages are, how much CPU you spend on encoding, which languages can talk to each other, and — most importantly over a system's lifetime — whether you can change your data model without breaking running services.
This page explains what serialization must solve, walks through how the major formats encode the same record byte by byte, gives safe/unsafe change rules for each, and ends with guidance for choosing. For RPC framing see RPC Fundamentals and gRPC Deep Dive; for evolving event schemas over time in an event-driven system see Event Schema Evolution.
What Serialization Must Solve
The requirements
| Requirement | Question it answers | Why it matters |
|---|---|---|
| Type fidelity | Does a 64-bit integer, a timestamp, raw bytes or a decimal survive the round trip? | Silent precision loss is a correctness bug |
| Size | How many bytes on the wire / on disk? | Network cost, storage cost, cache efficiency |
| Speed | How much CPU to encode and decode? | At high throughput, serialization can dominate CPU profiles |
| Schema evolution | Can old code read new data and new code read old data? | You can never upgrade every producer and consumer at once |
| Language support | Is there a mature library for every language you use? | A format with a weak library in one language becomes that team's problem |
| Self-description | Can you read the data without a separate schema? | Debuggability, ad-hoc tools, long-term archives |
| Human readability | Can you curl it and read it? | Developer experience, public APIs |
| Random access / zero-copy | Can you read one field without parsing the whole message? | Huge messages, memory-mapped files, latency-critical paths |
| Safety | Can malicious input crash or compromise the decoder? | Every decoder that touches network input is an attack surface |
No format wins on every row. JSON is readable and universal but large and loosely typed. Protobuf is compact and evolvable but needs a schema to read. Cap'n Proto and FlatBuffers make reads nearly free but produce larger messages and have stricter evolution rules.
Why evolution is the hardest requirement
In a distributed system, producers and consumers are deployed independently. During any rolling deploy, some instances run version N and some run N+1; data written months ago is still sitting in Kafka topics, caches and databases. So you need two properties:
- Backward compatibility — new code can read data written by old code.
- Forward compatibility — old code can read data written by new code (and, ideally, preserve fields it doesn't understand when it re-writes the data).
flowchart LR
subgraph Old["Old code (v1)"]
P1["Producer v1"]
C1["Consumer v1"]
end
subgraph New["New code (v2)"]
P2["Producer v2"]
C2["Consumer v2"]
end
P1 -->|"backward compat: new reads old"| C2
P2 -->|"forward compat: old reads new"| C1
P1 --> C1
P2 --> C2
In a rolling deploy, consumers are upgraded before or after producers depending on the change, so you usually need both. Event logs make it worse: a consumer replaying from the beginning of a topic must read every historical version. Event Schema Evolution treats the compatibility modes in depth; here we focus on what each format allows.
Text vs. binary
| Text formats (JSON, XML, CSV, YAML) | Binary formats (Protobuf, Avro, Thrift, MessagePack, CBOR, …) | |
|---|---|---|
| Readable without tools | Yes | No (need a decoder, often a schema) |
| Size | Larger: field names repeated, numbers as digits | Smaller: tags/positions instead of names, compact integers |
| Encode/decode speed | Slower: number parsing, escaping, tokenizing | Faster: mostly copying and varint decoding |
| Binary data | Needs base64 (+33% size) | Native |
| Number precision | Format-dependent and often ambiguous | Exact integer widths |
| Schema | Optional (JSON Schema, XSD) | Usually required (except self-describing ones like MessagePack/CBOR) |
The running example on this page is one small record:
{"id":150,"name":"testing"}
We'll encode it in each format and count bytes.
JSON
JSON (RFC 8259) is the lingua franca of web APIs. Minified, our record is 27 bytes: every field name is spelled out, the number is three ASCII digits, and the punctuation adds the rest.
Where JSON bites
- Numbers are underspecified. The JSON grammar allows arbitrary-precision numbers, but most parsers (including JavaScript's
JSON.parse) decode them as IEEE-754 doubles, which represent integers exactly only up to 253. A 64-bit ID like1234567890123456789silently becomes1234567890123456800. Twitter's API famously ships IDs twice —idas a number andid_stras a string — for exactly this reason. Send 64-bit integers as strings in JSON APIs. - No binary type. Bytes must be base64-encoded: +33% size and an encode/decode step.
- No dates, decimals or enums. Everything is a string by convention; every service picks its own convention.
- Duplicate keys are allowed by the grammar with unspecified semantics; different parsers keep the first, the last, or error. That inconsistency has been the root of real security bugs when two components disagree about a payload.
- Size and speed. Field names are repeated in every record. For large arrays of small records, names can be most of the payload. Compression (gzip, zstd) recovers much of this on the wire, at a CPU cost.
Evolution in JSON
JSON has no schema, so compatibility is a property of your code, not the format. The robust convention is the tolerant reader: ignore unknown fields, treat missing fields as defaults, never reuse a field name for a different meaning. JSON Schema can validate payloads and be checked for compatibility in CI, but nothing in the wire format enforces it.
| Change | Safe? | Notes |
|---|---|---|
| Add an optional field | ✅ if readers ignore unknown fields | Strict deserializers (e.g. "fail on unknown properties") break forward compatibility |
| Remove a field | ⚠️ | Safe only if all readers treat it as optional |
| Rename a field | ❌ | It's a remove plus an add |
| Change a type (number → string) | ❌ | Unless every reader accepts both |
| Change meaning of a field | ❌ | Never; add a new field instead |
MessagePack (and CBOR)
MessagePack is "binary JSON": the same data model (maps, arrays, strings, numbers, booleans, null, plus binary and extension types), but with compact type-tagged binary encoding. It's self-describing — no schema needed — and maps 1:1 onto JSON-like data. CBOR (RFC 8949) is a standardised format in the same family, used in WebAuthn and COSE.
Worked encoding
82 fixmap with 2 entries
a2 69 64 fixstr len 2: "id"
cc 96 uint8: 150 (values 0–127 fit in one byte; 150 needs the 0xcc prefix)
a4 6e 61 6d 65 fixstr len 4: "name"
a7 74 65 73 74 69 6e 67 fixstr len 7: "testing"
19 bytes, versus JSON's 27. The savings come from length-prefixed strings (no quotes or escaping), single-byte type tags, and binary numbers. Field names are still present, so evolution behaves like JSON: tolerant readers, no enforced schema.
Use MessagePack/CBOR when you want JSON's flexibility with less size and parsing cost — cache values, internal RPC between dynamically typed services, embedded devices — and don't need schema-enforced evolution.
Protocol Buffers
Protocol Buffers (Protobuf) was developed at Google and open-sourced in 2008. You describe messages in a .proto file; a compiler (protoc) generates code for each language. It is the default payload format for gRPC.
syntax = "proto3";
package demo.v1;
message Person {
int32 id = 1; // the number is the field's identity on the wire
string name = 2;
optional string email = 3; // explicit presence: can tell "unset" from ""
repeated string tags = 4;
reserved 5, 6; // never reuse numbers of deleted fields
reserved "nickname";
}
The wire format
A Protobuf message is a sequence of key–value records. Each key packs the field number and a wire type:
| Wire type | ID | Used for |
|---|---|---|
| VARINT | 0 | int32, int64, uint32, uint64, sint32, sint64, bool, enum |
| I64 | 1 | fixed64, sfixed64, double |
| LEN | 2 | string, bytes, embedded messages, packed repeated fields |
| SGROUP / EGROUP | 3 / 4 | Groups (deprecated) |
| I32 | 5 | fixed32, sfixed32, float |
Field names never appear on the wire. That is why Protobuf is compact, and why the field number — not the name — is the contract.
Varints, worked
A varint stores an integer in 7-bit groups, least significant group first; the top bit of each byte says "more bytes follow".
Encoding 150:
150 in binary: 1001 0110
split into 7-bit groups: 0000001 0010110 (high group, low group)
emit low group first: 0010110 → set continuation bit → 1001 0110 = 0x96
then high group: 0000001 → no continuation → 0000 0001 = 0x01
varint(150) = 96 01
Decoding reverses it: strip the top bits (0010110, 0000001), concatenate high-to-low (0000001 0010110 = 150).
Now the whole message Person { id: 150, name: "testing" }:
08 key: field 1, wire type 0 → (1 << 3) | 0 = 0x08
96 01 varint 150
12 key: field 2, wire type 2 → (2 << 3) | 2 = 0x12
07 length 7
74 65 73 74 69 6e 67 "testing"
12 bytes — less than half of JSON. Consequences that follow directly from the wire format:
- Field numbers 1–15 need a one-byte key; 16–2047 need two bytes. Put frequent fields in 1–15.
- The maximum field number is 229 − 1; numbers 19000–19999 are reserved for the implementation.
- Negative
int32/int64values take 10 bytes, because they are sign-extended to 64 bits before varint encoding. For fields that are often negative, usesint32/sint64, which apply ZigZag encoding first:(n << 1) ^ (n >> 63)maps 0, −1, 1, −2, 2 … to 0, 1, 2, 3, 4 …, so small magnitudes stay small. - Repeated scalar fields are packed by default in proto3: one LEN record containing all the varints back to back.
- In proto3, a scalar field with its default value (0, "", false) isn't written at all, unless declared
optional. Readers can't distinguish "set to 0" from "not set" for non-optional scalars — a common source of bugs (e.g. a discount of 0% vs. "no discount specified").
Code: encoding by hand (Go)
Doing it by hand once is the fastest way to internalise the format. This program reproduces the bytes above and checks varint and ZigZag round-trips.
// wire.go — go run wire.go
package main
import (
"encoding/hex"
"errors"
"fmt"
)
// AppendVarint encodes v as a base-128 varint: 7 bits per byte, low bits first,
// high bit set on every byte except the last.
func AppendVarint(b []byte, v uint64) []byte {
for v >= 0x80 {
b = append(b, byte(v)|0x80)
v >>= 7
}
return append(b, byte(v))
}
func ReadVarint(b []byte) (v uint64, n int, err error) {
for shift := uint(0); n < len(b) && shift < 64; shift += 7 {
c := b[n]
n++
v |= uint64(c&0x7f) << shift
if c < 0x80 {
return v, n, nil
}
}
return 0, 0, errors.New("truncated or overlong varint")
}
// ZigZag maps signed to unsigned so small magnitudes stay small: 0,-1,1,-2 -> 0,1,2,3
func ZigZag(n int64) uint64 { return uint64((n << 1) ^ (n >> 63)) }
func UnZigZag(u uint64) int64 { return int64(u>>1) ^ -int64(u&1) }
const (
wtVarint = 0
wtLen = 2
)
func tag(field int, wt int) uint64 { return uint64(field)<<3 | uint64(wt) }
// EncodePerson hand-encodes: message Person { int32 id = 1; string name = 2; }
func EncodePerson(id int32, name string) []byte {
var b []byte
b = AppendVarint(b, tag(1, wtVarint))
b = AppendVarint(b, uint64(int64(id))) // int32 negatives sign-extend to 10 bytes
b = AppendVarint(b, tag(2, wtLen))
b = AppendVarint(b, uint64(len(name)))
return append(b, name...)
}
func main() {
fmt.Println("varint(150) =", hex.EncodeToString(AppendVarint(nil, 150)))
fmt.Println("varint(300) =", hex.EncodeToString(AppendVarint(nil, 300)))
msg := EncodePerson(150, "testing")
fmt.Printf("Person(150,testing) = %s (%d bytes)\n", hex.EncodeToString(msg), len(msg))
minusOne := int32(-1)
fmt.Printf("int32 -1 as varint = %d bytes\n", len(AppendVarint(nil, uint64(int64(minusOne)))))
fmt.Printf("sint32 -1 zigzag = %d -> %d byte\n", ZigZag(-1), len(AppendVarint(nil, ZigZag(-1))))
// self-checks
if hex.EncodeToString(msg) != "089601120774657374696e67" {
panic("unexpected encoding")
}
for _, v := range []uint64{0, 1, 127, 128, 150, 300, 1 << 35, ^uint64(0)} {
got, _, err := ReadVarint(AppendVarint(nil, v))
if err != nil || got != v {
panic(fmt.Sprint("varint roundtrip failed for ", v))
}
}
for _, s := range []int64{0, -1, 1, -64, 63, -1 << 40} {
if UnZigZag(ZigZag(s)) != s {
panic("zigzag roundtrip failed")
}
}
fmt.Println("self-check passed")
}
Output:
varint(150) = 9601
varint(300) = ac02
Person(150,testing) = 089601120774657374696e67 (12 bytes)
int32 -1 as varint = 10 bytes
sint32 -1 zigzag = 1 -> 1 byte
self-check passed
Code: decoding without a schema (TypeScript)
Because every record carries its field number and wire type, you can walk any Protobuf message without its .proto — this is how protoc --decode_raw works, and it's how old code skips fields it doesn't know.
// protodump.ts — decode any protobuf message without its schema
type Field = { field: number; wireType: number; value: bigint | Uint8Array };
function readVarint(buf: Uint8Array, pos: number): [bigint, number] {
let result = 0n;
for (let shift = 0n; pos < buf.length && shift < 70n; shift += 7n) {
const byte = buf[pos++];
result |= BigInt(byte & 0x7f) << shift;
if ((byte & 0x80) === 0) return [result, pos];
}
throw new Error("truncated or overlong varint");
}
export function decode(buf: Uint8Array): Field[] {
const out: Field[] = [];
let pos = 0;
while (pos < buf.length) {
let key: bigint;
[key, pos] = readVarint(buf, pos);
const field = Number(key >> 3n);
const wireType = Number(key & 7n);
if (field === 0) throw new Error("field number 0 is invalid");
switch (wireType) {
case 0: { // VARINT
let v: bigint;
[v, pos] = readVarint(buf, pos);
out.push({ field, wireType, value: v });
break;
}
case 1: // I64 (fixed64, sfixed64, double)
out.push({ field, wireType, value: buf.subarray(pos, pos + 8) });
pos += 8;
break;
case 2: { // LEN (string, bytes, embedded message, packed repeated)
let len: bigint;
[len, pos] = readVarint(buf, pos);
const end = pos + Number(len);
if (end > buf.length) throw new Error("length exceeds buffer");
out.push({ field, wireType, value: buf.subarray(pos, end) });
pos = end;
break;
}
case 5: // I32 (fixed32, sfixed32, float)
out.push({ field, wireType, value: buf.subarray(pos, pos + 4) });
pos += 4;
break;
default:
throw new Error(`unsupported wire type ${wireType}`); // 3/4 = deprecated groups
}
}
return out;
}
// Usage: the Person{id:150, name:"testing"} bytes from above
const bytes = Uint8Array.from([0x08, 0x96, 0x01, 0x12, 0x07, ...new TextEncoder().encode("testing")]);
for (const f of decode(bytes)) {
const shown = typeof f.value === "bigint" ? f.value.toString() : new TextDecoder().decode(f.value);
console.log(`field ${f.field} (wire type ${f.wireType}):`, shown);
}
// field 1 (wire type 0): 150
// field 2 (wire type 2): testing
Note the bounds checks: a decoder facing untrusted input must reject lengths that run past the buffer and varints longer than 10 bytes, or it can be crashed by a few malicious bytes.
Protobuf evolution rules
Old readers skip fields with numbers they don't recognise (the wire type tells them how many bytes to skip), and modern proto3 runtimes retain unknown fields when re-serializing a message (restored in protobuf 3.5 after early proto3 dropped them). New readers see absent fields as defaults. From those mechanics the rules follow:
| Change | Safe? | Why |
|---|---|---|
| Add a new field with a new number | ✅ | Old readers skip it; new readers see default when absent |
Remove a field and reserved its number and name | ✅ | Old data's field is skipped; reservation prevents accidental reuse |
| Rename a field | ✅ on the binary wire | Names aren't on the wire — but breaks JSON mapping and text format, and generated-code callers |
| Reuse a deleted field's number for a new field | ❌ | Old data will be misinterpreted as the new field |
| Change a field's number | ❌ | It's a delete plus an add; existing data is lost to new readers |
int32 ↔ int64 ↔ uint32 ↔ uint64 ↔ bool | ⚠️ | Wire-compatible (all varints), but values are truncated/cast like C when the target is narrower |
sint32 ↔ sint64 | ⚠️ | Compatible with each other, not with int32/int64 (ZigZag vs plain) |
string ↔ bytes | ⚠️ | Compatible if the bytes are valid UTF-8 |
fixed32 ↔ sfixed32, fixed64 ↔ sfixed64 | ✅ | Same wire type and width |
Embedded message ↔ bytes | ⚠️ | Compatible if the bytes contain an encoded instance of the message |
int32 → string or any change of wire type | ❌ | Readers can't parse the value |
Singular → repeated for scalars | ⚠️ | Parsers accept both forms for many types, but packed vs non-packed and "last value wins" semantics make this risky — avoid |
Move a single field into a new oneof | ✅ | Wire-compatible |
Move multiple existing fields into a oneof | ❌ | Old data with several set would lose all but one |
Add a value to an enum | ⚠️ | Wire-safe; old readers see an unknown value (proto3 keeps the number). Always define a 0 = UNSPECIFIED value |
| Remove an enum value | ⚠️ | reserved the number; old data may still contain it |
| Change default values (proto2) | ❌ | Defaults aren't on the wire; readers with different schemas disagree |
Add/remove required (proto2) | ❌ | Old readers reject messages missing a required field — which is why proto3 removed required entirely |
Lint your schemas
Tools like buf breaking compare a .proto change against the previous version and flag wire- and source-incompatible changes in CI. That turns the table above into a build failure instead of an incident.
Protobuf has also introduced Editions (starting with "edition 2023"), which replace the proto2/proto3 syntax split with per-feature settings such as field presence. The wire format is unchanged; the evolution rules above still hold.
Avro
Apache Avro came out of the Hadoop project (Doug Cutting, 2009) with a different philosophy: the schema is always available at read time, so the data doesn't need to carry field tags at all. Schemas are written in JSON:
{
"type": "record",
"name": "Person",
"namespace": "demo",
"fields": [
{ "name": "id", "type": "int" },
{ "name": "name", "type": "string" }
]
}
Worked encoding
Avro writes fields in schema order with no keys and no field numbers. Integers are ZigZag varints; strings are a ZigZag-varint length followed by UTF-8 bytes.
ac 02 id: zigzag(150) = 300 → varint(300) = ac 02
0e name length: zigzag(7) = 14 = 0x0e
74 65 73 74 69 6e 67 "testing"
10 bytes — the smallest so far, because there's nothing but values. The cost: these bytes are meaningless without the exact schema that wrote them. You can't even tell where id ends without knowing it's an int.
Writer's schema and reader's schema
Avro's key idea is schema resolution. The decoder takes two schemas:
- the writer's schema — exactly what the producer used, needed to parse the bytes;
- the reader's schema — what the consumer's code expects.
The library parses with the writer's schema and then maps fields by name to the reader's schema: fields in both are copied (with allowed type promotions such as int → long, float → double, string ↔ bytes); fields only in the writer are skipped; fields only in the reader are filled from the reader's default — and if there's no default, resolution fails.
flowchart LR
B["Bytes (no field tags)"] --> D["Decoder"]
WS["Writer schema v1: id int, name string"] --> D
RS["Reader schema v2: id long, name string, email null-or-string default null"] --> D
D --> R["Record: id=150, name=testing, email=null"]
Where does the reader get the writer's schema?
| Context | Mechanism |
|---|---|
| Object Container Files (Avro files in HDFS/S3) | The writer's schema is embedded once in the file header; many records follow |
| Kafka with a schema registry | Each message starts with a small header containing a schema ID; the consumer fetches the schema from the registry once and caches it |
| RPC | Schemas are exchanged in the handshake |
| Database rows | Store a schema version alongside the data |
The Confluent Schema Registry wire format, widely used with Kafka, prefixes each message with one magic byte (0) and a 4-byte schema ID, followed by the Avro payload. The registry also enforces a compatibility mode per subject before accepting a new schema: BACKWARD (the default: new schema can read data written with the previous one), FORWARD, FULL (both), their _TRANSITIVE variants (checked against all previous versions, not just the last), or NONE. See Event Schema Evolution for registry workflows.
sequenceDiagram
participant P as Producer
participant SR as Schema Registry
participant K as Kafka
participant C as Consumer
P->>SR: register schema v2 for subject orders-value
SR-->>P: compatible with v1, id 42
P->>K: magic 0, schema id 42, avro bytes
K->>C: message
C->>SR: fetch schema 42 (cached afterwards)
SR-->>C: writer schema v2
Note over C: resolve writer v2 against the consumer's reader schema
Code: schema resolution in practice (Python)
# avro_demo.py — pip install fastavro msgpack
import io, json
import msgpack
from fastavro import parse_schema, schemaless_writer, schemaless_reader
record = {"id": 150, "name": "testing"}
# --- Size of the same record in three encodings ---
print("JSON :", len(json.dumps(record, separators=(",", ":")).encode()), "bytes")
packed = msgpack.packb(record)
print("MessagePack:", len(packed), "bytes", packed.hex(" "))
writer_v1 = parse_schema({
"type": "record", "name": "Person", "namespace": "demo",
"fields": [{"name": "id", "type": "int"}, {"name": "name", "type": "string"}],
})
buf = io.BytesIO()
schemaless_writer(buf, writer_v1, record)
v1_bytes = buf.getvalue()
print("Avro :", len(v1_bytes), "bytes", v1_bytes.hex(" "))
# --- Schema evolution: reader v2 adds a field WITH a default (backward compatible) ---
reader_v2 = parse_schema({
"type": "record", "name": "Person", "namespace": "demo",
"fields": [
{"name": "id", "type": "long"}, # int -> long promotion is allowed
{"name": "name", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": None},
],
})
old_data_new_reader = schemaless_reader(io.BytesIO(v1_bytes), writer_v1, reader_v2)
print("v1 data read with v2 schema:", old_data_new_reader)
# --- A reader field with NO default and absent from the writer schema fails ---
reader_bad = parse_schema({
"type": "record", "name": "Person", "namespace": "demo",
"fields": [{"name": "id", "type": "int"}, {"name": "name", "type": "string"},
{"name": "country", "type": "string"}],
})
try:
schemaless_reader(io.BytesIO(v1_bytes), writer_v1, reader_bad)
raise AssertionError("expected schema resolution failure")
except Exception as e: # fastavro raises SchemaResolutionError
print("v1 data read with incompatible schema ->", type(e).__name__)
assert v1_bytes.hex() == "ac020e74657374696e67"
assert old_data_new_reader == {"id": 150, "name": "testing", "email": None}
print("self-check passed")
Output:
JSON : 27 bytes
MessagePack: 19 bytes 82 a2 69 64 cc 96 a4 6e 61 6d 65 a7 74 65 73 74 69 6e 67
Avro : 10 bytes ac 02 0e 74 65 73 74 69 6e 67
v1 data read with v2 schema: {'id': 150, 'name': 'testing', 'email': None}
v1 data read with incompatible schema -> SchemaResolutionError
self-check passed
Avro evolution rules
| Change | Backward (new reads old) | Forward (old reads new) | Notes |
|---|---|---|---|
| Add field with default | ✅ | ✅ | The standard safe change |
| Add field without default | ❌ | ✅ | New readers can't fill it from old data |
| Remove field that had a default | ✅ | ✅ | Old readers fill it from their default |
| Remove field without default | ✅ | ❌ | Old readers require it |
| Rename field | ⚠️ | ⚠️ | Use aliases in the reader schema; plain rename is remove + add |
Promote int → long / float / double, long → float/double, float → double | ✅ | ❌ | Old readers can't narrow |
string ↔ bytes | ✅ | ✅ | Allowed promotion both ways |
| Add a branch to a union | ✅ | ❌ | Old readers fail on the new branch |
| Add an enum symbol | ✅ | ⚠️ | Old readers fail unless the enum declares a default symbol |
| Change field order | ✅ | ✅ | Resolution is by name |
| Change a field's type incompatibly | ❌ | ❌ | — |
Two Avro-specific traps:
- Union defaults must match the first branch. A field
{"type": ["null", "string"], "default": null}is valid;{"type": ["string", "null"], "default": null}is not. Put"null"first for optional fields. - Names and namespaces matter. Records resolve by full name; renaming a record type or changing its namespace breaks resolution unless aliased.
Apache Thrift
Thrift was created at Facebook (open-sourced in 2007, now an Apache project). Like Protobuf, it uses an IDL with numbered fields and generates code; unlike Protobuf, it bundles a full RPC framework (transports, servers) and offers several wire protocols for the same IDL.
struct Person {
1: required i32 id,
2: optional string name,
3: optional string email,
}
Encoding the example
TBinaryProtocol is simple and fixed-width: each field has a 1-byte type, a 2-byte field ID, then the value (i32 as 4 bytes, strings as 4-byte length + bytes), and a stop byte ends the struct:
08 00 01 00 00 00 96 type i32 (8), id 1, value 150
0b 00 02 00 00 00 07 74 65 73 74 69 6e 67 type string (11), id 2, length 7, "testing"
00 stop
22 bytes. TCompactProtocol shrinks this: the field header packs a delta from the previous field ID with a 4-bit type into one byte, and integers are ZigZag varints:
15 delta 1, compact type i32 (5)
ac 02 zigzag varint 150
18 delta 1, compact type binary/string (8)
07 varint length 7
74 65 73 74 69 6e 67 "testing"
00 stop
13 bytes — essentially Protobuf-sized.
Thrift evolution rules
Thrift's rules mirror Protobuf's because the mechanics are the same (field IDs on the wire, unknown fields skipped by type):
| Change | Safe? |
|---|---|
Add an optional field with a new ID | ✅ |
Remove an optional field (never reuse its ID) | ✅ |
| Rename a field | ✅ on binary/compact protocols (names aren't on the wire); ❌ for the JSON protocol |
| Change a field ID or reuse one | ❌ |
| Change a field's type | ❌ (no varint-family compatibility as in Protobuf) |
Add a required field | ❌ — old writers won't send it, new readers reject |
Make a required field optional or remove it | ❌ — old readers reject messages without it |
The Thrift community's advice is "required is forever": once a field is required, you can never remove it without coordinated upgrades. Most teams simply never use required, the same conclusion that led proto3 to drop it.
Cap'n Proto and FlatBuffers: Zero-Copy Formats
Protobuf, Avro and Thrift all have a parse step: the decoder reads the bytes and builds language objects. For large messages, or when you only need a few fields, that step is wasted work. Zero-copy formats lay out the bytes so that the encoded buffer is the data structure — you read fields directly from the buffer (or a memory-mapped file) with pointer arithmetic.
flowchart LR
subgraph Parse["Parse-based (Protobuf, Avro, Thrift)"]
B1["Bytes"] -->|"decode every field"| O1["Language objects"]
O1 --> A1["Read field"]
end
subgraph ZeroCopy["Zero-copy (Cap'n Proto, FlatBuffers)"]
B2["Bytes / mmap"] -->|"offset arithmetic"| A2["Read field"]
end
Cap'n Proto
Created by Kenton Varda, who previously led Protocol Buffers v2 at Google. Messages are made of 8-byte words; each struct has a fixed-size data section (scalars at fixed offsets) and a pointer section (relative pointers to text, lists and sub-structs). Fields are numbered with ordinals (@0, @1), and new fields must take the next ordinal.
struct Person {
id @0 :UInt32;
name @1 :Text;
email @2 :Text; # added later: appended, never inserted
}
For our example the unpacked encoding is roughly: a segment table (8 bytes), a root pointer (8), one data word holding id (8), one pointer word for name (8), and the NUL-terminated text padded to a word (8) — about 40 bytes. Cap'n Proto offers an optional packing scheme that compresses runs of zero bytes, bringing the size much closer to Protobuf's while still being cheap to unpack. Its RPC system adds promise pipelining (calling methods on results that haven't arrived yet).
FlatBuffers
Created at Google by Wouter van Oortmerssen, originally for games, and later used for TensorFlow Lite's model format. A FlatBuffer table starts with an offset to a vtable that lists, for each field, its offset within the table (or 0 if absent). Reading a field is: look up the vtable slot, add the offset, read. Absent fields cost nothing in the table and read as their default.
table Person {
id: uint (id: 0);
name: string (id: 1);
email: string (id: 2); // add at the end; deprecate, never delete
}
root_type Person;
Our record comes to a few tens of bytes, depending on alignment and vtable sharing.
Zero-copy trade-offs
| Benefit | Cost | |
|---|---|---|
| Read latency | Field access without a parse pass; ideal for mmap'd files | — |
| Partial reads | Read 1 field of a 10 MB message cheaply | — |
| Size | — | Alignment padding and offsets make messages larger than Protobuf (unless packed/compressed) |
| Mutation | — | Building messages uses builder APIs; in-place modification is limited |
| Evolution | Similar to Protobuf: add fields, never reuse | Stricter: fields must be added at the end of the ordinal/ID sequence; types fixed |
| Validation | — | Readers of untrusted data must run the verifier (FlatBuffers) or rely on bounds-checked readers (Cap'n Proto), because offsets in malicious buffers could point anywhere |
| Change | Cap'n Proto | FlatBuffers |
|---|---|---|
| Add field at the end of the ordinal/ID sequence | ✅ | ✅ |
| Insert a field in the middle | ❌ (ordinals must be sequential) | ❌ without explicit id: attributes |
| Remove a field | ❌ (keep it; rename to mark obsolete) | ⚠️ mark deprecated — the slot stays reserved |
| Rename a field | ✅ (binary) | ✅ (binary); breaks JSON conversion |
| Change a field's type | ❌ | ❌ |
| Change a default value | ❌ (defaults are XOR-encoded into data) | ❌ (readers with different defaults disagree) |
Comparison
Size of the running example
| Format | Bytes for {id:150, name:"testing"} | Verified |
|---|---|---|
| JSON (minified) | 27 | Computed |
| MessagePack | 19 | Computed (script above) |
| Thrift Binary | 22 | Hand-encoded |
| Thrift Compact | 13 | Hand-encoded |
| Protobuf | 12 | Computed (Go program above) |
| Avro (no header) | 10 | Computed (script above) |
| Cap'n Proto (unpacked, with framing) | ~40 | Approximate |
| FlatBuffers | a few tens | Approximate |
Tiny records exaggerate the differences in fixed overhead. For realistic payloads, the gap between JSON and binary formats is typically a few times in size, and compression narrows it further.
Representative performance characteristics
These are representative relative characteristics, not measurements: real numbers depend heavily on the library, language, message shape and hardware. Benchmark your own payloads before deciding.
| Format | Relative size | Encode CPU | Decode CPU | Partial read | Schema required to read |
|---|---|---|---|---|---|
| JSON | Largest | High | High (number/string parsing) | No | No |
| MessagePack / CBOR | Medium | Medium | Medium | No | No |
| Protobuf | Small | Low | Low | No (parse whole message) | Yes, for meaning (structure is walkable) |
| Thrift Compact | Small | Low | Low | No | Yes, for meaning |
| Avro | Smallest | Low | Low | No | Yes — can't parse without the writer's schema |
| Cap'n Proto / FlatBuffers | Medium–large (smaller if packed) | Low | ~None (access in place) | Yes | Yes |
Evolution model at a glance
| Format | Field identity on the wire | Unknown fields | Defaults | Typical guard rail |
|---|---|---|---|---|
| JSON | Name | Ignored if reader is tolerant | Reader code | JSON Schema checks in CI |
| MessagePack | Name | Same as JSON | Reader code | Conventions |
| Protobuf | Field number | Skipped and preserved | Type defaults; optional for presence | reserved, buf breaking |
| Thrift | Field ID | Skipped | Per-field defaults in IDL | Never use required |
| Avro | None (position + writer schema) | Skipped via schema resolution | Reader-schema defaults | Schema registry compatibility mode |
| Cap'n Proto | Ordinal / offset | Ignored | Defaults in schema | Append-only ordinals |
| FlatBuffers | vtable slot | Ignored | Defaults in schema | Append-only, deprecated |
Failure Modes
The silent reinterpretation
The worst serialization bug isn't a crash — it's data that decodes successfully into the wrong meaning. Reusing a Protobuf field number, reordering an Avro schema's fields without the writer's schema (e.g. a consumer hardcoding the schema), or changing a JSON field from cents to dollars all produce plausible-looking garbage.
War Story (illustrative)
A team deleted a deprecated int64 legacy_discount = 7 field from an order message and, a few months later, added int64 loyalty_points = 7 without a reserved statement. Orders still in a retry queue from before the change were decoded by the new consumers with their old discount value now appearing as loyalty points. Nothing errored. The discrepancy surfaced weeks later in a finance reconciliation. The fix was mechanical — reserved 7; and a CI breaking-change check — but cleaning up the corrupted ledger was not.
Unsafe deserialization
Formats that can instantiate arbitrary types from the input — Java's native ObjectInputStream, Python's pickle, YAML loaders that construct arbitrary objects, .NET's BinaryFormatter — can execute code chosen by whoever crafted the bytes. They must never be used on untrusted input. Prefer data-only formats (JSON, Protobuf, Avro …), use safe loaders (yaml.safe_load), and treat any "deserialize to an arbitrary class" feature as a code-execution feature. The OWASP pages under Security cover this class of vulnerability.
Resource exhaustion
Even data-only decoders must defend against hostile input: a length prefix claiming 4 GB, deeply nested messages that blow the stack, or a JSON body with millions of keys. Set maximum message sizes (gRPC, for example, limits received messages to 4 MB by default), cap nesting depth, and stream large payloads instead of buffering.
Precision and type mismatches across languages
64-bit integers in JavaScript, unsigned integers in Java (which has none), decimal money values in floating point, timestamps with and without time zones — these are cross-language problems that the format exposes rather than causes. Use explicit types: Protobuf's google.protobuf.Timestamp, Avro's logical types (timestamp-millis, decimal), strings for 64-bit IDs in JSON.
Choosing a Format
flowchart TD
Q1{"Public API or browser clients?"} -->|"yes"| J["JSON (maybe with JSON Schema / OpenAPI)"]
Q1 -->|"no"| Q2{"Data at rest in big-data files or Kafka with a registry?"}
Q2 -->|"yes"| A["Avro (or Protobuf with a registry)"]
Q2 -->|"no"| Q3{"Need to read fields without parsing, e.g. mmap or huge messages?"}
Q3 -->|"yes"| Z["FlatBuffers or Cap'n Proto"]
Q3 -->|"no"| Q4{"Service-to-service RPC with codegen?"}
Q4 -->|"yes"| P["Protobuf with gRPC (or Thrift if already invested)"]
Q4 -->|"no"| M["MessagePack or CBOR for schemaless binary"]
| Use case | Good default | Why |
|---|---|---|
| Public REST API | JSON | Universal tooling, readable, browser-native |
| Internal RPC | Protobuf + gRPC | Compact, fast, strong evolution story, codegen everywhere |
| Kafka events, data lake | Avro + schema registry (or Protobuf + registry) | Smallest payloads, registry-enforced compatibility, schema-on-read |
| Mobile/game assets, ML models, mmap'd indexes | FlatBuffers | Zero-copy reads, small runtime |
| Low-latency RPC with very large messages | Cap'n Proto | No parse step, promise pipelining |
| Cache values in dynamic languages | MessagePack / CBOR | Schemaless, smaller and faster than JSON |
| Config files | JSON / YAML / TOML | Human editing matters more than size |
| Long-term archives | Avro container files or Parquet | Schema travels with the data |
Rules that hold regardless of format:
- Treat schemas as code: version them, review them, and run breaking-change checks in CI.
- Never reuse identifiers — field numbers, IDs, ordinals or names with a changed meaning.
- Add, don't change. New meaning → new field. Deprecate old fields; delete only when no data or code uses them, and reserve the identifier.
- Default to optional. Required fields are a compatibility trap in every format that has them.
- Know your deploy order. Backward compatibility lets you upgrade consumers first; forward compatibility lets you upgrade producers first. Event logs need both, transitively.
Key Takeaways
- Serialization must balance type fidelity, size, speed and evolution; evolution is the requirement that bites hardest over time.
- JSON is universal and readable but loosely typed (beware 64-bit integers) and verbose.
- Protobuf identifies fields by number; varints and ZigZag keep integers small; unknown fields are skipped and preserved. Never reuse numbers.
- Avro writes no field tags at all; decoding needs the writer's schema, resolved against the reader's by name, with defaults filling gaps. A schema registry makes that practical.
- Thrift is Protobuf-like with multiple protocols; avoid
required. - Cap'n Proto and FlatBuffers skip parsing entirely, trading size and stricter evolution for near-free reads.
- Defend decoders: size limits, depth limits, bounds checks, and no arbitrary-type deserialization of untrusted input.
Interview Questions
1. Walk through how Protobuf encodes `id = 150` in field 1.
The key is (1 << 3) | 0 = 0x08 (field 1, wire type VARINT). 150 is 10010110 in binary; split into 7-bit groups low-first: 0010110 with the continuation bit → 0x96, then 0000001 → 0x01. So the bytes are 08 96 01.
2. Why do negative int32 values take 10 bytes in Protobuf, and what's the fix?
int32 values are sign-extended to 64 bits before varint encoding, so −1 becomes 64 one-bits, which needs 10 seven-bit groups. Use sint32/sint64, which ZigZag-encode first (−1 → 1, 1 → 2, −2 → 3), so small negative numbers are 1 byte.
3. How can Avro be smaller than Protobuf, and what does that cost?
Avro writes no field tags or numbers — just values in schema order. The cost is that bytes can't be parsed (not even skipped) without the exact writer's schema. Systems solve this by embedding the schema in file headers or putting a schema ID in each message and fetching the schema from a registry.
4. Which changes are safe in a Protobuf schema? Which aren't?
Safe: adding fields with new numbers; removing fields while reserving their numbers and names; renaming on the binary wire (though it breaks JSON); some type changes within the varint family with truncation caveats. Unsafe: reusing or changing a field number, changing wire type (e.g. int32 → string), moving multiple existing fields into a oneof, and in proto2 adding/removing required fields.
5. Explain backward vs forward compatibility and which you need for a rolling deploy.
Backward: new readers can read old data. Forward: old readers can read new data. During a rolling deploy both versions run simultaneously, so data written by either can be read by either — you need both (full compatibility), unless you strictly sequence upgrades (consumers first requires only backward; producers first requires only forward). Replaying long-lived logs requires transitive compatibility across all historical versions.
6. Why did proto3 remove required fields?
A required field can never be safely removed or added: old readers reject messages missing it, and new readers reject old messages without it. In large systems with independent deploys this turned routine schema cleanups into outages. Making everything optional at the wire level and validating in application code is safer.
7. When would you choose FlatBuffers or Cap'n Proto over Protobuf?
When parse cost or partial reads dominate: memory-mapped data files, large messages where you read a few fields, latency-critical paths (games, ML model loading, high-frequency systems). You accept larger messages (unless packed), builder-style APIs, append-only evolution, and the need to verify untrusted buffers.
8. A JavaScript client shows the wrong user ID. The API returns `{"id": 1234567890123456789}`. Why?
JSON numbers are decoded as IEEE-754 doubles in JavaScript, which represent integers exactly only up to 2^53. The 64-bit ID is rounded. Return IDs as strings (or add an id_str field), or use a parser with BigInt support.
Further Reading
- Protocol Buffers documentation: Encoding and Updating A Message Type (protobuf.dev)
- Apache Avro specification: Schema Resolution and Binary Encoding
- Confluent documentation: Schema Evolution and Compatibility, Wire Format
- Apache Thrift: Thrift: Scalable Cross-Language Services Implementation (Slee, Agarwal, Kwiatkowski, Facebook, 2007)
- Cap'n Proto Encoding spec (capnproto.org) and FlatBuffers Internals (flatbuffers.dev)
- Martin Kleppmann, Designing Data-Intensive Applications, chapter 4 (Encoding and Evolution)
- Related pages: RPC Fundamentals, gRPC Deep Dive, gRPC Internals, Event Schema Evolution, API Versioning