RPC Fundamentals
A Remote Procedure Call (RPC) is a promise: call a function on another machine as if it were a local function. You write user := users.Get(ctx, 42), and somewhere a request is serialized, sent over the network, executed on a different process, and the result comes back as if nothing unusual happened.
That promise is one of the most useful ideas in distributed computing, and it is also a lie. This lesson covers both halves: how RPC frameworks build the illusion, and where the illusion breaks — because the breaks are exactly where production incidents come from.
You just finished REST Best Practices, which models an API as resources you manipulate with a small set of verbs. RPC models an API as procedures you call with arguments. Neither is "better"; they optimise for different things, and by the end of this page you should be able to pick one deliberately.
The Core Idea: Make a Remote Call Look Local
In 1984, Andrew Birrell and Bruce Nelson published Implementing Remote Procedure Calls, describing the system at Xerox PARC. Their goal was explicit: programmers already understood procedure calls, so if remote communication looked like a procedure call, distributed programming would become accessible to everyone.
Compare the two worlds:
// Local call: nanoseconds, cannot "half fail", shares memory with the caller.
price := pricing.Quote(cart)
// Remote call: milliseconds, can time out, can succeed on the server
// while the reply is lost, and receives a *copy* of cart.
price, err := pricingClient.Quote(ctx, &pb.QuoteRequest{Cart: cart})
They look nearly identical. That is the point — and the danger.
Why the Abstraction Leaks
In 1994, Jim Waldo, Geoff Wyant, Ann Wollrath and Sam Kendall at Sun Microsystems wrote A Note on Distributed Computing, arguing that the vision of "unified objects" — treating local and remote objects identically — was fundamentally flawed. Three decades later, their four points are still the checklist every RPC user should know.
| Difference | Local call | Remote call | Consequence |
|---|---|---|---|
| Latency | ~1–100 ns | ~0.1–100+ ms (10⁴–10⁶× slower) | A chatty interface that is fine locally (getName(), getEmail(), getAddress()) becomes unusable remotely. Remote APIs must be coarse-grained. |
| Memory access | Caller and callee share an address space; pointers work | No shared memory; everything is copied | You cannot pass a pointer, a file handle, or a lock. Mutating an argument on the server does nothing to the client's copy. |
| Partial failure | Either the whole process works or the whole process dies | The client, network, or server can fail independently | After a timeout, you do not know whether the operation ran. This is the big one. |
| Concurrency | Controlled by your program | Many independent clients hit the server at once | The server must handle interleaving calls; there is no global lock across machines. |
The partial failure point deserves emphasis. When a local function returns, you know what happened. When a remote call times out, there are three indistinguishable possibilities:
- The request never reached the server.
- The server received it, crashed mid-execution, and nothing was committed.
- The server executed it successfully, and the response was lost.
No amount of framework cleverness can tell these apart from the client side. Everything in the "failure semantics" section below is about living with that uncertainty.
The modern consensus
Good RPC frameworks no longer hide the network — they make it visible in the signature. In gRPC every call takes a context (deadline, cancellation) and returns an error with a status code. Code that looks like Quote(ctx, req) (resp, err) is honest about being remote. Code that looks like quote(cart) and throws a surprise RemoteException is not.
Anatomy of an RPC
Every RPC system, from Sun RPC in 1984 to gRPC today, has the same four moving parts.
sequenceDiagram
participant App as "Client code"
participant Stub as "Client stub"
participant Net as "Transport"
participant Skel as "Server skeleton / dispatcher"
participant Impl as "Service implementation"
App->>Stub: Quote(ctx, req)
Stub->>Stub: Marshal req to bytes, attach method name + deadline
Stub->>Net: Send frame
Net->>Skel: Deliver frame
Skel->>Skel: Look up method, unmarshal bytes
Skel->>Impl: Quote(ctx, req)
Impl-->>Skel: resp or error
Skel-->>Net: Marshal resp + status
Net-->>Stub: Deliver frame
Stub-->>App: resp, err
| Component | Job | Example in gRPC |
|---|---|---|
| Client stub (proxy) | Exposes a typed, local-looking method; hides marshalling and transport | Generated InventoryClient in Go, Java, Python… |
| Serialization / marshalling | Turns in-memory structures into bytes and back, in a language-neutral format | Protocol Buffers binary encoding |
| Transport | Moves bytes between processes; handles connections, framing, multiplexing | HTTP/2 streams over TCP (see gRPC Internals) |
| Server skeleton / dispatcher | Maps an incoming method name to a handler, unmarshals arguments, invokes it, marshals the result | Generated RegisterInventoryServer + the gRPC server's method router |
The stubs and skeletons are almost always generated code. You describe the interface once, and a compiler produces matching client and server code in every language you need. That description is the IDL.
IDLs and Schemas
An Interface Definition Language (IDL) is a language-neutral contract: which methods exist, what they take, what they return. It is the single source of truth that keeps a Go client and a Rust server agreeing on what byte 7 means.
syntax = "proto3";
package inventory.v1;
service Inventory {
rpc GetItem(GetItemRequest) returns (Item);
rpc ReserveItem(ReserveItemRequest) returns (Reservation);
}
message GetItemRequest {
string sku = 1;
}
message Item {
string sku = 1;
string name = 2;
int32 stock = 3;
}
The numbers (= 1, = 2) are field tags. On the wire, Protobuf does not send field names, only tags and values. That is why the encoding is compact — and why tags are sacred.
| IDL / schema | Origin | Wire format | Notable trait |
|---|---|---|---|
| Protocol Buffers | Google (open-sourced 2008) | Compact binary, tag-based | The default for gRPC and Connect; strong codegen everywhere |
| Thrift IDL | Facebook (2007) | Pluggable (binary, compact, JSON) | Bundles IDL, serialization and RPC framework in one project |
| Avro | Apache / Hadoop (2009) | Binary, no tags; schema travels with the data or via a registry | Reader and writer schemas are resolved at read time; dominant in Kafka pipelines |
| OpenAPI | Swagger (2011), now OpenAPI Initiative | JSON/YAML describing HTTP endpoints | Not an RPC IDL — it describes resource-oriented HTTP APIs. Useful contrast: it documents a REST API rather than defining procedures. See OpenAPI & Swagger. |
Schema Evolution Rules
Clients and servers are never deployed at the same instant. For minutes, hours, or (for mobile apps) years, old clients talk to new servers and vice versa. Your schema must be backward compatible (new code reads old data) and forward compatible (old code reads new data). For Protobuf, the rules are short and strict:
| Change | Safe? | Why |
|---|---|---|
| Add a new field with a new tag | Yes | Old readers skip unknown tags; new readers see the default value when it is missing |
| Remove a field | Yes, if you reserved its tag and name | Prevents someone reusing tag 3 later for a different meaning |
| Rename a field | Yes on the binary wire, risky for JSON | Binary uses tags only; JSON mapping uses names |
| Change a field's tag number | No | Every existing client now reads the wrong field |
Change a field's type (e.g. int32 to string) | No | Bytes are reinterpreted as garbage or fail to parse |
| Reuse a deleted field's tag | No | Old clients still sending tag 3 will be misread |
| Add a new RPC method | Yes | Old clients just never call it |
| Add a new enum value | Mostly | Old readers see an unknown value; make sure they handle it (always keep a _UNSPECIFIED = 0) |
message Item {
reserved 4; // was: string warehouse = 4;
reserved "warehouse";
string sku = 1;
string name = 2;
int32 stock = 3;
string location_id = 5; // new field, new tag
}
Proto3 has no "required"
Protobuf 3 removed required because a required field can never be safely removed — every old reader would reject messages without it. Treat every field as optional on the wire and validate presence in application code.
Failure Semantics
Because of partial failure, every RPC system must decide what happens when it does not know whether a call succeeded. There are three possible guarantees.
| Semantics | What the system does | Risk | Fits |
|---|---|---|---|
| At-most-once | Send once, never retry. On timeout, report failure. | The operation might have run anyway — or might not have. Never duplicated. | Non-idempotent operations where a duplicate is worse than a miss (and you can reconcile later) |
| At-least-once | Retry until you get a response. | The server may execute the operation two or more times. | Idempotent operations (GetItem, SetPrice(42, 9.99)) |
| Exactly-once | Execute precisely once despite failures. | Impossible to guarantee at the network level. | — |
Exactly-Once Is a Myth (At the Transport Level)
You cannot build exactly-once delivery across an unreliable network — it is the same reason the Two Generals' problem has no solution. What you can build is exactly-once effect: at-least-once delivery plus server-side deduplication. The client retries freely; the server remembers which requests it has already processed and returns the stored result instead of running the work again.
That memory is keyed by an idempotency key — a unique ID the client generates once per logical operation and sends with every retry of it. This is the same pattern Stripe uses for payments, covered in depth in Idempotency, and the same idea Kafka uses for exactly-once semantics.
sequenceDiagram
participant C as "Client"
participant S as "Server"
participant DB as "Dedup store"
C->>S: ReserveItem(key=abc-123, sku=X)
S->>DB: Seen abc-123?
DB-->>S: No
S->>S: Reserve stock, store result under abc-123
S--xC: Response lost in network
Note over C: Deadline hits, retry with SAME key
C->>S: ReserveItem(key=abc-123, sku=X)
S->>DB: Seen abc-123?
DB-->>S: Yes, reservation R-77
S-->>C: Reservation R-77 (no second reservation)
Timeouts and Deadline Propagation
A call without a timeout is a call that can hang forever, holding a thread, a connection, and a user's patience. But a per-hop timeout is not enough. Consider a request that flows gateway → orders → inventory → pricing, where each hop has a fixed 2-second timeout. If the user's request has a 1-second budget, the gateway gives up at 1 second — but pricing keeps working for up to 2 more seconds on an answer nobody will read.
The fix is a deadline: an absolute point in time ("finish by 12:00:01.000") that travels with the request. Each service passes the remaining budget downstream. gRPC does this automatically via the grpc-timeout header when you pass the incoming ctx to outgoing calls.
flowchart LR
U["User request<br/>budget 1000 ms"] --> G["Gateway<br/>spends 50 ms"]
G -->|"deadline: 950 ms left"| O["Orders<br/>spends 200 ms"]
O -->|"deadline: 750 ms left"| I["Inventory<br/>spends 100 ms"]
I -->|"deadline: 650 ms left"| P["Pricing"]
Retries and Cancellation
Retries turn transient failures into successes — and turn a struggling server into a dead one if done naively. The rules:
- Retry only idempotent operations, or non-idempotent ones carrying an idempotency key.
- Retry only retryable errors.
UNAVAILABLE(server down, connection reset) is retryable.INVALID_ARGUMENTorPERMISSION_DENIEDwill fail identically every time. - Use exponential backoff with jitter, so a thousand clients do not retry in lockstep.
- Cap attempts and respect the deadline. A retry that cannot finish before the deadline is wasted work.
- Retry at one layer only. If the gateway, the service and the client library each retry 3 times, one failure becomes 27 requests at the bottom of the stack.
Cancellation is the other half: when a caller gives up (deadline passed, user closed the tab), the cancellation should propagate so downstream servers stop working. In gRPC, cancelling the client context sends an HTTP/2 RST_STREAM, and the server's ctx.Done() fires. Handlers should check it before expensive work.
War Story (illustrative)
A checkout service called inventory with a 5-second timeout and 3 retries, while inventory called a database with its own 10-second timeout. During a database slowdown, each user request produced up to 4 inventory calls, each holding a database connection for 10 seconds after the user had already given up. The connection pool exhausted within a minute and the outage spread to every service sharing the database. The fix was boring: propagate the caller's deadline instead of fixed timeouts, retry only on UNAVAILABLE, and add a retry budget. The same slowdown a month later degraded checkout latency instead of taking the site down.
Sync, Async and Streaming RPC
"RPC" does not have to mean "block until a single reply arrives".
| Style | Shape | Example use |
|---|---|---|
| Unary (synchronous) | One request, one response | GetItem, CreateOrder |
| Asynchronous / future-based | Client fires the call and gets a future/promise; does other work meanwhile | Fan-out to 5 services in parallel, then join |
| Server streaming | One request, a stream of responses | Tail logs, stream search results, LLM token streaming |
| Client streaming | A stream of requests, one response | Upload telemetry batches, then get a summary |
| Bidirectional streaming | Both sides stream independently | Chat, collaborative editing, game state sync |
service Logs {
rpc Tail(TailRequest) returns (stream LogLine); // server streaming
rpc Ingest(stream LogLine) returns (IngestSummary); // client streaming
rpc Session(stream Command) returns (stream Event); // bidirectional
}
Streaming RPCs are long-lived, which changes the operational picture: load balancers must handle long connections, deadlines apply to the whole stream, and retries are only safe before the first message is received. The gRPC Deep Dive covers the four modes in detail.
Service Discovery and Load Balancing at a Glance
A stub needs an address. In a world of autoscaling containers, that address changes constantly, so RPC systems add two layers:
- Service discovery — resolving a logical name (
inventory.prod) to a live set of endpoints, via DNS, Consul, etcd, or the Kubernetes API. See Service Discovery. - Load balancing — choosing an endpoint per call. HTTP/2 multiplexes many RPCs over one long-lived connection, so a classic L4 (connection-level) load balancer pins all of a client's traffic to one backend. RPC systems therefore balance per request, either in the client (gRPC's
round_robinpolicy over resolved addresses) or in an L7 proxy or service mesh (Envoy, Linkerd) that understands individual streams.
The L4 load balancer trap
Putting gRPC behind a plain TCP load balancer is one of the most common gRPC mistakes. New backends receive no traffic because existing connections never move. Use client-side balancing, an L7 proxy, or set a maximum connection age so clients periodically reconnect.
A Short History of RPC
| Year | System | What it contributed |
|---|---|---|
| 1984 | Birrell & Nelson / Sun RPC (ONC RPC) | Stubs, binding, XDR serialization; NFS ran on Sun RPC |
| ~1990 | DCE/RPC | Enterprise RPC with IDL, security and naming; later the base of Microsoft's MS-RPC/DCOM |
| 1991 | CORBA | Language-neutral distributed objects via an ORB; powerful, notoriously complex |
| 1997 | Java RMI | Remote objects for Java-only systems; the "location transparency" Waldo et al. warned about |
| 1998–2000 | XML-RPC → SOAP | RPC over HTTP with XML envelopes and WSDL contracts; enterprise web services |
| 2007 | Apache Thrift | Facebook's cross-language IDL + serialization + RPC framework |
| 2015 | gRPC | Google's open-source successor to its internal Stubby; Protobuf over HTTP/2, deadlines, streaming |
| 2021 | tRPC | TypeScript-only RPC with no IDL — types flow from server to client through inference. See tRPC. |
| 2022 | Connect | Buf's protocol: gRPC-compatible, but also plain HTTP/1.1 + JSON callable from browsers and curl |
| 2005–today | JSON-RPC | Tiny spec (method, params, id) that quietly powers the Language Server Protocol, the Model Context Protocol (MCP), and Ethereum node APIs |
The arc is instructive. Early systems tried to make remote calls invisible; modern ones make them explicit but convenient: typed, generated, and carrying deadlines and status codes in every signature.
RPC vs REST vs Messaging
| Question | RPC (gRPC, Thrift, Connect) | REST (HTTP + JSON) | Messaging (Kafka, SQS, NATS) |
|---|---|---|---|
| Mental model | Call a procedure | Manipulate a resource | Publish an event or command |
| Coupling | Caller waits on a specific service | Caller waits on a specific service | Producer does not know who consumes |
| Contract | Strong IDL, codegen | Optional (OpenAPI), often informal | Schema registry, often Avro/Protobuf |
| Performance | High: binary, multiplexed | Moderate: text, HTTP/1.1 or 2 | Throughput-optimised; latency varies |
| Browser support | Needs gRPC-Web or Connect | Native | Not directly |
| Caching | Hard (POST-only for gRPC) | Native HTTP caching | N/A |
| Failure handling | Caller must handle timeout/retry | Caller must handle timeout/retry | Broker buffers; consumer retries; dead-letter queues |
| Best fit | Internal service-to-service calls, polyglot microservices, streaming | Public APIs, third-party developers, cacheable reads | Async workflows, fan-out, decoupling, load levelling |
A practical default for many companies: REST for public APIs, RPC for internal synchronous calls, messaging for anything that does not need an immediate answer. For a broader comparison including GraphQL and tRPC, see REST vs GraphQL vs gRPC vs tRPC.
A Minimal Working Example (Go + gRPC)
This example puts the whole lesson into ~120 lines: an IDL, a server that honours cancellation and deduplicates by idempotency key, and a client that sets deadlines and retries only what is safe.
1. The contract
// proto/inventory/v1/inventory.proto
syntax = "proto3";
package inventory.v1;
option go_package = "example.com/inventory/gen/inventory/v1;inventoryv1";
service Inventory {
// Idempotent: safe to retry.
rpc GetItem(GetItemRequest) returns (Item);
// Not naturally idempotent: made safe by idempotency_key.
rpc ReserveItem(ReserveItemRequest) returns (Reservation);
}
message GetItemRequest { string sku = 1; }
message Item {
string sku = 1;
string name = 2;
int32 stock = 3;
}
message ReserveItemRequest {
string idempotency_key = 1;
string sku = 2;
int32 quantity = 3;
}
message Reservation {
string reservation_id = 1;
string sku = 2;
int32 quantity = 3;
}
Generate code with protoc --go_out=. --go-grpc_out=. proto/inventory/v1/inventory.proto (or buf generate).
2. The server
// server/main.go
package main
import (
"context"
"fmt"
"log"
"net"
"sync"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
pb "example.com/inventory/gen/inventory/v1"
)
type server struct {
pb.UnimplementedInventoryServer
mu sync.Mutex
stock map[string]int32
seen map[string]*pb.Reservation // idempotency key -> result
next int
}
func (s *server) GetItem(ctx context.Context, req *pb.GetItemRequest) (*pb.Item, error) {
s.mu.Lock()
defer s.mu.Unlock()
n, ok := s.stock[req.GetSku()]
if !ok {
return nil, status.Errorf(codes.NotFound, "sku %q not found", req.GetSku())
}
return &pb.Item{Sku: req.GetSku(), Name: "Widget", Stock: n}, nil
}
func (s *server) ReserveItem(ctx context.Context, req *pb.ReserveItemRequest) (*pb.Reservation, error) {
if req.GetIdempotencyKey() == "" {
return nil, status.Error(codes.InvalidArgument, "idempotency_key is required")
}
// The caller may already have given up; don't do work nobody will read.
if err := ctx.Err(); err != nil {
return nil, status.FromContextError(err).Err()
}
s.mu.Lock()
defer s.mu.Unlock()
// Retry of an operation we already performed: return the stored result.
if r, ok := s.seen[req.GetIdempotencyKey()]; ok {
return r, nil
}
if s.stock[req.GetSku()] < req.GetQuantity() {
return nil, status.Error(codes.FailedPrecondition, "insufficient stock")
}
s.stock[req.GetSku()] -= req.GetQuantity()
s.next++
r := &pb.Reservation{
ReservationId: fmt.Sprintf("R-%d", s.next),
Sku: req.GetSku(),
Quantity: req.GetQuantity(),
}
// In production this must be written in the same DB transaction
// as the stock decrement, with a TTL.
s.seen[req.GetIdempotencyKey()] = r
return r, nil
}
func main() {
lis, err := net.Listen("tcp", ":50051")
if err != nil {
log.Fatal(err)
}
g := grpc.NewServer()
pb.RegisterInventoryServer(g, &server{
stock: map[string]int32{"WIDGET-1": 10},
seen: map[string]*pb.Reservation{},
})
log.Println("inventory listening on :50051")
log.Fatal(g.Serve(lis))
}
3. The client
// client/main.go
package main
import (
"context"
"log"
"time"
"github.com/google/uuid"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
pb "example.com/inventory/gen/inventory/v1"
)
// Transparent retries for GetItem only, and only on UNAVAILABLE.
const serviceConfig = `{
"methodConfig": [{
"name": [{"service": "inventory.v1.Inventory", "method": "GetItem"}],
"retryPolicy": {
"maxAttempts": 4,
"initialBackoff": "0.1s",
"maxBackoff": "1s",
"backoffMultiplier": 2,
"retryableStatusCodes": ["UNAVAILABLE"]
}
}]
}`
func main() {
conn, err := grpc.NewClient("localhost:50051",
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithDefaultServiceConfig(serviceConfig),
)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
client := pb.NewInventoryClient(conn)
// Deadline for the whole operation; retries must fit inside it.
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
item, err := client.GetItem(ctx, &pb.GetItemRequest{Sku: "WIDGET-1"})
if err != nil {
log.Fatalf("GetItem: %v", err)
}
log.Printf("stock=%d", item.GetStock())
// One key per logical operation, reused on every retry of it.
req := &pb.ReserveItemRequest{
IdempotencyKey: uuid.NewString(),
Sku: "WIDGET-1",
Quantity: 2,
}
var res *pb.Reservation
for attempt := 1; attempt <= 3; attempt++ {
callCtx, callCancel := context.WithTimeout(ctx, 500*time.Millisecond)
res, err = client.ReserveItem(callCtx, req)
callCancel()
if err == nil {
break
}
log.Printf("attempt %d failed: %v (retrying with same key)", attempt, err)
time.Sleep(time.Duration(attempt) * 100 * time.Millisecond)
}
if err != nil {
log.Fatalf("ReserveItem: %v", err)
}
log.Printf("reserved %s", res.GetReservationId())
}
What the example demonstrates:
- Deadlines: the outer 2-second context bounds everything; each
ReserveItemattempt gets a shorter per-attempt timeout derived from it, so retries can never exceed the overall budget. - Safe retries:
GetItemis retried transparently by gRPC's service config, because reading is idempotent.ReserveItemis retried manually, and it is safe only because the same idempotency key is reused. - Cancellation: the server checks
ctx.Err()before doing work, so abandoned calls are cheap. - Status codes:
NotFound,InvalidArgumentandFailedPreconditiontell the client not to retry. A production client would inspectstatus.Code(err)and only loop onUnavailableorDeadlineExceeded.
Want the TypeScript version?
Connect generates TypeScript clients from the same .proto file and speaks gRPC, gRPC-Web and plain JSON over HTTP. If both ends are TypeScript and you control them, tRPC removes the IDL entirely.
Key Takeaways
- RPC makes a remote call look local, but latency, separate memory and partial failure make it fundamentally different. Design coarse-grained interfaces and put
ctxanderrin every signature. - Every RPC has a stub, a serializer, a transport and a dispatcher; IDLs generate the first and last.
- Schema evolution is about field tags: add with new tags,
reserveremoved ones, never change or reuse them. - Exactly-once delivery is impossible; exactly-once effect is at-least-once delivery plus idempotency keys.
- Propagate deadlines, not fixed timeouts. Retry only idempotent calls, only on retryable codes, with backoff, at one layer.
- Choose RPC for internal synchronous calls, REST for public and cacheable APIs, messaging for asynchronous decoupling.
Further Reading
- gRPC Deep Dive — the four streaming modes, interceptors, error model and production setup
- gRPC Internals — how gRPC maps onto HTTP/2 frames, headers and trailers
- tRPC — end-to-end type-safe RPC for TypeScript without an IDL
- Idempotency — idempotency keys, storage and race conditions in depth
- REST vs GraphQL vs gRPC vs tRPC — side-by-side comparison of API styles
- REST Best Practices — the resource-oriented alternative
- Service Discovery — how stubs find live endpoints
- Waldo, Wyant, Wollrath, Kendall — A Note on Distributed Computing (Sun Microsystems, 1994)
- Birrell & Nelson — Implementing Remote Procedure Calls (ACM TOCS, 1984)