On this page

RPC Fundamentals

A Remote Procedure Call (RPC) is a promise: call a function on another machine as if it were a local function. You write user := users.Get(ctx, 42), and somewhere a request is serialized, sent over the network, executed on a different process, and the result comes back as if nothing unusual happened.

That promise is one of the most useful ideas in distributed computing, and it is also a lie. This lesson covers both halves: how RPC frameworks build the illusion, and where the illusion breaks — because the breaks are exactly where production incidents come from.

You just finished REST Best Practices, which models an API as resources you manipulate with a small set of verbs. RPC models an API as procedures you call with arguments. Neither is "better"; they optimise for different things, and by the end of this page you should be able to pick one deliberately.

The Core Idea: Make a Remote Call Look Local

In 1984, Andrew Birrell and Bruce Nelson published Implementing Remote Procedure Calls, describing the system at Xerox PARC. Their goal was explicit: programmers already understood procedure calls, so if remote communication looked like a procedure call, distributed programming would become accessible to everyone.

Compare the two worlds:

// Local call: nanoseconds, cannot "half fail", shares memory with the caller.
price := pricing.Quote(cart)

// Remote call: milliseconds, can time out, can succeed on the server
// while the reply is lost, and receives a *copy* of cart.
price, err := pricingClient.Quote(ctx, &pb.QuoteRequest{Cart: cart})

They look nearly identical. That is the point — and the danger.

Why the Abstraction Leaks

In 1994, Jim Waldo, Geoff Wyant, Ann Wollrath and Sam Kendall at Sun Microsystems wrote A Note on Distributed Computing, arguing that the vision of "unified objects" — treating local and remote objects identically — was fundamentally flawed. Three decades later, their four points are still the checklist every RPC user should know.

DifferenceLocal callRemote callConsequence
Latency~1–100 ns~0.1–100+ ms (10⁴–10⁶× slower)A chatty interface that is fine locally (getName(), getEmail(), getAddress()) becomes unusable remotely. Remote APIs must be coarse-grained.
Memory accessCaller and callee share an address space; pointers workNo shared memory; everything is copiedYou cannot pass a pointer, a file handle, or a lock. Mutating an argument on the server does nothing to the client's copy.
Partial failureEither the whole process works or the whole process diesThe client, network, or server can fail independentlyAfter a timeout, you do not know whether the operation ran. This is the big one.
ConcurrencyControlled by your programMany independent clients hit the server at onceThe server must handle interleaving calls; there is no global lock across machines.

The partial failure point deserves emphasis. When a local function returns, you know what happened. When a remote call times out, there are three indistinguishable possibilities:

  1. The request never reached the server.
  2. The server received it, crashed mid-execution, and nothing was committed.
  3. The server executed it successfully, and the response was lost.

No amount of framework cleverness can tell these apart from the client side. Everything in the "failure semantics" section below is about living with that uncertainty.

The modern consensus

Good RPC frameworks no longer hide the network — they make it visible in the signature. In gRPC every call takes a context (deadline, cancellation) and returns an error with a status code. Code that looks like Quote(ctx, req) (resp, err) is honest about being remote. Code that looks like quote(cart) and throws a surprise RemoteException is not.

Anatomy of an RPC

Every RPC system, from Sun RPC in 1984 to gRPC today, has the same four moving parts.

sequenceDiagram
    participant App as "Client code"
    participant Stub as "Client stub"
    participant Net as "Transport"
    participant Skel as "Server skeleton / dispatcher"
    participant Impl as "Service implementation"

    App->>Stub: Quote(ctx, req)
    Stub->>Stub: Marshal req to bytes, attach method name + deadline
    Stub->>Net: Send frame
    Net->>Skel: Deliver frame
    Skel->>Skel: Look up method, unmarshal bytes
    Skel->>Impl: Quote(ctx, req)
    Impl-->>Skel: resp or error
    Skel-->>Net: Marshal resp + status
    Net-->>Stub: Deliver frame
    Stub-->>App: resp, err
ComponentJobExample in gRPC
Client stub (proxy)Exposes a typed, local-looking method; hides marshalling and transportGenerated InventoryClient in Go, Java, Python…
Serialization / marshallingTurns in-memory structures into bytes and back, in a language-neutral formatProtocol Buffers binary encoding
TransportMoves bytes between processes; handles connections, framing, multiplexingHTTP/2 streams over TCP (see gRPC Internals)
Server skeleton / dispatcherMaps an incoming method name to a handler, unmarshals arguments, invokes it, marshals the resultGenerated RegisterInventoryServer + the gRPC server's method router

The stubs and skeletons are almost always generated code. You describe the interface once, and a compiler produces matching client and server code in every language you need. That description is the IDL.

IDLs and Schemas

An Interface Definition Language (IDL) is a language-neutral contract: which methods exist, what they take, what they return. It is the single source of truth that keeps a Go client and a Rust server agreeing on what byte 7 means.

syntax = "proto3";

package inventory.v1;

service Inventory {
  rpc GetItem(GetItemRequest) returns (Item);
  rpc ReserveItem(ReserveItemRequest) returns (Reservation);
}

message GetItemRequest {
  string sku = 1;
}

message Item {
  string sku = 1;
  string name = 2;
  int32 stock = 3;
}

The numbers (= 1, = 2) are field tags. On the wire, Protobuf does not send field names, only tags and values. That is why the encoding is compact — and why tags are sacred.

IDL / schemaOriginWire formatNotable trait
Protocol BuffersGoogle (open-sourced 2008)Compact binary, tag-basedThe default for gRPC and Connect; strong codegen everywhere
Thrift IDLFacebook (2007)Pluggable (binary, compact, JSON)Bundles IDL, serialization and RPC framework in one project
AvroApache / Hadoop (2009)Binary, no tags; schema travels with the data or via a registryReader and writer schemas are resolved at read time; dominant in Kafka pipelines
OpenAPISwagger (2011), now OpenAPI InitiativeJSON/YAML describing HTTP endpointsNot an RPC IDL — it describes resource-oriented HTTP APIs. Useful contrast: it documents a REST API rather than defining procedures. See OpenAPI & Swagger.

Schema Evolution Rules

Clients and servers are never deployed at the same instant. For minutes, hours, or (for mobile apps) years, old clients talk to new servers and vice versa. Your schema must be backward compatible (new code reads old data) and forward compatible (old code reads new data). For Protobuf, the rules are short and strict:

ChangeSafe?Why
Add a new field with a new tagYesOld readers skip unknown tags; new readers see the default value when it is missing
Remove a fieldYes, if you reserved its tag and namePrevents someone reusing tag 3 later for a different meaning
Rename a fieldYes on the binary wire, risky for JSONBinary uses tags only; JSON mapping uses names
Change a field's tag numberNoEvery existing client now reads the wrong field
Change a field's type (e.g. int32 to string)NoBytes are reinterpreted as garbage or fail to parse
Reuse a deleted field's tagNoOld clients still sending tag 3 will be misread
Add a new RPC methodYesOld clients just never call it
Add a new enum valueMostlyOld readers see an unknown value; make sure they handle it (always keep a _UNSPECIFIED = 0)
message Item {
  reserved 4;                  // was: string warehouse = 4;
  reserved "warehouse";
  string sku = 1;
  string name = 2;
  int32 stock = 3;
  string location_id = 5;      // new field, new tag
}

Proto3 has no "required"

Protobuf 3 removed required because a required field can never be safely removed — every old reader would reject messages without it. Treat every field as optional on the wire and validate presence in application code.

Failure Semantics

Because of partial failure, every RPC system must decide what happens when it does not know whether a call succeeded. There are three possible guarantees.

SemanticsWhat the system doesRiskFits
At-most-onceSend once, never retry. On timeout, report failure.The operation might have run anyway — or might not have. Never duplicated.Non-idempotent operations where a duplicate is worse than a miss (and you can reconcile later)
At-least-onceRetry until you get a response.The server may execute the operation two or more times.Idempotent operations (GetItem, SetPrice(42, 9.99))
Exactly-onceExecute precisely once despite failures.Impossible to guarantee at the network level.—

Exactly-Once Is a Myth (At the Transport Level)

You cannot build exactly-once delivery across an unreliable network — it is the same reason the Two Generals' problem has no solution. What you can build is exactly-once effect: at-least-once delivery plus server-side deduplication. The client retries freely; the server remembers which requests it has already processed and returns the stored result instead of running the work again.

That memory is keyed by an idempotency key — a unique ID the client generates once per logical operation and sends with every retry of it. This is the same pattern Stripe uses for payments, covered in depth in Idempotency, and the same idea Kafka uses for exactly-once semantics.

sequenceDiagram
    participant C as "Client"
    participant S as "Server"
    participant DB as "Dedup store"

    C->>S: ReserveItem(key=abc-123, sku=X)
    S->>DB: Seen abc-123?
    DB-->>S: No
    S->>S: Reserve stock, store result under abc-123
    S--xC: Response lost in network
    Note over C: Deadline hits, retry with SAME key
    C->>S: ReserveItem(key=abc-123, sku=X)
    S->>DB: Seen abc-123?
    DB-->>S: Yes, reservation R-77
    S-->>C: Reservation R-77 (no second reservation)

Timeouts and Deadline Propagation

A call without a timeout is a call that can hang forever, holding a thread, a connection, and a user's patience. But a per-hop timeout is not enough. Consider a request that flows gateway → orders → inventory → pricing, where each hop has a fixed 2-second timeout. If the user's request has a 1-second budget, the gateway gives up at 1 second — but pricing keeps working for up to 2 more seconds on an answer nobody will read.

The fix is a deadline: an absolute point in time ("finish by 12:00:01.000") that travels with the request. Each service passes the remaining budget downstream. gRPC does this automatically via the grpc-timeout header when you pass the incoming ctx to outgoing calls.

flowchart LR
    U["User request<br/>budget 1000 ms"] --> G["Gateway<br/>spends 50 ms"]
    G -->|"deadline: 950 ms left"| O["Orders<br/>spends 200 ms"]
    O -->|"deadline: 750 ms left"| I["Inventory<br/>spends 100 ms"]
    I -->|"deadline: 650 ms left"| P["Pricing"]

Retries and Cancellation

Retries turn transient failures into successes — and turn a struggling server into a dead one if done naively. The rules:

  • Retry only idempotent operations, or non-idempotent ones carrying an idempotency key.
  • Retry only retryable errors. UNAVAILABLE (server down, connection reset) is retryable. INVALID_ARGUMENT or PERMISSION_DENIED will fail identically every time.
  • Use exponential backoff with jitter, so a thousand clients do not retry in lockstep.
  • Cap attempts and respect the deadline. A retry that cannot finish before the deadline is wasted work.
  • Retry at one layer only. If the gateway, the service and the client library each retry 3 times, one failure becomes 27 requests at the bottom of the stack.

Cancellation is the other half: when a caller gives up (deadline passed, user closed the tab), the cancellation should propagate so downstream servers stop working. In gRPC, cancelling the client context sends an HTTP/2 RST_STREAM, and the server's ctx.Done() fires. Handlers should check it before expensive work.

War Story (illustrative)

A checkout service called inventory with a 5-second timeout and 3 retries, while inventory called a database with its own 10-second timeout. During a database slowdown, each user request produced up to 4 inventory calls, each holding a database connection for 10 seconds after the user had already given up. The connection pool exhausted within a minute and the outage spread to every service sharing the database. The fix was boring: propagate the caller's deadline instead of fixed timeouts, retry only on UNAVAILABLE, and add a retry budget. The same slowdown a month later degraded checkout latency instead of taking the site down.

Sync, Async and Streaming RPC

"RPC" does not have to mean "block until a single reply arrives".

StyleShapeExample use
Unary (synchronous)One request, one responseGetItem, CreateOrder
Asynchronous / future-basedClient fires the call and gets a future/promise; does other work meanwhileFan-out to 5 services in parallel, then join
Server streamingOne request, a stream of responsesTail logs, stream search results, LLM token streaming
Client streamingA stream of requests, one responseUpload telemetry batches, then get a summary
Bidirectional streamingBoth sides stream independentlyChat, collaborative editing, game state sync
service Logs {
  rpc Tail(TailRequest) returns (stream LogLine);          // server streaming
  rpc Ingest(stream LogLine) returns (IngestSummary);      // client streaming
  rpc Session(stream Command) returns (stream Event);      // bidirectional
}

Streaming RPCs are long-lived, which changes the operational picture: load balancers must handle long connections, deadlines apply to the whole stream, and retries are only safe before the first message is received. The gRPC Deep Dive covers the four modes in detail.

Service Discovery and Load Balancing at a Glance

A stub needs an address. In a world of autoscaling containers, that address changes constantly, so RPC systems add two layers:

  • Service discovery — resolving a logical name (inventory.prod) to a live set of endpoints, via DNS, Consul, etcd, or the Kubernetes API. See Service Discovery.
  • Load balancing — choosing an endpoint per call. HTTP/2 multiplexes many RPCs over one long-lived connection, so a classic L4 (connection-level) load balancer pins all of a client's traffic to one backend. RPC systems therefore balance per request, either in the client (gRPC's round_robin policy over resolved addresses) or in an L7 proxy or service mesh (Envoy, Linkerd) that understands individual streams.

The L4 load balancer trap

Putting gRPC behind a plain TCP load balancer is one of the most common gRPC mistakes. New backends receive no traffic because existing connections never move. Use client-side balancing, an L7 proxy, or set a maximum connection age so clients periodically reconnect.

A Short History of RPC

YearSystemWhat it contributed
1984Birrell & Nelson / Sun RPC (ONC RPC)Stubs, binding, XDR serialization; NFS ran on Sun RPC
~1990DCE/RPCEnterprise RPC with IDL, security and naming; later the base of Microsoft's MS-RPC/DCOM
1991CORBALanguage-neutral distributed objects via an ORB; powerful, notoriously complex
1997Java RMIRemote objects for Java-only systems; the "location transparency" Waldo et al. warned about
1998–2000XML-RPC → SOAPRPC over HTTP with XML envelopes and WSDL contracts; enterprise web services
2007Apache ThriftFacebook's cross-language IDL + serialization + RPC framework
2015gRPCGoogle's open-source successor to its internal Stubby; Protobuf over HTTP/2, deadlines, streaming
2021tRPCTypeScript-only RPC with no IDL — types flow from server to client through inference. See tRPC.
2022ConnectBuf's protocol: gRPC-compatible, but also plain HTTP/1.1 + JSON callable from browsers and curl
2005–todayJSON-RPCTiny spec (method, params, id) that quietly powers the Language Server Protocol, the Model Context Protocol (MCP), and Ethereum node APIs

The arc is instructive. Early systems tried to make remote calls invisible; modern ones make them explicit but convenient: typed, generated, and carrying deadlines and status codes in every signature.

RPC vs REST vs Messaging

QuestionRPC (gRPC, Thrift, Connect)REST (HTTP + JSON)Messaging (Kafka, SQS, NATS)
Mental modelCall a procedureManipulate a resourcePublish an event or command
CouplingCaller waits on a specific serviceCaller waits on a specific serviceProducer does not know who consumes
ContractStrong IDL, codegenOptional (OpenAPI), often informalSchema registry, often Avro/Protobuf
PerformanceHigh: binary, multiplexedModerate: text, HTTP/1.1 or 2Throughput-optimised; latency varies
Browser supportNeeds gRPC-Web or ConnectNativeNot directly
CachingHard (POST-only for gRPC)Native HTTP cachingN/A
Failure handlingCaller must handle timeout/retryCaller must handle timeout/retryBroker buffers; consumer retries; dead-letter queues
Best fitInternal service-to-service calls, polyglot microservices, streamingPublic APIs, third-party developers, cacheable readsAsync workflows, fan-out, decoupling, load levelling

A practical default for many companies: REST for public APIs, RPC for internal synchronous calls, messaging for anything that does not need an immediate answer. For a broader comparison including GraphQL and tRPC, see REST vs GraphQL vs gRPC vs tRPC.

A Minimal Working Example (Go + gRPC)

This example puts the whole lesson into ~120 lines: an IDL, a server that honours cancellation and deduplicates by idempotency key, and a client that sets deadlines and retries only what is safe.

1. The contract

// proto/inventory/v1/inventory.proto
syntax = "proto3";

package inventory.v1;

option go_package = "example.com/inventory/gen/inventory/v1;inventoryv1";

service Inventory {
  // Idempotent: safe to retry.
  rpc GetItem(GetItemRequest) returns (Item);
  // Not naturally idempotent: made safe by idempotency_key.
  rpc ReserveItem(ReserveItemRequest) returns (Reservation);
}

message GetItemRequest { string sku = 1; }

message Item {
  string sku = 1;
  string name = 2;
  int32 stock = 3;
}

message ReserveItemRequest {
  string idempotency_key = 1;
  string sku = 2;
  int32 quantity = 3;
}

message Reservation {
  string reservation_id = 1;
  string sku = 2;
  int32 quantity = 3;
}

Generate code with protoc --go_out=. --go-grpc_out=. proto/inventory/v1/inventory.proto (or buf generate).

2. The server

// server/main.go
package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"sync"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	pb "example.com/inventory/gen/inventory/v1"
)

type server struct {
	pb.UnimplementedInventoryServer

	mu    sync.Mutex
	stock map[string]int32
	seen  map[string]*pb.Reservation // idempotency key -> result
	next  int
}

func (s *server) GetItem(ctx context.Context, req *pb.GetItemRequest) (*pb.Item, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	n, ok := s.stock[req.GetSku()]
	if !ok {
		return nil, status.Errorf(codes.NotFound, "sku %q not found", req.GetSku())
	}
	return &pb.Item{Sku: req.GetSku(), Name: "Widget", Stock: n}, nil
}

func (s *server) ReserveItem(ctx context.Context, req *pb.ReserveItemRequest) (*pb.Reservation, error) {
	if req.GetIdempotencyKey() == "" {
		return nil, status.Error(codes.InvalidArgument, "idempotency_key is required")
	}
	// The caller may already have given up; don't do work nobody will read.
	if err := ctx.Err(); err != nil {
		return nil, status.FromContextError(err).Err()
	}

	s.mu.Lock()
	defer s.mu.Unlock()

	// Retry of an operation we already performed: return the stored result.
	if r, ok := s.seen[req.GetIdempotencyKey()]; ok {
		return r, nil
	}
	if s.stock[req.GetSku()] < req.GetQuantity() {
		return nil, status.Error(codes.FailedPrecondition, "insufficient stock")
	}
	s.stock[req.GetSku()] -= req.GetQuantity()
	s.next++
	r := &pb.Reservation{
		ReservationId: fmt.Sprintf("R-%d", s.next),
		Sku:           req.GetSku(),
		Quantity:      req.GetQuantity(),
	}
	// In production this must be written in the same DB transaction
	// as the stock decrement, with a TTL.
	s.seen[req.GetIdempotencyKey()] = r
	return r, nil
}

func main() {
	lis, err := net.Listen("tcp", ":50051")
	if err != nil {
		log.Fatal(err)
	}
	g := grpc.NewServer()
	pb.RegisterInventoryServer(g, &server{
		stock: map[string]int32{"WIDGET-1": 10},
		seen:  map[string]*pb.Reservation{},
	})
	log.Println("inventory listening on :50051")
	log.Fatal(g.Serve(lis))
}

3. The client

// client/main.go
package main

import (
	"context"
	"log"
	"time"

	"github.com/google/uuid"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	pb "example.com/inventory/gen/inventory/v1"
)

// Transparent retries for GetItem only, and only on UNAVAILABLE.
const serviceConfig = `{
  "methodConfig": [{
    "name": [{"service": "inventory.v1.Inventory", "method": "GetItem"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}`

func main() {
	conn, err := grpc.NewClient("localhost:50051",
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithDefaultServiceConfig(serviceConfig),
	)
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()
	client := pb.NewInventoryClient(conn)

	// Deadline for the whole operation; retries must fit inside it.
	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
	defer cancel()

	item, err := client.GetItem(ctx, &pb.GetItemRequest{Sku: "WIDGET-1"})
	if err != nil {
		log.Fatalf("GetItem: %v", err)
	}
	log.Printf("stock=%d", item.GetStock())

	// One key per logical operation, reused on every retry of it.
	req := &pb.ReserveItemRequest{
		IdempotencyKey: uuid.NewString(),
		Sku:            "WIDGET-1",
		Quantity:       2,
	}
	var res *pb.Reservation
	for attempt := 1; attempt <= 3; attempt++ {
		callCtx, callCancel := context.WithTimeout(ctx, 500*time.Millisecond)
		res, err = client.ReserveItem(callCtx, req)
		callCancel()
		if err == nil {
			break
		}
		log.Printf("attempt %d failed: %v (retrying with same key)", attempt, err)
		time.Sleep(time.Duration(attempt) * 100 * time.Millisecond)
	}
	if err != nil {
		log.Fatalf("ReserveItem: %v", err)
	}
	log.Printf("reserved %s", res.GetReservationId())
}

What the example demonstrates:

  • Deadlines: the outer 2-second context bounds everything; each ReserveItem attempt gets a shorter per-attempt timeout derived from it, so retries can never exceed the overall budget.
  • Safe retries: GetItem is retried transparently by gRPC's service config, because reading is idempotent. ReserveItem is retried manually, and it is safe only because the same idempotency key is reused.
  • Cancellation: the server checks ctx.Err() before doing work, so abandoned calls are cheap.
  • Status codes: NotFound, InvalidArgument and FailedPrecondition tell the client not to retry. A production client would inspect status.Code(err) and only loop on Unavailable or DeadlineExceeded.

Want the TypeScript version?

Connect generates TypeScript clients from the same .proto file and speaks gRPC, gRPC-Web and plain JSON over HTTP. If both ends are TypeScript and you control them, tRPC removes the IDL entirely.

Key Takeaways

  • RPC makes a remote call look local, but latency, separate memory and partial failure make it fundamentally different. Design coarse-grained interfaces and put ctx and err in every signature.
  • Every RPC has a stub, a serializer, a transport and a dispatcher; IDLs generate the first and last.
  • Schema evolution is about field tags: add with new tags, reserve removed ones, never change or reuse them.
  • Exactly-once delivery is impossible; exactly-once effect is at-least-once delivery plus idempotency keys.
  • Propagate deadlines, not fixed timeouts. Retry only idempotent calls, only on retryable codes, with backoff, at one layer.
  • Choose RPC for internal synchronous calls, REST for public and cacheable APIs, messaging for asynchronous decoupling.

Further Reading

  • gRPC Deep Dive — the four streaming modes, interceptors, error model and production setup
  • gRPC Internals — how gRPC maps onto HTTP/2 frames, headers and trailers
  • tRPC — end-to-end type-safe RPC for TypeScript without an IDL
  • Idempotency — idempotency keys, storage and race conditions in depth
  • REST vs GraphQL vs gRPC vs tRPC — side-by-side comparison of API styles
  • REST Best Practices — the resource-oriented alternative
  • Service Discovery — how stubs find live endpoints
  • Waldo, Wyant, Wollrath, Kendall — A Note on Distributed Computing (Sun Microsystems, 1994)
  • Birrell & Nelson — Implementing Remote Procedure Calls (ACM TOCS, 1984)