BGP — Border Gateway Protocol
BGP is the protocol that makes the internet work. Every major ISP, cloud provider, CDN, and large enterprise runs BGP. It's the system by which networks tell each other "I can reach these IP addresses — route traffic to me."
Understanding BGP explains: why Cloudflare can absorb DDoS attacks globally, how anycast CDNs work, why an ISP "going dark" can disconnect millions of users, and how a single misconfigured router can accidentally route half the internet's traffic through Pakistan.
Correction (2026-11-13)
The 2008 Pakistan Telecom incident did not route "half the internet's traffic" through Pakistan. It diverted traffic for YouTube (one /24 prefix) from much of the world into Pakistan Telecom's network, where it was dropped, for about two hours. The full story is in Real incidents below. The broader point stands: one wrong announcement from one router can break reachability for the whole planet.
What this lesson assumes. You know what an IP address and a CIDR prefix like 203.0.113.0/24 are (IP Addressing & Subnetting). You've seen how a router forwards a packet with longest-prefix match, and how OSPF finds shortest paths inside one network (The Network Layer). You know TCP opens a connection with a handshake (TCP/IP Deep Dive). That's all. Every BGP term is defined when it first appears.
What you'll be able to do after reading it. Explain how a packet from a laptop in India finds a server on another continent. Read an AS path. Predict which route a router will choose and why. Explain why money, not distance, shapes most internet paths. Explain how hijacks and leaks work and how RPKI stops some of them. Read the BGP parts of an outage postmortem without getting lost.
The Problem BGP Solves
The internet is not one network
When people say "the internet", they picture one big network. It isn't. It's about 75,000 independent networks that agreed to connect to each other. Your home ISP is one. Google is one. Amazon is one. Your university, a bank, a mobile carrier in Brazil, a hosting company in Germany: each runs its own network, buys its own routers, and makes its own decisions. Nobody is in charge of the whole thing.
Each of those independent networks is called an autonomous system (AS). "Autonomous" means self-governing: inside its own network, an AS can do whatever it likes.
An analogy: countries and postal services
Think of each AS as a country with its own postal service.
- Inside India, India Post knows every street and every post office. It moves letters along the fastest internal roads. Nobody outside India needs to know how India Post sorts its mail.
- When you post a letter to an address in Brazil, India Post doesn't know Brazilian streets. It only needs to know which neighbouring postal service to hand the letter to so that it gets closer to Brazil.
- Postal services tell each other, in effect, "send me mail for these countries, I can get it there." Some charge for carrying mail across their territory. Some swap mail for free because they exchange about the same amount.
- A postal service can also lie, or make a mistake: "send me all mail for Brazil." If neighbours believe it, Brazil's mail goes to the wrong place.
That last bullet is the entire BGP security problem, in one sentence. Hold on to it.
Map the analogy onto the internet:
| Postal world | Internet |
|---|---|
| A country's postal service | An autonomous system (AS) |
| Streets and local post offices inside one country | Routers and links inside one AS |
| How a postal service routes mail internally | An interior routing protocol: OSPF or IS-IS |
| Postal services telling each other which countries they can deliver to | The exterior routing protocol: BGP |
| A block of addresses, like "all of Brazil" | An IP prefix, like 203.0.113.0/24 |
| "Hand it to the Turkish post, they'll pass it on" | The next hop and AS path of a route |
Interior routing vs exterior routing
Inside one AS, routing is the intuitive kind you met in The Network Layer. One organisation owns every router. It wants the fastest path. Every router can see the whole map (OSPF floods the topology to everyone) and runs Dijkstra to find the shortest path. That's an interior gateway protocol (IGP).
Between ASes, none of that holds:
- Nobody sees the whole map. Google won't publish its internal topology to you, and you wouldn't want 75,000 networks' internal maps in your router.
- "Fastest" isn't the goal. Each AS wants the path that suits its business: the cheapest one, the one that earns money, the one its contracts allow.
- Nobody trusts anybody completely. Each AS decides for itself which routes to accept and which to pass on.
So between ASes we need a different kind of protocol, an exterior gateway protocol. There is exactly one in use: BGP-4, the Border Gateway Protocol, version 4, specified in RFC 4271. It's called "border" because it runs on the routers at the edge of each AS, where one network meets another.
flowchart LR
subgraph ISP["IndiaISP AS64530 (runs OSPF inside)"]
L["Your laptop"] --> R1["Router"] --> R2["Border router"]
end
subgraph TR["Transit provider AS64520 (runs IS-IS inside)"]
T1["Border router"] --> T2["Border router"]
end
subgraph CC["ContentCo AS64500 (runs OSPF inside)"]
C1["Border router"] --> S["Web server 203.0.113.10"]
end
R2 ===|"BGP"| T1
T2 ===|"BGP"| C1
Inside each box an IGP finds the fastest path. Across the thick lines, BGP carries one kind of message: "these addresses can be reached through me, and here is the list of networks the route has passed through."
| Interior (IGP: OSPF, IS-IS) | Exterior (BGP) | |
|---|---|---|
| Runs | Inside one organisation | Between organisations |
| Knows | The full internal topology | Only what neighbours choose to tell it |
| Optimises | A technical metric (link cost, delay) | Policy: money, contracts, preferences |
| Scale | Hundreds to thousands of routers | ~75,000 ASes, ~1 million IPv4 prefixes |
| Trust | Full: same owner | Partial: every neighbour is a separate company |
| Typical convergence | Under a second | Seconds to minutes |
How does your packet get to the server?
Here is the question this whole lesson answers. You're in Bengaluru. You type a URL. DNS gives you the address 203.0.113.10. Your laptop sends a packet to your home router, which sends it to your ISP. Your ISP's routers look up 203.0.113.10 in their forwarding table and find an entry like:
203.0.113.0/24 -> send to the IXP link toward AS64500
Where did that entry come from? Nobody at your ISP typed it. It arrived in a BGP announcement that started at the network that owns 203.0.113.0/24 and hopped from AS to AS until it reached your ISP. Every router along the way chose one route from the many it heard about. The rest of the lesson unpacks that sentence.
Key Terms
Each term is defined in plain words with a tiny example. The worked toy internet later uses all of them.
Autonomous Systems
The internet is divided into Autonomous Systems (AS) — independently operated networks with their own routing policies.
AS15169 = Google
AS16509 = Amazon (AWS)
AS13335 = Cloudflare
AS7018 = AT&T
AS1299 = Telia (major backbone)
Each AS gets an ASN (Autonomous System Number) assigned by IANA/RIRs (ARIN, RIPE, APNIC, etc.).
graph LR
AS15169["Google AS15169"] ---|peer| AS1299["Telia AS1299"]
AS16509["AWS AS16509"] ---|peer| AS1299
AS1299 ---|peer| AS7018["AT&T AS7018"]
AS13335["Cloudflare AS13335"] ---|peer| AS1299
AS13335 ---|peer| AS7018
A few details worth knowing about ASNs:
- Who hands them out. IANA gives blocks of numbers to the five Regional Internet Registries (RIRs): ARIN (North America), RIPE NCC (Europe, Middle East, Central Asia), APNIC (Asia-Pacific, including India), LACNIC (Latin America) and AFRINIC (Africa). The RIRs, and in some countries national registries such as IRINN in India, assign individual ASNs to organisations.
- Size. ASNs started as 16-bit numbers (0 to 65535). They ran short, so RFC 6793 made them 32-bit. Most newly assigned public ASNs are now above 65535.
- Private ASNs.
64512–65534and4200000000–4294967294are private, like10.0.0.0/8for addresses. You use them inside your own infrastructure (data centres, Kubernetes clusters) and strip them before talking to the internet. - Documentation ASNs.
64496–64511and65536–65551are reserved for examples (RFC 5398). This page's toy internet uses documentation and private ASNs, and documentation prefixes like203.0.113.0/24, so it can't be confused with real networks. - The name
AS1299above belonged to Telia Carrier, which was renamed Arelion in 2022. The number didn't change. ASNs outlive company names.
Tiny example: "ContentCo is AS64500" means every router in ContentCo's network follows one routing policy, and other networks see it as a single hop called 64500.
Prefix
A prefix is a block of IP addresses written in CIDR notation: the first address, a slash, and how many leading bits are fixed. You met this in IP Addressing & Subnetting.
203.0.113.0/24fixes the first 24 bits, so it covers 256 addresses,203.0.113.0to203.0.113.255.203.0.113.0/25covers the first half of that,203.0.113.0to203.0.113.127. It is more specific (longer prefix, fewer addresses).203.0.0.0/16covers 65,536 addresses. It is less specific.
BGP never talks about single addresses. It always talks about prefixes. Tiny example: ContentCo doesn't announce "I have 203.0.113.10", it announces "I have 203.0.113.0/24".
Route
A route is a prefix plus the information needed to reach it: which neighbour to send packets to, and a set of path attributes describing the route. Tiny example: "203.0.113.0/24, via neighbour AS64510, AS path 64510 64500."
Path (AS path)
The AS path (attribute AS_PATH) is the list of ASes a route announcement has travelled through. Each AS adds its own number to the front when it passes the announcement on. So:
- the rightmost number is the origin, the AS that owns the prefix;
- the leftmost number is the neighbour you heard it from;
- packets travel left to right, the opposite direction to the announcement.
Tiny example: the path 64520 64510 64500 means "the route was born in AS64500, passed through AS64510, then AS64520, then reached me. If I use it, my packets go to 64520, then 64510, then 64500."
Peer / neighbour
Two BGP routers that have a BGP session with each other are peers or neighbours. The words mean the same thing at the protocol level.
Two meanings of "peer"
"Peer" also has a business meaning: two networks that exchange traffic for free (see Business relationships). A customer and its paying provider are BGP peers (they have a session) without being business peers. When this page means the business sense it says "settlement-free peer" or "peering relationship".
Announcement (advertisement)
An announcement or advertisement is a BGP message that says "I can reach this prefix, and here are the attributes." It's carried in a BGP UPDATE message. Tiny example: AS64500 sends AS64510 an UPDATE: "203.0.113.0/24, AS path 64500, next hop 192.0.2.1."
Withdrawal
A withdrawal is the opposite: "I can no longer reach this prefix, forget the route I gave you." It's also carried in an UPDATE message, in a separate field. A route is also implicitly withdrawn if the neighbour sends a new announcement for the same prefix (it replaces the old one), or if the BGP session goes down (everything learned from that neighbour is discarded). Tiny example: the Facebook outage of October 2021 was, at the BGP level, a wave of withdrawals.
A few more you'll meet
| Term | Plain meaning | Tiny example |
|---|---|---|
| Next hop | The IP address to send packets to for this route | 192.0.2.1, the ContentCo router on the IXP LAN |
| Origin AS | The AS at the right end of the path, which claims to own the prefix | AS64500 for 203.0.113.0/24 |
| eBGP | A BGP session between two different ASes | IndiaISP's router to ContentCo's router |
| iBGP | A BGP session between two routers in the same AS | IndiaISP's Mumbai router to its Delhi router |
| RIB | Routing Information Base: every route the router knows, including losers | Two routes to 203.0.113.0/24, one marked best |
| FIB | Forwarding Information Base: only the winners, programmed into hardware | 203.0.113.0/24 -> port 7 |
| Transit | Carrying traffic between two other networks, usually for money | A Tier 1 carrying your ISP's traffic to Europe |
| IXP | Internet Exchange Point: a building with a shared switch where many networks connect | DE-CIX Mumbai, NIXI, AMS-IX |
| Policy | The rules an AS uses to accept, prefer, and pass on routes | "Prefer customer routes, never give peer routes to other peers" |
What BGP Does
BGP routers exchange reachability information — "I can reach these CIDR prefixes, and here's the path to get there."
BGP announcement:
Prefix: 203.0.113.0/24
AS path: 15169 → 1299 → 7018
Next hop: 192.0.2.1
Meaning: "To reach 203.0.113.0/24, send traffic to 192.0.2.1.
It will traverse AS15169, then AS1299, then reach AS7018."
In that example the router that receives the announcement is talking to AS15169. AS7018 is the origin (rightmost). The arrows show the direction traffic will flow, which is the reverse of the direction the announcement travelled.
The AS path serves two purposes:
- Loop prevention — if your own ASN appears in the path, reject it
- Path length as a routing metric — shorter AS path is preferred (by default)
BGP is a path-vector protocol
In The Network Layer you met two families of routing algorithm. Link-state protocols (OSPF) share the whole map. Distance-vector protocols (RIP) share only "I can reach X at cost N", and suffer from count-to-infinity loops.
BGP is a path-vector protocol: a distance-vector protocol that sends the whole path instead of a number. That one change fixes the loop problem. A router that sees its own ASN in a path knows that route would send packets in a circle, and throws it away. It also gives every AS a way to apply policy ("never send my traffic through AS64999") because it can see exactly which networks a route passes through.
What BGP does not do
- It doesn't measure anything. BGP doesn't know link speeds, latency, or congestion. A 3-hop path over slow links beats a 4-hop path over fast ones unless an operator configures otherwise.
- It doesn't forward packets. BGP fills the routing table. The forwarding hardware does longest-prefix match on the FIB.
- It doesn't verify anything on its own. A router believes what its neighbours say, unless the operator adds filters or RPKI validation (Security).
A Worked Toy Internet
Time to watch BGP work. Here is a small internet with five ASes and one IXP. Every table in this section comes from the runnable simulation near the end of the page.
flowchart TB
TA["TransitA AS64510<br/>global transit provider"] ---|"settlement-free peers"| TB["TransitB AS64520<br/>global transit provider"]
TA -->|"sells transit to"| CC["ContentCo AS64500<br/>owns 203.0.113.0/24"]
TB -->|"sells transit to"| ISP["IndiaISP AS64530<br/>your laptop is here"]
TB -->|"sells transit to"| CN["CampusNet AS64540<br/>a university"]
ISP -->|"sells transit to"| CN
CC ---|"peering port"| IXP["Mumbai IXP<br/>(shared switch, not an AS hop)"]
IXP ---|"peering port"| ISP
| AS | Name | Role | Its neighbours, from its own point of view |
|---|---|---|---|
| 64500 | ContentCo | Content company, owns 203.0.113.0/24 | TransitA = provider, IndiaISP = peer (at the IXP) |
| 64510 | TransitA | Transit provider | ContentCo = customer, TransitB = peer |
| 64520 | TransitB | Transit provider | TransitA = peer, IndiaISP = customer, CampusNet = customer |
| 64530 | IndiaISP | Eyeball ISP (serves homes) | TransitB = provider, ContentCo = peer, CampusNet = customer |
| 64540 | CampusNet | University, buys from two ISPs ("multihomed") | IndiaISP = provider, TransitB = provider |
The IXP is a switch that both ContentCo and IndiaISP plug into. They run a BGP session directly with each other across it. The IXP itself does not appear in the AS path. (Many IXPs also run a route server that relays routes between members to save them from configuring hundreds of sessions. Route servers are normally transparent too: they don't add their ASN to the path, per RFC 7947.)
The two rules every AS follows here
Each AS applies the same two rules, which mirror how real networks behave (the Business relationships section explains why):
- Preference (import policy). Set
LOCAL_PREF, a "how much do I like this route" number, by who the route came from: customer 200, peer 150, provider 100. Higher wins. Customers pay you, peers are free, providers cost you money. - Export policy. Routes learned from customers (or originated yourself) go to everyone. Routes learned from peers or providers go only to customers. You never carry traffic between two networks that aren't paying you.
Step 0: ContentCo originates the route
ContentCo configures its border router to announce 203.0.113.0/24. Its own routing table now has one entry with an empty AS path: "this is mine."
Step 1 (round 1): ContentCo tells its neighbours
ContentCo's own route may go to everyone, so it sends it to its provider (TransitA) and its peer (IndiaISP). Before sending, it prepends its ASN, so the path is 64500.
| From | To | Message | Prefix | AS_PATH | Receiver does |
|---|---|---|---|---|---|
| AS64500 | AS64510 | UPDATE | 203.0.113.0/24 | 64500 | stores (from customer, LOCAL_PREF 200), best changed |
| AS64500 | AS64530 | UPDATE | 203.0.113.0/24 | 64500 | stores (from peer, LOCAL_PREF 150), best changed |
Routing tables after round 1:
| AS | Prefix | Candidate paths | Best path | Decided by |
|---|---|---|---|---|
| AS64500 ContentCo | 203.0.113.0/24 | (origin) [LP 200] | (origin) | only route |
| AS64510 TransitA | 203.0.113.0/24 | 64500 [LP 200] | 64500 | only route |
| AS64530 IndiaISP | 203.0.113.0/24 | 64500 [LP 150] | 64500 | only route |
TransitB and CampusNet don't know the prefix exists yet. If you sent a packet to 203.0.113.10 from CampusNet right now, it would be dropped: no route.
Step 2 (round 2): the news spreads one more hop
TransitA learned the route from a customer, so it may tell everyone. It tells its peer TransitB, adding its own number: 64510 64500. It does not send it back to ContentCo (pointless, and ContentCo would reject it because 64500 is already in the path).
IndiaISP learned the route from a peer, so it may only tell customers. It tells CampusNet: 64530 64500. It does not tell its provider TransitB. Doing so would mean IndiaISP paying TransitB to carry traffic from the rest of the world to ContentCo, for free, on ContentCo's behalf. That's the "valley" that valley-free routing forbids.
| From | To | Message | Prefix | AS_PATH | Receiver does |
|---|---|---|---|---|---|
| AS64510 | AS64520 | UPDATE | 203.0.113.0/24 | 64510 64500 | stores (from peer, LOCAL_PREF 150), best changed |
| AS64530 | AS64540 | UPDATE | 203.0.113.0/24 | 64530 64500 | stores (from provider, LOCAL_PREF 100), best changed |
| AS | Prefix | Candidate paths | Best path | Decided by |
|---|---|---|---|---|
| AS64500 ContentCo | 203.0.113.0/24 | (origin) [LP 200] | (origin) | only route |
| AS64510 TransitA | 203.0.113.0/24 | 64500 [LP 200] | 64500 | only route |
| AS64520 TransitB | 203.0.113.0/24 | 64510 64500 [LP 150] | 64510 64500 | only route |
| AS64530 IndiaISP | 203.0.113.0/24 | 64500 [LP 150] | 64500 | only route |
| AS64540 CampusNet | 203.0.113.0/24 | 64530 64500 [LP 100] | 64530 64500 | only route |
Every AS can now reach ContentCo. But BGP isn't finished, because TransitB just learned something new.
Step 3 (round 3): second opinions arrive
TransitB learned the route from a peer, so it tells only its customers, IndiaISP and CampusNet, with path 64520 64510 64500.
| From | To | Message | Prefix | AS_PATH | Receiver does |
|---|---|---|---|---|---|
| AS64520 | AS64530 | UPDATE | 203.0.113.0/24 | 64520 64510 64500 | stores (from provider, LOCAL_PREF 100) |
| AS64520 | AS64540 | UPDATE | 203.0.113.0/24 | 64520 64510 64500 | stores (from provider, LOCAL_PREF 100) |
Notice "best changed" is missing. Both receivers store the new route as a backup but keep their current choice:
| AS | Prefix | Candidate paths | Best path | Decided by |
|---|---|---|---|---|
| AS64500 ContentCo | 203.0.113.0/24 | (origin) [LP 200] | (origin) | only route |
| AS64510 TransitA | 203.0.113.0/24 | 64500 [LP 200] | 64500 | only route |
| AS64520 TransitB | 203.0.113.0/24 | 64510 64500 [LP 150] | 64510 64500 | only route |
| AS64530 IndiaISP | 203.0.113.0/24 | 64500 [LP 150], 64520 64510 64500 [LP 100] | 64500 | highest LOCAL_PREF |
| AS64540 CampusNet | 203.0.113.0/24 | 64530 64500 [LP 100], 64520 64510 64500 [LP 100] | 64530 64500 | shortest AS_PATH |
Two different attributes decided:
- IndiaISP keeps the IXP route because it came from a peer (LOCAL_PREF 150) and the other came from a paid provider (LOCAL_PREF 100). LOCAL_PREF is checked before path length, so the decision is made on money before distance is even considered. Here the cheaper route is also shorter, but it would win even if it were longer.
- CampusNet has two routes from two providers. Both have LOCAL_PREF 100, a tie. The next tie-breaker is AS path length: 2 beats 3, so it uses IndiaISP.
Since no best route changed, nobody has anything new to say. The network has converged: every router has made a stable choice.
Trace a packet
Now the forwarding part. A packet from your laptop in IndiaISP to 203.0.113.10, and one from a student at CampusNet. Each AS looks up the destination in its FIB (longest-prefix match), finds the best route, and sends the packet to the neighbour that route came from. The simulation's output:
Packet AS64530 -> 203.0.113.10: AS64530 (matches 203.0.113.0/24) -> AS64500 (delivered, owns 203.0.113.0/24)
Packet AS64540 -> 203.0.113.10: AS64540 (matches 203.0.113.0/24) -> AS64530 (matches 203.0.113.0/24) -> AS64500 (delivered, owns 203.0.113.0/24)
sequenceDiagram
participant L as Laptop in CampusNet
participant CN as CampusNet AS64540
participant ISP as IndiaISP AS64530
participant CC as ContentCo AS64500
L->>CN: packet to 203.0.113.10
Note over CN: FIB lookup, 203.0.113.0/24 via AS64530
CN->>ISP: forward
Note over ISP: FIB lookup, 203.0.113.0/24 via IXP peer AS64500
ISP->>CC: forward across the IXP switch
Note over CC: 203.0.113.0/24 is local, the IGP takes it to the server
The packet followed the AS path 64530 64500 exactly, left to right. The reply from ContentCo back to the student follows whatever route ContentCo picked for CampusNet's prefix, which can be a different path. Internet routing is often asymmetric: the way there and the way back are chosen independently by different networks.
What the toy example already teaches
- Announcements flow outward from the owner. Traffic flows back toward the owner, along the reverse of the chosen announcement path.
- Each AS only knows what its neighbours told it. TransitB never learns about the IXP link.
- Each AS chooses one best route and only passes on that one. That's part of why BGP scales: a router tells its neighbours one route per prefix, not every route it knows. (The ADD-PATH extension, RFC 7911, lets routers send more than one when they need to.)
- Policy shapes everything. IndiaISP didn't tell TransitB about the IXP route. Because of that one export rule, TransitB's customers reach ContentCo via TransitA, not via IndiaISP.
How a BGP Session Works
So far ASes "told" each other things. Concretely, two routers run a BGP session: a long-lived TCP connection over which they exchange BGP messages.
TCP port 179
BGP runs over TCP port 179. Using TCP means BGP gets reliable, in-order delivery for free and doesn't need its own retransmission logic, unlike OSPF, which runs directly on IP. Either router may open the connection. If both try at once, the collision is resolved by comparing BGP identifiers and keeping one connection.
eBGP neighbours are usually directly connected (a cable, or the same IXP LAN), so by default eBGP packets are sent with an IP TTL of 1. iBGP neighbours can be many hops apart inside an AS, often using loopback addresses that the IGP makes reachable.
The five message types
Every BGP message starts with a 19-byte header: a 16-byte marker (all ones), a 2-byte length, and a 1-byte type. Messages were limited to 4,096 bytes until RFC 8654 allowed extended messages up to 65,535 bytes.
| Type | Name | When it's sent | What it carries |
|---|---|---|---|
| 1 | OPEN | First message after TCP connects | BGP version (4), my ASN, proposed hold time, BGP identifier (router ID), capabilities |
| 2 | UPDATE | Whenever routes change | Withdrawn prefixes, path attributes, and the prefixes (NLRI) those attributes apply to |
| 3 | NOTIFICATION | When something is wrong | An error code and subcode. The session is closed right after |
| 4 | KEEPALIVE | Periodically when there's nothing else to say | Nothing: just the 19-byte header. Means "I'm still here" |
| 5 | ROUTE-REFRESH | When a router wants its neighbour to resend everything (RFC 2918) | Address family. Used after changing an inbound policy |
Hold time and keepalives. In OPEN, each side proposes a hold time and both use the smaller one. If no message arrives for the hold time, the router assumes the neighbour is dead, sends a NOTIFICATION, and drops the session, along with every route learned from it. RFC 4271 suggests a 90-second hold time with keepalives every third of that (30 s). Cisco IOS defaults to 180 s and 60 s. That's a long time to wait for a dead link, which is why operators add BFD (Bidirectional Forwarding Detection, RFC 5880) to detect failures in milliseconds.
Deeper: what is inside an UPDATE
An UPDATE has three parts:
- Withdrawn Routes: a list of prefixes to forget.
- Path Attributes:
ORIGIN,AS_PATH,NEXT_HOP, optionallyLOCAL_PREF(iBGP only),MULTI_EXIT_DISC(MED),COMMUNITIES, and others. Each attribute has flags saying whether it's optional and whether it's transitive (passed on to other ASes). - NLRI (Network Layer Reachability Information): the IPv4 prefixes that share those attributes.
One UPDATE can withdraw some prefixes and announce others in the same message. All announced prefixes in one UPDATE share the same attributes, which is why routers pack many prefixes with identical attributes into one message.
IPv6 and other address families don't fit in the original IPv4-only NLRI field. Multiprotocol BGP (RFC 4760) carries them inside two attributes, MP_REACH_NLRI and MP_UNREACH_NLRI. That's how one protocol carries IPv4, IPv6, VPN routes (MPLS L3VPN), EVPN for data-centre overlays, and FlowSpec filtering rules.
Deeper: capabilities in the OPEN message
Capabilities (RFC 5492) let two routers agree on features without breaking older software. Common ones: multiprotocol (which address families to carry), 4-byte ASN support (RFC 6793), route refresh, graceful restart (RFC 4724), ADD-PATH, and extended messages. A router with a 4-byte ASN talking to an old 2-byte-only router puts the reserved value AS_TRANS (23456) in the OPEN's "My AS" field and carries its real ASN in the capability. That's why you sometimes see AS23456 in old routing data.
The finite state machine
A BGP session moves through six states. RFC 4271 calls this the finite state machine (FSM).
stateDiagram-v2
[*] --> Idle
Idle --> Connect: start event, open TCP to port 179
Connect --> OpenSent: TCP connected, send OPEN
Connect --> Active: TCP attempt failed, wait and listen
Active --> OpenSent: TCP connected, send OPEN
Active --> Connect: ConnectRetry timer expires, try again
OpenSent --> OpenConfirm: valid OPEN received, send KEEPALIVE
OpenSent --> Active: TCP connection closed
OpenConfirm --> Established: KEEPALIVE received
Established --> Established: UPDATE or KEEPALIVE received
OpenSent --> Idle: bad OPEN, send NOTIFICATION
OpenConfirm --> Idle: error or hold timer expired
Established --> Idle: NOTIFICATION, hold timer expired, or TCP closed
| State | What the router is doing | If you see it stuck here |
|---|---|---|
| Idle | Doing nothing. Refuses connections. Waits for a start event (config, or a retry timer after an error) | Session administratively down, or backing off after repeated errors |
| Connect | Trying to open a TCP connection to the neighbour | TCP isn't completing: wrong IP, routing, firewall dropping port 179 |
| Active | The TCP attempt failed. Listening for the neighbour to connect, and will retry | Same as Connect. "Active" sounds healthy but means not connected |
| OpenSent | TCP is up, OPEN sent, waiting for the neighbour's OPEN | Neighbour not configured for you, or rejects your OPEN |
| OpenConfirm | Got a valid OPEN, sent KEEPALIVE, waiting for its KEEPALIVE | Rare. Usually a mismatch caught right away |
| Established | Up. Exchanging UPDATEs and KEEPALIVEs | This is the goal |
The most common causes of a session that won't reach Established are boring: the wrong neighbour IP, the wrong remote-as (OPEN carries your ASN, and a mismatch earns a NOTIFICATION "Bad Peer AS"), a firewall blocking TCP 179, or a mismatched TCP-MD5/TCP-AO password.
Two routers peering, end to end
sequenceDiagram
participant A as IndiaISP router AS64530
participant B as ContentCo router AS64500
Note over A,B: Both start in Idle, then Connect
A->>B: TCP SYN to port 179
B->>A: SYN-ACK
A->>B: ACK, TCP is up
A->>B: OPEN, AS 64530, hold 90, ID 192.0.2.30, capabilities
B->>A: OPEN, AS 64500, hold 90, ID 192.0.2.1, capabilities
Note over A,B: Each checks the other's ASN and capabilities, now OpenConfirm
A->>B: KEEPALIVE
B->>A: KEEPALIVE
Note over A,B: Established
B->>A: UPDATE, announce 203.0.113.0/24, AS_PATH 64500, NEXT_HOP 192.0.2.1
A->>B: UPDATE, announce 198.51.100.0/24, AS_PATH 64530, NEXT_HOP 192.0.2.30
Note over A,B: Full table exchanged, then only changes are sent
loop every 30 seconds with nothing to say
A->>B: KEEPALIVE
B->>A: KEEPALIVE
end
B->>A: UPDATE, withdraw 203.0.113.0/24
Note over A: Removes the route, picks a backup, tells its own neighbours
After the initial exchange, BGP is incremental: it only sends changes. A full IPv4 table is around a million prefixes, so resending it every few seconds (as RIP does with its whole table) would be impossible.
Deeper: what this looks like in router configuration
A minimal eBGP setup in FRRouting for IndiaISP's side. Real networks add filters, prefix limits and authentication on top.
router bgp 64530
bgp router-id 192.0.2.30
neighbor 192.0.2.1 remote-as 64500
neighbor 192.0.2.1 description ContentCo-at-IXP
address-family ipv4 unicast
network 198.51.100.0/24
neighbor 192.0.2.1 route-map FROM-PEER in
neighbor 192.0.2.1 route-map TO-PEER out
neighbor 192.0.2.1 maximum-prefix 1000
exit-address-family
!
route-map FROM-PEER permit 10
set local-preference 150
!
route-map TO-PEER permit 10
match ip address prefix-list MY-AND-CUSTOMER-PREFIXES
The two route-map lines are not optional decoration. RFC 8212 says an eBGP session with no policy configured should accept and send nothing, and FRR follows it by default (bgp ebgp-requires-policy). Older software accepted and re-announced everything by default, which made accidental leaks trivially easy.
Path Attributes and Best-Path Selection
The attributes that matter, in plain words
Every route carries path attributes. Most of BGP's behaviour comes from a handful of them.
| Attribute | Plain question it answers | Set by | Seen by |
|---|---|---|---|
LOCAL_PREF | "Which exit do I prefer?" Higher wins | You, on routes you receive | Only your own AS (sent over iBGP, never eBGP) |
AS_PATH | "Which networks does this route go through?" Shorter wins | Each AS prepends itself | Everyone |
ORIGIN | "How did the origin inject this route?" IGP < EGP < INCOMPLETE | The origin AS | Everyone |
MED (MULTI_EXIT_DISC) | "Which of my links should you use to send me traffic?" Lower wins | The neighbour AS | You only (not passed further) |
NEXT_HOP | "Which IP do I send packets to?" | The advertising router | Everyone it's passed to |
COMMUNITIES | Tags that mean whatever the networks agree on (RFC 1997) | Anyone | Passed along unless stripped |
A few notes:
- LOCAL_PREF is your own lever. It only exists inside your AS, so other ASes can't see or influence it. That's why it's checked first.
- ORIGIN values are historical.
IGPmeans "the origin announced it deliberately with anetworkstatement".EGPrefers to the Exterior Gateway Protocol, BGP's 1980s predecessor, which is extinct.INCOMPLETEmeans "redistributed from somewhere else", for example from static routes. - MED is a hint from a neighbour. You can ignore it, and by default it's only compared between routes from the same neighbouring AS.
- Communities are labels like
64510:100. Providers publish what their communities mean: "tag your route64510:666and I'll drop traffic to it" (blackholing, RFC 7999), or "tag it64510:3and I'll lower its LOCAL_PREF". Large communities (RFC 8092) exist because 32-bit ASNs don't fit in the original format.
BGP Route Selection
When a router receives multiple routes to the same prefix, it selects the best one using a priority-ordered list of attributes:
| Priority | Attribute | Prefer |
|---|---|---|
| 1 | Weight (Cisco-specific) | Highest |
| 2 | LOCAL_PREF | Highest (prefer this exit) |
| 3 | Locally originated | Local routes win |
| 4 | AS path length | Shortest |
| 5 | Origin type | IGP > EGP > incomplete |
| 6 | MED (Multi-Exit Discriminator) | Lowest |
| 7 | eBGP over iBGP | eBGP preferred |
| 8 | IGP metric to next hop | Lowest |
| 9 | Router ID | Lowest (tiebreaker) |
LOCAL_PREF is the most important attribute for traffic engineering. Set it higher on preferred exit points to control which path outbound traffic takes.
MED lets you tell your upstream "prefer this entry point." You set MED; your upstream may or may not honor it.
In the "Origin type" row, "IGP > EGP > incomplete" means IGP is preferred over EGP, which is preferred over incomplete. (Implementations encode them as 0, 1, 2 and prefer the lowest number, which is why other tables write it IGP < EGP < INCOMPLETE.)
The full ordered decision table
The table above is the version most people memorise. Here's the fuller list, as implemented by most vendors. The router walks down it only until one route is left. The moment a step separates the routes, the rest are never looked at.
| Step | Rule | Prefer | Intuition |
|---|---|---|---|
| 0 | Is the route usable? | Next hop reachable, passed import policy, no own ASN in the path, not RPKI-invalid if ROV is on | Ignore routes you can't or won't use |
| 1 | Weight (Cisco/Arista local-only) | Highest | A per-router override. Never leaves the router |
| 2 | LOCAL_PREF | Highest | Business policy: which exit do I want? |
| 3 | Locally originated | Own routes | My own prefixes beat anything I hear |
| 4 | AS_PATH length | Shortest | Fewer networks in the way, roughly "closer" |
| 5 | ORIGIN | IGP, then EGP, then INCOMPLETE | Historical. Rarely decides |
| 6 | MED | Lowest (same neighbour AS only) | The neighbour's hint about its preferred entry |
| 7 | eBGP over iBGP | eBGP | Hand traffic out of my network now rather than carry it |
| 8 | IGP cost to the next hop | Lowest | Hot potato: use the exit closest to me |
| 9 | Oldest eBGP route | Oldest | Stability: don't flip between equal routes |
| 10 | Lowest router ID | Lowest | Deterministic tie-break |
| 11 | Shortest cluster list | Shortest | Route reflector detail (see iBGP) |
| 12 | Lowest neighbour address | Lowest | Final tie-break |
RFC 4271 section 9.1.2.2 defines the standard part of this (LOCAL_PREF, AS path, origin, MED, eBGP over iBGP, IGP cost, then BGP identifier and neighbour address). Weight, "locally originated" and "oldest route" are vendor additions. If routers in one AS come from different vendors, check their exact orders.
Worked examples
These come from the simulation's best_path function, which walks the same steps in order and reports which step decided.
| Example | Candidates | Winner | Decided by |
|---|---|---|---|
| 1. LOCAL_PREF beats a shorter path | 64520 64510 64500 (LP 100, MED 0, eBGP, IGP 0, RID 0.0.0.0) 64530 64999 64998 64500 (LP 200, MED 0, eBGP, IGP 0, RID 0.0.0.0) | 64530 64999 64998 64500 | highest LOCAL_PREF |
| 2. MED decides between two links to the same neighbour | 64510 64500 (LP 100, MED 50, eBGP, IGP 0, RID 10.0.0.1) 64510 64500 (LP 100, MED 10, eBGP, IGP 0, RID 10.0.0.2) | 64510 64500 | lowest MED |
| 3. Everything ties until IGP cost (hot potato) | 64510 64500 (LP 100, MED 0, iBGP, IGP 30, RID 10.0.0.1) 64520 64500 (LP 100, MED 0, iBGP, IGP 5, RID 10.0.0.9) | 64520 64500 | lowest IGP cost |
| 4. Last resort: lowest router ID | 64510 64500 (LP 100, MED 0, eBGP, IGP 0, RID 192.0.2.7) 64520 64500 (LP 100, MED 0, eBGP, IGP 0, RID 192.0.2.3) | 64520 64500 | lowest router ID |
Example 1: money beats distance. One route comes from a paying customer with a 4-hop path. The other comes from a provider with a 3-hop path. The customer route wins at the LOCAL_PREF step and path length is never examined. This is the normal case on the real internet, and it's why the path BGP picks is often not the shortest one.
Example 2: MED. ContentCo buys from TransitA over two links, one in Mumbai and one in Chennai. It wants inbound traffic to arrive in Mumbai, so it sends its routes over the Mumbai link with MED 10 and over Chennai with MED 50. TransitA sees two otherwise identical routes from the same neighbour AS and picks the lower MED. If the routes had come from two different neighbour ASes, MED would not be compared (unless an operator turns on always-compare-med).
Example 3: hot potato. Inside a large AS, two border routers both learned a route to the same prefix, and the decision is being made by a third router that hears both over iBGP. Everything ties until IGP cost: one exit is 5 away, the other 30. The router picks the nearer exit and gets the traffic out of its own network as fast as possible. That's hot-potato routing: hand the packet off quickly, so the other network carries it the long distance. It's the default behaviour, and it's a big reason internet paths are asymmetric.
Example 4: tie-break. When everything is equal, the lowest router ID wins. It's arbitrary but deterministic, so the same inputs always give the same answer.
Influencing inbound traffic
LOCAL_PREF controls how traffic leaves your network. Controlling how traffic arrives is harder, because that's other people's decision. The tools, roughly from gentlest to bluntest:
| Tool | How it works | Limits |
|---|---|---|
| MED | Hint to one neighbour about which of your links to use | Only that neighbour, and only if it honours MED |
| AS-path prepending | Announce 64500 64500 64500 on the link you want less traffic on, so that path looks longer | Loses to anyone's LOCAL_PREF. Diminishing returns |
| Communities | Ask a provider to lower its LOCAL_PREF for your route, or not announce it in a region | Depends on each provider's published community set |
| More-specific prefixes | Announce /24s on one link and the covering /23 on both | Adds to everyone's routing table. Same mechanism as a hijack |
Business Relationships: Why Routes Look the Way They Do
The decision table tells you how a router chooses. It doesn't tell you why LOCAL_PREF is set the way it is. The answer is money. To understand real internet paths, you have to understand who pays whom.
Three relationships
Every link between two ASes is, roughly, one of these:
| Relationship | Who pays | What each side announces to the other |
|---|---|---|
| Customer → provider (transit) | The customer pays the provider, usually per Mbit/s of traffic | Customer sends its own routes (and its customers'). Provider sends everything it knows: a full table or a default route |
| Peer ↔ peer (settlement-free peering) | Nobody. Each covers its own costs | Each sends only its own routes and its customers' routes. Not routes from its providers or other peers |
| Sibling | Same organisation, two ASNs | Everything. Rare, mostly after mergers |
From the point of view of one AS, that gives three kinds of neighbour, and a clear preference:
- Customer routes are the best. Traffic sent to a customer earns money (the customer pays for the traffic you deliver).
- Peer routes are next. They're free.
- Provider routes are last. Traffic sent to a provider costs money.
That's where the LOCAL_PREF 200/150/100 in the toy internet came from. Real networks use exactly this pattern, often with communities to tag each route with where it was learned.
BGP Peering Types
Tier 1 ISPs (AT&T, Telia, NTT, etc.) peer with each other for free — settlement-free peering. They have agreements not to charge each other because traffic exchange is roughly equal.
Peering (IX): Two networks directly exchange traffic, usually at an Internet Exchange Point (IXP) like DE-CIX, LINX, or AMS-IX. Free or low-cost. Requires similar traffic volumes.
Transit: A smaller network pays a larger one (upstream) to carry its traffic to the global internet. A startup buys transit from an ISP; the ISP buys transit from a Tier 1.
Your startup → ISP (pays transit) → Tier 1 → Global internet
Cloudflare → peers with everyone at IXPs (doesn't pay transit)
Correction (2026-11-13)
"Cloudflare doesn't pay transit" overstates it. Cloudflare peers very widely (it has an open peering policy and connects at hundreds of IXPs), but it also buys IP transit to reach networks that won't peer with it. Most large content networks mix the two. Also, "requires similar traffic volumes" applies mainly to private peering between large networks. Many networks peer openly at IXPs, including over route servers, with very unequal traffic.
What makes a network "Tier 1" is that it can reach the entire internet without paying anyone for transit: every route it has comes from its own customers or from settlement-free peers. There are roughly fifteen to twenty such networks. Commonly listed examples include Lumen (AS3356), Arelion, formerly Telia (AS1299), NTT (AS2914), Cogent (AS174), GTT (AS3257), Tata Communications (AS6453), Verizon (AS701) and AT&T (AS7018). The exact list is debated because peering contracts are private.
IXPs
An Internet Exchange Point is mostly a big Ethernet switch fabric in a data centre. Members plug a router port into it and get an IP address on a shared subnet. Any two members can then run a BGP session across the switch and exchange traffic directly, without paying a transit provider in between. The largest IXPs connect over a thousand networks.
Why IXPs matter: without one, a packet from IndiaISP to a content company in the same city might travel to a transit provider's router in Singapore and back. With one, it crosses a switch in the same building. That's lower latency and cheaper for both. India's NIXI and DE-CIX Mumbai, and Europe's DE-CIX Frankfurt, AMS-IX and LINX, are examples.
Most IXPs offer a route server: one BGP session to the route server gets you routes from every member that uses it, instead of one session per member. Route servers relay routes without inserting their own ASN into the path.
Valley-free routing
Put the export rules together:
- Routes from customers (and your own) → tell everyone.
- Routes from peers or providers → tell only customers.
The consequence is a shape rule for every path on the internet. Read a path from the origin outward: it climbs up through providers (customer → provider) zero or more times, crosses at most one peering link, then goes down through customers zero or more times. It never goes down and then back up. A path that does is a valley, and in a well-behaved internet it doesn't exist.
flowchart TB
subgraph Top["Tier 1s, peering with each other"]
T1["Tier 1 X"] ---|"peer"| T2["Tier 1 Y"]
end
T1 -->|"provider of"| R1["Regional ISP A"]
T2 -->|"provider of"| R2["Regional ISP B"]
R1 -->|"provider of"| S1["Small ISP"]
R2 -->|"provider of"| S2["Startup"]
R1 ---|"peer at IXP"| R2
R2 -.->|"also provider of Small ISP"| S1
Good paths from Startup to Small ISP:
- Up to Regional B, across the IXP peering to Regional A, down to Small ISP. Up, across, down. Valley-free.
- Up to Regional B, up to Tier 1 Y, across to Tier 1 X, down to Regional A, down to Small ISP. Valley-free, but longer, and Regional B would only use it if the IXP link failed.
A bad path: Small ISP is a customer of both Regional A and Regional B (the dotted line). If Small ISP takes the routes it learned from Regional A and announces them to Regional B, then Regional B may send traffic for Regional A's customers down into Small ISP and back up to Regional A. Small ISP would be paying both providers to carry other people's traffic through its own small links, which then overload. That's a route leak, defined formally in RFC 7908, and it's what happened in the 2019 Verizon incident.
Deeper: Gao-Rexford and why the rules make BGP stable
BGP is not guaranteed to converge. With arbitrary policies, three ASes can each prefer the route through the next one, and the system oscillates forever. Griffin, Shepherd and Wilfong called this structure a "dispute wheel". Lixin Gao and Jennifer Rexford showed in 2001 ("Stable Internet Routing Without Global Coordination") that if every AS (1) prefers customer routes over peer and provider routes, (2) follows the valley-free export rules, and (3) the provider-customer graph has no cycles, then BGP always converges to a stable state. The internet's economic habits turn out to be what keeps its routing stable. Gao's earlier paper "On Inferring Autonomous System Relationships in the Internet" used the valley-free property in reverse, to guess business relationships from public AS paths. CAIDA's AS Relationships dataset still does a version of this today.
Longest-Prefix Match vs BGP's Choice
There are two separate decisions in a router, and mixing them up is the most common source of confusion about BGP.
Two tables, two jobs
- BGP's choice (in the RIB). For each prefix, BGP picks one best route using the decision table. It never compares routes for different prefixes.
203.0.113.0/24and203.0.113.0/25are two different prefixes to BGP, each with its own winner. - The forwarding lookup (in the FIB). The winners from BGP (plus connected, static and IGP routes) are installed in the FIB. When a packet arrives, the hardware finds every FIB entry that contains the destination address and uses the longest one. That's longest-prefix match, which you already know.
flowchart LR
U1["UPDATE 203.0.113.0/24 from AS64510"] --> RIB
U2["UPDATE 203.0.113.0/24 from AS64530"] --> RIB
U3["UPDATE 203.0.113.0/25 from AS64520"] --> RIB
RIB["RIB<br/>best path chosen per prefix"] -->|"winner for /24"| FIB
RIB -->|"winner for /25"| FIB
FIB["FIB<br/>longest-prefix match per packet"] --> P1["Packet to 203.0.113.10<br/>matches /24 and /25, uses /25"]
FIB --> P2["Packet to 203.0.113.200<br/>matches only /24"]
So BGP attributes like LOCAL_PREF and AS path length decide between routes for the same prefix. Prefix length decides between different prefixes, and it always wins. No LOCAL_PREF, however high, can make a /24 route beat a /25 route for an address inside the /25.
Why a /24 beats a /16
Say your router has two routes in its FIB:
| Prefix | Learned from | AS path |
|---|---|---|
198.51.0.0/16 | A provider, LOCAL_PREF 100 | 64510 64500 |
198.51.100.0/24 | Anyone at all | 64520 64777 |
A packet to 198.51.100.7 matches both. The /24 has 24 matching bits, the /16 only 16. The /24 wins. It doesn't matter that the /16 came from the real owner, has a shorter path, or a higher LOCAL_PREF. Those were never compared.
That's useful. It lets a company that owns a big block send part of it via a different provider by announcing a more-specific, and it's how traffic engineering with more-specifics works. It's also exactly how sub-prefix hijacks work. Anyone who can get a more-specific route accepted by enough networks attracts all traffic for that slice of address space, everywhere the route reaches.
Why /24 (and /48) is the usual limit
If more-specifics always win, why doesn't an attacker announce a /32 for one address? Because most networks filter out IPv4 prefixes longer than /24, and IPv6 prefixes longer than /48. This is a convention, not a protocol rule, and it exists to stop the global table from exploding. A side effect: a network that announces its own space as /24s can't be out-specificed by a hijacker on the public internet. Many networks deliberately originate /24s for important services for that reason.
The simulation shows the hijack effect directly in Scenario D: a /25 announced by the wrong AS wins traffic for 203.0.113.10 everywhere it's accepted, while 203.0.113.200, outside the /25, keeps flowing to the real owner.
iBGP vs eBGP
eBGP (External BGP): BGP sessions between routers in different Autonomous Systems. This is what "BGP on the internet" means.
iBGP (Internal BGP): BGP sessions between routers within the same AS. Used to propagate external routes learned at one edge router to all other edge routers in the same AS.
eBGP session
AS15169 ────────────────────────────────── AS1299
(Google) │ │ (Telia)
Router1 iBGP Router2
└──────────────────┘
(both in AS15169)
iBGP requirement: all routers in an AS that run iBGP must be fully meshed (or use route reflectors), because iBGP does not re-advertise routes learned from iBGP to other iBGP peers (loop prevention).
Why iBGP exists at all
So far each AS was one dot. A real AS has many border routers. IndiaISP might connect to TransitB in Mumbai and Chennai, and to the IXP in Mumbai. The Chennai router hears TransitB's route. The Mumbai router hears the IXP route. Every router in IndiaISP, including ones in Delhi that touch no other AS, needs to know both so that it can make the same best-path choice. iBGP carries those externally learned routes across the AS.
Why not just inject the million-route BGP table into OSPF? Because IGPs are built for a few thousand internal routes, and they would lose BGP's attributes (AS path, LOCAL_PREF, communities). So the division of labour is:
- The IGP (OSPF/IS-IS) carries the AS's internal addresses, including each router's loopback, so routers can reach each other.
- iBGP carries the external routes between those loopbacks, with all attributes intact.
What changes on iBGP
| eBGP | iBGP | |
|---|---|---|
| Neighbours | Different AS | Same AS |
| AS path | Sender prepends its ASN | Unchanged (same AS) |
LOCAL_PREF | Not sent | Sent, so the whole AS agrees on preference |
NEXT_HOP | Rewritten to the sender's address | Unchanged by default, so the external next hop must be reachable via the IGP, or the border router sets next-hop-self |
| Re-advertising | Routes from eBGP go to iBGP and eBGP neighbours | Routes from iBGP are not sent to other iBGP neighbours |
| Default IP TTL | 1 (directly connected) | 255 (can be many hops away) |
That last rule is the important one. On eBGP, the AS path catches loops. On iBGP, the AS path doesn't change, so it can't catch loops. BGP's answer is a blunt rule: a route learned over iBGP is never passed to another iBGP neighbour. That only works if every router has an iBGP session to every other router, a full mesh, so each one hears every external route directly from the border router that learned it.
The full-mesh problem
A full mesh of n routers needs n(n − 1)/2 sessions. 5 routers need 10. 100 routers need 4,950. 1,000 routers need 499,500. Adding one router means reconfiguring every other router. That doesn't scale.
Route reflectors
A route reflector (RR, RFC 4456) relaxes the rule. Routers become clients of a reflector. The reflector is allowed to pass routes learned from one client to all its other clients. Each client needs only one or two sessions, to the reflectors.
flowchart TB
subgraph Mesh["Full mesh, 5 routers = 10 iBGP sessions"]
M1["R1"] --- M2["R2"]
M1 --- M3["R3"]
M1 --- M4["R4"]
M1 --- M5["R5"]
M2 --- M3
M2 --- M4
M2 --- M5
M3 --- M4
M3 --- M5
M4 --- M5
end
subgraph Reflect["Route reflector, 5 routers = 4 iBGP sessions"]
RR["Route reflector"] --- C1["Client R1<br/>eBGP to TransitB"]
RR --- C2["Client R2<br/>eBGP to IXP"]
RR --- C3["Client R3"]
RR --- C4["Client R4"]
end
Two attributes stop reflected routes from looping: ORIGINATOR_ID (the router that first put the route into iBGP, which ignores it if it comes back) and CLUSTER_LIST (the reflector clusters the route has passed through). Large networks run at least two reflectors per cluster for redundancy.
Deeper: the costs of route reflection
A reflector passes on only its own best path, chosen from its own position in the network. A client in Chennai may receive the path that's best from the reflector's point of view in Mumbai, and lose hot-potato routing. Fixes include placing reflectors carefully, BGP ADD-PATH (sending several paths), and "optimal route reflection" (RFC 9107), where the reflector computes the best path from the client's position. Reflection can also cause persistent oscillation in rare MED setups (RFC 3345). The other scaling tool, confederations (RFC 5065), splits one AS into private sub-ASes that run eBGP-like sessions with each other but look like one AS to the outside. Route reflectors are far more common.
Convergence and Failures
Convergence is the time from a change (a link fails, a prefix is withdrawn) until every router has settled on a new stable choice. Inside one AS, OSPF converges in well under a second. Across the internet, BGP can take minutes. Here is why.
Failure 1: a link goes down
Back to the toy internet. The IXP link between ContentCo and IndiaISP fails. The BGP session drops (immediately if the interface goes down, or after the hold time if it fails silently), and each side discards every route learned from the other. IndiaISP's best route for 203.0.113.0/24 disappears, so it falls back to its stored backup, the provider route via TransitB. Then it tells CampusNet. Simulation output, Scenario B:
| From | To | Message | Prefix | AS_PATH | Receiver does |
|---|---|---|---|---|---|
| AS64530 | AS64540 | UPDATE | 203.0.113.0/24 | 64530 64520 64510 64500 | stores (from provider, LOCAL_PREF 100), best changed |
| AS | Prefix | Candidate paths | Best path | Decided by |
|---|---|---|---|---|
| AS64500 ContentCo | 203.0.113.0/24 | (origin) [LP 200] | (origin) | only route |
| AS64510 TransitA | 203.0.113.0/24 | 64500 [LP 200] | 64500 | only route |
| AS64520 TransitB | 203.0.113.0/24 | 64510 64500 [LP 150] | 64510 64500 | only route |
| AS64530 IndiaISP | 203.0.113.0/24 | 64520 64510 64500 [LP 100] | 64520 64510 64500 | only route |
| AS64540 CampusNet | 203.0.113.0/24 | 64530 64520 64510 64500 [LP 100], 64520 64510 64500 [LP 100] | 64520 64510 64500 | shortest AS_PATH |
Packet AS64530 -> 203.0.113.10: AS64530 (matches 203.0.113.0/24) -> AS64520 (matches 203.0.113.0/24) -> AS64510 (matches 203.0.113.0/24) -> AS64500 (delivered, owns 203.0.113.0/24)
Packet AS64540 -> 203.0.113.10: AS64540 (matches 203.0.113.0/24) -> AS64520 (matches 203.0.113.0/24) -> AS64510 (matches 203.0.113.0/24) -> AS64500 (delivered, owns 203.0.113.0/24)
Three things to notice:
- IndiaISP's message to CampusNet is an UPDATE, not a WITHDRAW. A new announcement for the same prefix implicitly withdraws the old one.
- CampusNet now has a 4-hop path via IndiaISP and a 3-hop path via TransitB. It switches to TransitB directly.
- IndiaISP now pays TransitB for traffic that used to cross the IXP for free. Peering links save real money, which is why their failure shows up on invoices.
Failure 2: the owner withdraws the prefix, and path hunting
Now something worse: ContentCo stops announcing 203.0.113.0/24 altogether, as Facebook effectively did in 2021. You might expect the withdrawal to sweep across the network in one wave. Watch what CampusNet does instead (simulation, Scenario C):
| Round | From | To | Message | AS_PATH | Receiver does |
|---|---|---|---|---|---|
| 1 | AS64500 | AS64510 | WITHDRAW | - | removes route, best changed |
| 1 | AS64500 | AS64530 | WITHDRAW | - | removes route, best changed |
| 2 | AS64510 | AS64520 | WITHDRAW | - | removes route, best changed |
| 2 | AS64530 | AS64540 | UPDATE | 64530 64520 64510 64500 | stores (from provider, LOCAL_PREF 100), best changed |
| 3 | AS64520 | AS64530 | WITHDRAW | - | removes route, best changed |
| 3 | AS64520 | AS64540 | WITHDRAW | - | removes route, best changed |
| 4 | AS64530 | AS64540 | WITHDRAW | - | removes route, best changed |
CampusNet's best path, round by round:
| After round | CampusNet's candidates | Best | Reality |
|---|---|---|---|
| 0 | 64530 64500, 64520 64510 64500 | 64530 64500 (2 hops) | Valid |
| 1 | same, it hasn't heard anything yet | 64530 64500 | Stale, the origin is gone |
| 2 | 64530 64520 64510 64500, 64520 64510 64500 | 64520 64510 64500 (3 hops) | Stale |
| 3 | 64530 64520 64510 64500 | 64530 64520 64510 64500 (4 hops) | Stale |
| 4 | none | no route | Correct at last |
Packet AS64540 -> 203.0.113.10: AS64540 (no route, dropped)
This is path hunting. When a prefix disappears, routers don't know it's gone everywhere. They only know their best path vanished, so they fall back to the next-best path they've stored, which is usually another path to the same dead origin that hasn't been withdrawn yet. Each fallback is announced, which triggers more fallbacks downstream. A router can try every possible path, each longer than the last, before the final withdrawal reaches it. With 5 ASes it took 4 rounds. On the real internet, with many alternative paths, it takes many more.
Why BGP convergence takes minutes
| Factor | What it does | Typical value |
|---|---|---|
| Failure detection | A silent failure (the link stays up but packets stop) is only noticed when the hold timer expires | 90 to 180 s without BFD, under 1 s with BFD |
| MRAI timer | Minimum Route Advertisement Interval: a router waits before sending another update for the same destination, to batch changes | RFC 4271 suggests 30 s for eBGP, 5 s for iBGP. Many modern implementations use 0 for both |
| Path hunting | Each fallback costs another round of updates, and each round may wait for MRAI | Multiplies MRAI by the number of alternative paths |
| Processing load | A big event (a Tier 1 session reset) means hundreds of thousands of prefix changes | Seconds to minutes of CPU on older routers |
| FIB programming | Pushing changes into forwarding hardware | Can lag the RIB by seconds for large changes |
Labovitz, Ahuja, Bose and Jahanian measured this in "Delayed Internet Routing Convergence" (SIGCOMM 2000). They found that withdrawals and failovers commonly took minutes to converge across the internet, and explained path hunting as the cause. Things are faster now, but a failure deep in the internet still takes tens of seconds to minutes to settle everywhere.
Route flap dampening
A flap is a route that keeps disappearing and coming back, for example because a link keeps bouncing. Each flap sends updates across the whole internet. Route flap dampening (RFC 2439) gives each prefix a penalty that rises with each flap and decays over time. Above a threshold, the router suppresses the route until the penalty decays. The idea is to protect everyone else from one unstable network.
Dampening turned out to hurt more than it helped. Because path hunting makes even one real withdrawal look like several flaps, dampening with the original aggressive defaults suppressed healthy prefixes for up to an hour after a single event. RIPE recommended against it in 2006 (RIPE-378), then, with RFC 7196 raising the suppress threshold, recommended it again with much gentler parameters (RIPE-580, 2013). If you meet it, check which parameters are in use.
Tools that make failures less painful
- BFD detects a dead neighbour in milliseconds instead of minutes.
- Graceful restart (RFC 4724) keeps forwarding with old routes while a neighbour's BGP process restarts, instead of withdrawing everything.
- BGP PIC (prefix-independent convergence) pre-computes backup paths in the FIB so a failure switches millions of prefixes at once.
- ADD-PATH lets iBGP routers know a backup before they need it.
BGP Security Problems
BGP was designed in 1989 on the assumption that operators would behave honestly. They often don't — accidentally or maliciously.
The core problem, in plain words: a BGP router believes what its neighbours tell it. Nothing in the protocol proves that the origin AS actually holds the prefix it announces, or that the AS path is real. Remember the postal service that says "send me all mail for Brazil". Most of the time the cause is a typo or a broken filter, not malice. The effect is the same either way.
This section is defensive. It explains how incidents happen so you can recognise them, and how operators prevent them.
The kinds of incident
| Incident | What happens | Why it works | Example |
|---|---|---|---|
| Origin hijack (exact prefix) | AS X announces a prefix that belongs to AS Y, same length | Networks closer to X may prefer X's shorter or cheaper path | Part of the 2010 China Telecom incident |
| Sub-prefix hijack | AS X announces a more specific prefix inside Y's block | Longest-prefix match: the more-specific wins everywhere it's accepted | Pakistan/YouTube 2008, MyEtherWallet 2018 |
| Route leak | An AS passes routes to a neighbour in violation of the export rules, such as provider routes to another provider | Neighbours prefer customer routes, so leaked routes get high LOCAL_PREF | Verizon/Allegheny 2019 |
| Path manipulation | A forged AS path, for example prepending the real origin to pass origin checks | No protocol check on the path itself | Defended against by BGPsec and ASPA, below |
| Accidental withdrawal | An AS stops announcing its own prefixes | Withdrawals propagate as fast as announcements | Facebook 2021, Cloudflare BYOIP 2026 |
BGP Hijacking
An AS announces a prefix it doesn't own. Other routers accept it because BGP has no built-in authentication.
2010 China Telecom incident: AS23724 (China Telecom) briefly advertised routes for 50,000+ prefixes — including Google, YouTube, and major US networks. Traffic was rerouted through China for 18 minutes.
Correction (2026-11-13)
Counts for the April 8, 2010 incident vary between analyses: BGPmon, which monitored it, reported roughly 37,000 prefixes, and other reports give larger figures, so "50,000+" should be read as an upper estimate. Only some networks accepted the routes, so "traffic was rerouted through China" applies to the share of traffic from those networks, not to all traffic for those prefixes. The roughly 18-minute duration is consistent across reports. It is usually classed as an accidental leak or mis-origination; no public evidence shows it was deliberate.
2018 MyEtherwallet attack: AS10297 hijacked AWS Route 53 IP addresses for 2 hours. Users were redirected to a fake MyEtherWallet site that stole ~$150K in cryptocurrency.
BGP Route Leaks
An AS re-advertises routes it received from one peer to another peer, unintentionally becoming a transit provider.
2019 Cloudflare incident: A small Pennsylvania ISP (Verizon customer) leaked routes from DQE Communications to Verizon. Verizon propagated them globally. Traffic for Cloudflare, Amazon, and others routed through the ISP's tiny pipe. 30+ minutes of widespread outages.
Correction (2026-11-13)
The leaking network wasn't an ISP. It was Allegheny Technologies (AS396531), a metals company in Pittsburgh, Pennsylvania, which bought connectivity from both DQE Communications (AS33154) and Verizon (AS701). DQE ran a "BGP optimizer" that generated more-specific routes for prefixes from other networks, and passed them to Allegheny, which passed them to Verizon. Verizon accepted them without filtering and propagated them. Because they were more-specific, they beat the legitimate routes everywhere they spread. Cloudflare's write-up describes about 20,000 prefixes from about 2,400 networks being affected. The impact lasted well over 30 minutes; analyses describe it as lasting roughly two hours on June 24, 2019. It's better called "the 2019 Verizon/Allegheny route leak": Cloudflare was a prominent victim, not the cause. Source: Cloudflare, How Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline Today.
Real incidents: hijacks and leaks
Three well-documented incidents, each showing a different mechanism.
Pakistan Telecom and YouTube, February 24, 2008. Pakistan's government ordered YouTube blocked inside the country. Pakistan Telecom (AS17557) did it by announcing 208.65.153.0/24, a more-specific of YouTube's 208.65.152.0/22, intending to route that traffic into a black hole internally. The announcement leaked to its upstream provider PCCW (AS3491), which didn't filter it and passed it to the world. Because the /24 was more specific than YouTube's /22, it won by longest-prefix match on every network that accepted it, and YouTube traffic from much of the internet flowed to Pakistan and was dropped. YouTube responded about 80 minutes later by announcing the same /24, then two even more specific /25s to win the traffic back. PCCW then withdrew the route. Total disruption was about two hours. Source: RIPE NCC, YouTube Hijacking: A RIPE NCC RIS case study.
Mechanism: sub-prefix hijack plus a provider with no customer filter. Lesson: providers must filter what their customers announce.
MyEtherWallet and Amazon Route 53, April 24, 2018. For about two hours, eNet (AS10297), a provider in Columbus, Ohio, announced more-specific /24 prefixes covering part of the address space used by Amazon Route 53's authoritative DNS servers. Some large networks, including one used by Google Public DNS, accepted them. DNS queries for myetherwallet.com that went to the hijacked addresses reached impostor DNS servers, which answered with the address of a phishing server. That server presented a TLS certificate browsers didn't trust, and users who clicked through the warning had their wallets drained. Reported losses were around $150,000 in Ether. Source: Cloudflare, BGP leaks and cryptocurrencies.
Mechanism: sub-prefix hijack aimed at DNS, chained with phishing. Lessons: a BGP hijack of your DNS provider is a hijack of your domain. TLS still worked as designed: the attackers couldn't get a valid certificate. HSTS and users not clicking through warnings are part of the defence. DNSSEC-validating resolvers would also have rejected forged answers for signed zones.
Verizon and Allegheny Technologies, June 24, 2019. Covered in the correction above. Mechanism: a route leak of more-specifics generated by an optimizer, through a multihomed customer, to a Tier 1 that didn't filter its customer. Lessons: never let route-optimizer more-specifics leave your network. Tier 1s must filter customers by prefix lists and max-prefix limits. RPKI ROV would have rejected many of the leaked routes as invalid where maxLength didn't allow them.
RPKI — The Fix
RPKI (Resource Public Key Infrastructure) is the modern solution. Network operators cryptographically sign their IP prefix announcements, associating prefixes with their ASN. Routers can validate: "Is AS15169 actually authorized to originate 8.8.8.0/24?"
Route Origin Authorization (ROA):
Prefix: 8.8.8.0/24
Origin: AS15169
Signed by: Google's RPKI certificate
Cloudflare, AWS, and major Tier 1s have deployed RPKI. Adoption is ~50% as of 2025 — enough to block most accidental hijacks, not yet universal.
Correction (2026-11-13)
Two different numbers get called "RPKI adoption", and the sentence above blurs them. ROA coverage is the share of announced prefixes that have a ROA. It passed roughly 50% of IPv4 prefixes around 2024 (see the NIST RPKI Monitor). ROV enforcement is the share of networks that actually drop invalid routes, and that's what protects you. Many large transit networks enforce ROV, which limits how far invalid routes spread, but plenty of networks still don't. Also, ROAs and ROV only check the origin AS, so even full deployment wouldn't stop every hijack (see "What ROV does not stop" below).
RPKI step by step
RPKI answers one question: "Is this AS allowed to originate this prefix?" It works like the certificate system behind HTTPS, but for address space.
Step 1: a chain of trust that mirrors address allocation. The five RIRs are the trust anchors. When an RIR allocates 203.0.113.0/24 to ContentCo, it can also issue ContentCo a resource certificate that says, cryptographically, "the holder of this key holds 203.0.113.0/24."
Step 2: the holder signs a ROA. ContentCo uses that certificate to sign a Route Origin Authorization (ROA): a small signed object with three fields.
| Field | Meaning | ContentCo's value |
|---|---|---|
| Prefix | The address block | 203.0.113.0/24 |
maxLength | The longest prefix that may be announced from this block | 24 |
| Origin ASN | The AS allowed to originate it | AS64500 |
maxLength matters. A ROA for 203.0.113.0/24 with maxLength 24 means "only the /24 itself". If ContentCo set maxLength 25, the /25s would also be authorised, and a hijacker who forged the origin could announce a /25 that passes validation. RFC 9319 recommends keeping maxLength equal to the prefix length unless you actually announce the more-specifics.
Step 3: ROAs are published. Each RIR (or the holder, if it runs its own CA) publishes certificates and ROAs in public repositories.
Step 4: validators fetch and check them. Each network runs relying-party software (validators such as Routinator, rpki-client or FORT). It downloads every repository, checks every signature back to the trust anchors, and produces a list of validated ROA payloads: (prefix, maxLength, ASN) triples.
Step 5: routers get the list. Validators feed that list to routers over the RPKI-to-Router (RTR) protocol (RFC 8210). Routers do no cryptography themselves. They just compare each route against the list.
Step 6: Route Origin Validation (ROV). For every route received, the router assigns one of three states, defined in RFC 6811:
| State | Rule | Example with ContentCo's ROA (203.0.113.0/24, max 24, AS64500) | Typical action |
|---|---|---|---|
| Valid | A ROA covers the prefix, the origin matches, and the length is within maxLength | 203.0.113.0/24 from AS64500 | Accept |
| Invalid | At least one ROA covers the prefix, but none matches both origin and length | 203.0.113.0/25 from AS64666 (wrong origin and too long). 203.0.113.0/24 from AS64666 (wrong origin). 203.0.113.0/25 from AS64500 (too long) | Drop |
| NotFound | No ROA covers the prefix at all | 192.0.2.0/24 from anyone | Accept (no information) |
The simulation's rov() function implements exactly this, and prints:
ROV states: {'203.0.113.0/24': 'valid', '203.0.113.0/25': 'invalid', '192.0.2.0/24': 'not-found'}
flowchart LR
RIR["RIR trust anchor<br/>e.g. APNIC"] -->|"issues resource certificate"| CA["ContentCo certificate<br/>holds 203.0.113.0/24"]
CA -->|"signs"| ROA["ROA<br/>203.0.113.0/24, max 24, AS64500"]
ROA --> Repo["Public repository"]
Repo -->|"fetch and verify signatures"| RP["Validator<br/>Routinator, rpki-client"]
RP -->|"RTR protocol, validated list"| Rtr["Border router"]
UPD["Incoming UPDATE"] --> Rtr
Rtr -->|"Valid or NotFound"| Acc["Accept, run best-path"]
Rtr -->|"Invalid"| Drop["Drop"]
What RPKI changes in the toy internet
Add a sixth AS to the toy internet: Misconfig (AS64666), another customer of TransitB. Through a typo, it announces 203.0.113.0/25, half of ContentCo's space. ContentCo has published the ROA above. The simulation runs three versions (Scenario D).
Version 1: no ROV anywhere. TransitB treats it as a customer route (LOCAL_PREF 200) and sends it to everyone, as the export rules allow:
| Round | From | To | AS_PATH | Receiver does |
|---|---|---|---|---|
| 1 | AS64666 | AS64520 | 64666 | stores (from customer, LOCAL_PREF 200), best changed |
| 2 | AS64520 | AS64510 | 64520 64666 | stores (from peer, LOCAL_PREF 150), best changed |
| 2 | AS64520 | AS64530 | 64520 64666 | stores (from provider, LOCAL_PREF 100), best changed |
| 2 | AS64520 | AS64540 | 64520 64666 | stores (from provider, LOCAL_PREF 100), best changed |
| 3 | AS64510 | AS64500 | 64510 64520 64666 | stores (from provider, LOCAL_PREF 100), best changed |
| 3 | AS64530 | AS64540 | 64530 64520 64666 | stores (from provider, LOCAL_PREF 100) |
Every AS keeps its /24 route too. BGP didn't choose between them, because they're different prefixes. The FIB does, per packet:
Packet AS64530 -> 203.0.113.10: AS64530 (matches 203.0.113.0/25) -> AS64520 (matches 203.0.113.0/25) -> AS64666 (delivered, owns 203.0.113.0/25)
Packet AS64530 -> 203.0.113.200: AS64530 (matches 203.0.113.0/24) -> AS64500 (delivered, owns 203.0.113.0/24)
Your laptop in IndiaISP has a perfectly good peering route to ContentCo, and still sends traffic for 203.0.113.10 to the wrong network, because /25 beats /24.
Version 2: ROV only at TransitA. TransitA drops the invalid route (rejects: RPKI invalid), so it never reaches TransitA or ContentCo. But TransitB already accepted it and gave it to its own customers.
Version 3: ROV at TransitB, the hijacker's provider. TransitB drops it in round 1 and nobody else ever hears of it.
Summary, from the simulation's packet traces for 203.0.113.10:
| Source AS | No ROV | ROV at TransitA only | ROV at TransitB |
|---|---|---|---|
| AS64510 TransitA | Hijacked, to AS64666 | Correct, to AS64500 | Correct |
| AS64520 TransitB | Hijacked | Hijacked | Correct |
| AS64530 IndiaISP | Hijacked | Hijacked | Correct |
| AS64540 CampusNet | Hijacked | Hijacked | Correct |
The lesson: ROV protects the network that runs it and everyone downstream of it. It's most effective at the edge, where routes enter the system, and at large transit networks, where many paths pass. Every network that deploys it shrinks the area a bad route can reach.
What ROV does not stop
- Forged origins. ROV checks only the rightmost ASN. An attacker who announces
203.0.113.0/24with path64666 64500claims the right origin and is "Valid". The path is fake, but ROV can't tell. - Route leaks of valid routes. In a leak, the origin is genuine. The problem is the path, which ROV doesn't check.
- Prefixes with no ROA ("NotFound") get no protection at all.
BGPsec and ASPA, briefly
Two mechanisms aim at the path itself.
- BGPsec (RFC 8205) has each AS cryptographically sign the path as it passes the route on, so the full AS path can be verified. It's standardised but practically undeployed: every router must sign and verify in real time, the signatures make updates much larger, and it only helps when every AS on a path supports it.
- ASPA (Autonomous System Provider Authorization) is lighter. Each AS publishes an RPKI object listing its providers. A router can then check whether an AS path is valley-free: if a path goes "down" from a provider to a customer and then back "up", someone leaked. ASPA catches many leaks and forged-path hijacks without per-update cryptography. As of late 2026 it is still an IETF draft (draft-ietf-sidrops-aspa-verification and its companion profile draft). Several RIRs and validators have added support for creating and validating ASPA objects, and router support is arriving, but enforcement on the internet is at an early stage.
Other everyday defences
| Defence | What it stops |
|---|---|
| Prefix filters from IRR data | Customers announcing prefixes they don't hold. Built from route objects in Internet Routing Registries such as RADb or RIPE's database |
| Max-prefix limits | A neighbour suddenly sending far more routes than usual (a full-table leak). The session shuts down instead |
| RFC 8212 default-deny | Sessions with no policy accidentally passing everything |
| Peer locking | Large networks refusing to accept routes containing other large networks' ASNs from anyone but those networks directly |
| TCP-AO / TCP-MD5, and GTSM | Spoofed packets resetting or injecting into BGP sessions. GTSM (RFC 5082) only accepts BGP packets with TTL 255, which can't come from far away |
| Monitoring | Services like Cloudflare Radar, bgp.tools and RIPE RIS alert when your prefixes appear with an unexpected origin or path |
MANRS
MANRS (Mutually Agreed Norms for Routing Security, manrs.org) is an industry initiative launched in 2014 under the Internet Society, and transferred to the Global Cyber Alliance in 2024. Networks that join commit to a short set of concrete actions:
- Filtering: prevent propagation of incorrect routing information, both your own and your customers'.
- Anti-spoofing: block packets with forged source addresses (BCP 38).
- Coordination: keep contact details current so others can reach you during an incident.
- Global validation: publish your routing data (IRR objects and ROAs) so others can validate your announcements.
There are separate programmes for network operators, IXPs, CDNs and cloud providers, and equipment vendors. MANRS doesn't add protocol features. It turns known good practice into a public commitment.
BGP in the Cloud and the Data Centre
BGP isn't only for ISPs. If you run anything at scale, it is probably under your feet.
BGP and Anycast
Anycast is the killer feature that CDNs and DNS providers use to serve users from the nearest location.
The idea: announce the same IP prefix from multiple locations globally. BGP's shortest-path routing automatically directs users to the nearest data center.
Cloudflare announces 1.1.1.0/24 from:
- San Jose
- Frankfurt
- Singapore
- São Paulo
User in Germany → BGP routes to Frankfurt (shortest path)
User in Australia → BGP routes to Singapore
This is how Cloudflare's DDoS mitigation works — attack traffic is absorbed at the nearest PoP rather than overwhelming a single datacenter. Each PoP can absorb traffic independently.
DNS anycast: 8.8.8.8 (Google DNS) and 1.1.1.1 (Cloudflare DNS) use anycast. Queries go to the nearest Google/Cloudflare server automatically.
Correction (2026-11-13)
BGP doesn't know geography, so anycast sends users to the BGP-nearest site, not the geographically nearest one. "Nearest" means "the site whose announcement won best-path selection in the user's ISP", decided by LOCAL_PREF (peer routes beat transit routes), then AS path length, then hot-potato IGP cost. A user in Germany usually lands in Frankfurt, but if their ISP only reaches the operator through a transit provider that hands traffic off in Amsterdam, they land in Amsterdam. Anycast operators spend real effort finding and fixing such cases. The city list above is illustrative: Cloudflare announces 1.1.1.0/24 from all of its data centres (hundreds of cities, including several in Australia), so an Australian user normally stays in Australia.
How one address lives in many places
Go back to the toy internet and imagine ContentCo has two sites, one in Mumbai and one in Frankfurt, each with its own border router, and both announce 203.0.113.0/24 with origin AS64500. Every AS in the world now hears two (or more) routes for the same prefix with the same origin, and picks one using the normal decision table. An ISP in India picks the Mumbai announcement because it arrives over a peering link. An ISP in Germany picks Frankfurt. Neither knows the other site exists. The same address, 203.0.113.10, now reaches different machines depending on where you ask from.
flowchart LR
subgraph Asia["India"]
U1["User, IndiaISP"] --> I1["IndiaISP AS64530"]
end
subgraph Europe["Germany"]
U2["User, German ISP"] --> G1["German ISP AS64550"]
end
I1 -->|"best path: peer route"| M["ContentCo Mumbai<br/>announces 203.0.113.0/24"]
G1 -->|"best path: peer route"| F["ContentCo Frankfurt<br/>announces 203.0.113.0/24"]
I1 -.->|"backup path via transit"| F
G1 -.->|"backup path via transit"| M
Questions people ask:
- Doesn't TCP break if packets go to different sites? It would, if routes changed mid-connection. In practice a user's best path is stable for minutes to hours, so a TCP connection stays at one site. Anycast is used most for short exchanges (DNS over UDP, HTTP requests). Operators who run long connections over anycast use extra tricks, such as forwarding stray packets between sites.
- How does a site leave the anycast group? It withdraws its announcement. Traffic shifts to the next-best site within seconds to minutes, as networks converge. That's also how operators drain a site for maintenance, and how a site automatically drops out when its health checks fail.
- Why does anycast help with DDoS? Attack traffic from bots spread across the world is split across all the sites, by the same routing that splits legitimate traffic. No single site sees the full attack.
CDNs and global load balancing
Content delivery networks use BGP in two ways. Some (Cloudflare, Fastly's edge, parts of Google) use anycast, so the same IP addresses everywhere and BGP picks the site. Others use DNS-based steering: the CDN's DNS server returns a different unicast IP depending on where the resolver is. Many combine both. See CDN Deep Dive and Global Load Balancing for the trade-offs.
BGP in Cloud Networking
AWS Direct Connect
BGP sessions between your on-premises routers and AWS. You advertise your on-prem prefixes; AWS advertises VPC prefixes.
In more detail: a Direct Connect virtual interface (VIF) is a VLAN on the physical link, and each VIF carries one eBGP session (per address family). You pick your side's ASN (private ASNs are fine). The AWS side uses the ASN of the virtual private gateway or Direct Connect gateway, which defaults to 64512 unless you set one. MD5 authentication is supported. On a private VIF, AWS documents a limit of 100 prefixes you may advertise per session. Exceed it and the session goes down, which surprises teams that announce many small subnets. Summarise into fewer, larger prefixes. Check the AWS Direct Connect quotas page for current numbers. For path control, AWS honours BGP communities for local preference and AS-path prepending on your announcements, which is how you make one Direct Connect link primary and another backup.
AWS Transit Gateway with BGP
Used for hub-and-spoke VPC connectivity at scale. BGP dynamically propagates routes between VPCs and on-premises networks.
Correction (2026-11-13)
Transit Gateway uses BGP on its VPN, Direct Connect gateway, and Connect (GRE tunnel plus BGP, for SD-WAN appliances) attachments. VPC attachments don't run BGP: a VPC's CIDRs are propagated into Transit Gateway route tables by AWS's own route propagation. So "BGP propagates routes between VPCs" is inaccurate. BGP brings on-premises and appliance routes in, and Transit Gateway's route tables do the rest. See AWS VPC Networking.
Kubernetes (MetalLB, Cilium)
BGP-based load balancer controllers advertise pod/service IP ranges from Kubernetes nodes. Used for bare-metal Kubernetes to integrate with the datacenter's BGP fabric.
How it works, by project:
- MetalLB gives
LoadBalancerservices an external IP on bare metal. In L2 mode, one node answers ARP for the IP (simple, but one node takes all traffic). In BGP mode, every eligible node runs a BGP session to the top-of-rack routers and announces the service IP as a/32. The routers see several equal routes and spread traffic across nodes with ECMP (equal-cost multipath). - Calico can run BIRD on every node and announce pod CIDRs over BGP, so pods are routable without any overlay network. Small clusters use a node-to-node full mesh by default. Larger ones switch to route reflectors, for the same full-mesh reasons as in iBGP, or peer each node with its rack switch.
- Cilium has a BGP control plane that announces pod CIDRs, service IPs and load-balancer IPs to upstream routers, configured with Kubernetes custom resources.
See CNI Networking for how pods get their addresses in the first place.
Cloudflare Workers / CDN Edge
Cloudflare uses BGP anycast for every request. When you set a DNS record to Cloudflare, your traffic is being routed by BGP to the nearest Cloudflare PoP before it ever reaches your origin server.
Customers who bring their own addresses (BYOIP) have Cloudflare announce their prefixes from its anycast network. That puts the customer's reachability in Cloudflare's BGP automation, which is the setting of the February 2026 incident.
BGP as the data-centre fabric
Inside large data centres, the network is usually a Clos (leaf-spine) topology: every rack's top-of-rack switch (a leaf) connects to every spine switch. Any two servers are at most a few hops apart, with many equal-cost paths between them.
You'd expect such a fabric to run OSPF or IS-IS. Many of the largest operators instead run eBGP on every link, as described in RFC 7938, "Use of BGP for Routing in Large-Scale Data Centers", written by engineers from Microsoft and others.
flowchart TB
S1["Spine 1<br/>AS 65100"] --- L1["Leaf 1<br/>AS 65001"]
S1 --- L2["Leaf 2<br/>AS 65002"]
S1 --- L3["Leaf 3<br/>AS 65003"]
S2["Spine 2<br/>AS 65100"] --- L1
S2 --- L2
S2 --- L3
L1 --- H1["Servers in rack 1<br/>10.1.1.0/24"]
L2 --- H2["Servers in rack 2<br/>10.1.2.0/24"]
L3 --- H3["Servers in rack 3<br/>10.1.3.0/24"]
How RFC 7938's design works:
- Each tier gets private ASNs. In a common scheme all spines share one ASN and each leaf has its own. Every link is an eBGP session.
- Leaves announce their rack's prefix. Leaf 1 announces
10.1.1.0/24to both spines. Each spine passes it to the other leaves. Leaf 2 hears it from Spine 1 and Spine 2 with equal AS paths (65100 65001), and installs both with ECMP, spreading traffic across them. - Loops are blocked by the AS path itself. Leaf 1 never accepts a route back from a spine that contains
65001. Because all spines share an ASN, a route can't go leaf → spine → leaf → spine. That's exactly the valley that shouldn't exist in a Clos fabric. - Why BGP, not an IGP? One protocol everywhere, from server to internet edge. BGP's failure is local: a link failure sends a withdrawal along affected paths, rather than flooding link-state changes to every switch. Policy is easy to apply per session. And BGP implementations are mature and scale to huge tables.
- Operational niceties. "BGP unnumbered" runs sessions over IPv6 link-local addresses, with IPv4 routes carried using IPv6 next hops (RFC 8950), so no one has to assign an IPv4 address to each of thousands of links. Some operators also run BGP on the servers themselves, announcing service addresses so that a server can drop out of rotation by withdrawing a route.
Data-centre BGP usually turns on multipath (install several equal paths instead of one best path) and sets aggressive timers or BFD for fast failover. The decision table still applies, with "equal enough" routes all installed instead of being reduced to one.
Case Studies
Both incidents have their own War Room pages. Here is what they teach about BGP specifically.
Case study: Facebook, October 2021
What happened at the BGP level. On October 4, 2021, a maintenance command meant to assess backbone capacity instead disconnected Facebook's backbone, cutting its data centres off from each other and from the internet. Facebook's authoritative DNS servers are built to withdraw their own BGP announcements if they lose contact with the data centres, a safety feature meant to steer traffic away from an unhealthy site. With every data centre unreachable, every DNS site withdrew at once. Within a couple of minutes the prefixes that held Facebook's DNS servers disappeared from routing tables everywhere. Resolvers worldwide could no longer reach facebook.com's name servers, and returned errors. Facebook, Instagram, WhatsApp and Messenger were unreachable for about six hours. Sources: Facebook Engineering, More details about the October 4 outage, and Cloudflare, Understanding How Facebook Disappeared from the Internet, which shows the burst of withdrawals as seen from outside.
What it teaches about BGP:
- A withdrawal is as powerful as an announcement. Nobody attacked anything. A network told the world "you can't reach me", and the world believed it within minutes.
- Anycast health-withdrawal is a double-edged sword. Withdrawing one unhealthy site is good. Withdrawing every site at once, because they all share a failure, is a total outage. Health-based withdrawal needs a floor ("never withdraw the last N sites").
- DNS failure was a symptom. The DNS servers were working. There was no route to them.
- Out-of-band access matters. Remote tools depended on the network that was down.
Full timeline, the audit-tool bug, and the physical-access problems: Facebook's 6-Hour Global Outage (October 2021).
Case study: Cloudflare BYOIP, February 2026
What happened at the BGP level. On February 20, 2026, a new automated cleanup task in Cloudflare's addressing system sent an API query flag with no value. The server read it as "no filter" and returned every Bring-Your-Own-IP prefix, and the task began deleting them. Over about 50 minutes, roughly 1,100 customer prefixes were withdrawn from BGP, and some also lost the edge configuration that told Cloudflare's servers what to do with their traffic. Impact lasted a little over six hours, mostly spent rebuilding deleted state rather than re-announcing routes.
What it teaches about BGP:
- Withdrawal is cheap to undo. Deletion isn't. Prefixes that were only withdrawn could be re-announced from a dashboard in seconds. Prefixes whose configuration was deleted needed a global rollout.
- Automation that controls announcements needs rate limits. A circuit breaker on "withdrawn more than N prefixes in M minutes" would have stopped the job early.
- Monitor what the internet sees. The first signal was a broken website, not an alert on "announced prefixes dropped by 25%".
- Anycast amplifies both ways. One API call changed routes at every Cloudflare location at once, because every location announces the same prefixes.
Full root cause, recovery states and the engineering controls that would have prevented it: Cloudflare BYOIP Outage (February 2026).
The common thread
Both outages were self-inflicted withdrawals caused by automation acting on its own announcements, and neither involved an attacker. RPKI wouldn't have helped either one. The defences are operational: staged rollouts, sanity limits on withdrawals, health checks with floors, external monitoring of your own prefixes, and out-of-band access.
Hands-On: Look at Real BGP
You don't need a router to see real BGP data. Public route collectors peer with hundreds of networks and publish what they see.
BGP Looking Glass
You can inspect live BGP routes without running a router:
- BGP.he.net — Hurricane Electric's BGP toolkit
- RIPE RIS — real-time BGP data from 900+ peers
- Route Views — University of Oregon BGP archive
- Cloudflare Radar — BGP routing changes and outages in near real-time
# Check BGP path to an IP (requires a BGP-capable system)
# Or use online looking glass tools
The RIPE RIS peer count changes over time, so treat "900+" as approximate and check the RIS site for the current number. A few more tools:
- bgp.tools: fast, readable pages for any prefix (
https://bgp.tools/prefix/1.1.1.0/24) or AS (https://bgp.tools/as/13335), showing upstreams, downstreams, peers, RPKI status and which IXPs a network is present at. - RIPEstat: RIPE NCC's analysis portal, with a JSON API.
- Looking glasses run by individual networks: web pages where you type
show ip bgp 1.1.1.1and see that network's own view. Most large ISPs publish one. Lists exist on bgp.tools and PeeringDB. - PeeringDB: which networks are at which IXPs and data centres, and their peering policies.
Commands to try
Each command below is real. Run them yourself. The output changes as the internet changes, so it isn't reproduced here.
# 1. Which AS announces an IP, and which prefix covers it (Team Cymru's IP-to-ASN service)
whois -h whois.cymru.com " -v 1.1.1.1"
# 2. The same from bgp.tools' whois interface
whois -h bgp.tools " -v 1.1.1.1"
# 3. Route objects registered in RADb (an Internet Routing Registry) for one AS
whois -h whois.radb.net -- '-i origin AS13335'
# 4. The aut-num object (contact and policy notes) for an AS
whois -h whois.radb.net AS13335
# 5. Traceroute with AS numbers for every hop
mtr -z -r -c 5 1.1.1.1 # -z: look up the AS of each hop, -r -c 5: report after 5 rounds
traceroute -A 1.1.1.1 # Linux traceroute: -A does AS path lookups
# 6. RIPEstat's JSON API
curl -s "https://stat.ripe.net/data/routing-status/data.json?resource=1.1.1.0/24"
curl -s "https://stat.ripe.net/data/rpki-validation/data.json?resource=AS13335&prefix=1.1.1.0/24"
curl -s "https://stat.ripe.net/data/looking-glass/data.json?resource=1.1.1.0/24"
# 7. A public route server with a router-style CLI (University of Oregon Route Views)
telnet route-views.routeviews.org
# then, at the prompt: show ip bgp 1.1.1.0/24
# 8. If you run FRRouting or BIRD yourself
vtysh -c "show bgp ipv4 unicast 1.1.1.0/24"
birdc show route for 1.1.1.1 all
What each one tells you:
- Origin ASN, covering prefix, registry and AS name for an address.
- The same, from a different data source. Disagreement is itself interesting.
- Which prefixes an AS has registered as originating. Compare with what it actually announces (from bgp.tools or RIPEstat) to spot gaps.
- Who runs the AS and any policy they've published.
- Which networks your packets actually cross. Watch where the AS changes: that's where traffic crosses an eBGP boundary. Some hops show no AS (private or unannounced addresses) or the AS of the other side of a link, because router interfaces on a link are often numbered from one side's space.
- Whether a prefix is visible, from how many collectors, its RPKI status, and the AS paths RIS peers see.
- A real BGP table with hundreds of thousands of routes, from many peers' points of view. Each line shows next hop, MED, LOCAL_PREF and AS path for one candidate, with the best path marked.
- Your own router's view, with all attributes and why one route was chosen.
Reading a real AS path
When a tool shows an AS path, read it like this. The example uses documentation ASNs and is for illustration, not output from a real tool:
Prefix 198.51.100.0/24 AS path: 64496 64511 64511 64511 65551
│ │ │
│ │ └ origin: the AS that announced it
│ └ the same AS three times: prepending,
│ this AS wants the path to look longer
└ the neighbour of the collector that reported this path
Questions to ask of any path:
- Is the origin who you expect? Compare with the ROA (RIPEstat's
rpki-validation) and with the AS's registered route objects. An unexpected origin is how hijacks are spotted. - Is there prepending? It tells you an operator is steering traffic away from that path.
- Is it valley-free? If a small regional network appears between two large transit networks, that is often a leak. bgp.tools and Cloudflare Radar flag suspected leaks automatically.
- How many different paths do different collectors see? Many paths means a well-connected prefix. One path means a single upstream: a single point of failure.
Runnable Simulation: Best-Path and Propagation
This is the program that produced every trace table on this page. It's standard-library Python, about 300 lines, and runs in a fraction of a second. Save it as bgp_sim.py and run python bgp_sim.py. It prints all the tables as Markdown, and ends with assert checks that fail loudly if the logic is broken.
It models each AS as a single router (so no iBGP), exchanges messages in synchronous rounds (real BGP is asynchronous, with timers), and uses the customer/peer/provider policy from the toy internet. Within those limits, it really does run BGP's algorithm: loop rejection by AS path, valley-free export, the ordered decision process with MED compared only within the same neighbour AS, implicit and explicit withdrawals, longest-prefix match in forwarding, and RFC 6811 origin validation.
"""Toy BGP: announcement propagation, valley-free export, best-path selection,
withdrawal, a more-specific leak, and RPKI route origin validation.
Run: python bgp_sim.py (Python 3.10+, standard library only)
It prints the trace tables used on the Archon BGP page.
"""
from dataclasses import dataclass
from ipaddress import ip_address, ip_network
CUSTOMER, PEER, PROVIDER = "customer", "peer", "provider"
# Money decides LOCAL_PREF: customers pay me, peers are free, providers cost me.
LOCAL_PREF = {CUSTOMER: 200, PEER: 150, PROVIDER: 100}
ORIGIN_RANK = {"IGP": 0, "EGP": 1, "INCOMPLETE": 2}
@dataclass(frozen=True)
class Route:
prefix: str
as_path: tuple # leftmost = the neighbour that sent it, rightmost = origin
learned_from: int | None # neighbour ASN, None if we originated it
rel: str | None # what that neighbour is to us
local_pref: int = 100
origin: str = "IGP"
med: int = 0
ebgp: bool = True
igp_cost: int = 0
router_id: str = "0.0.0.0"
def keep_min(key):
return lambda cands: [r for r in cands if key(r) == min(key(c) for c in cands)]
def med_step(cands):
# MED is only compared between routes from the same neighbouring AS.
return [r for r in cands
if not any(s.as_path[:1] == r.as_path[:1] and s.med < r.med for s in cands)]
STEPS = [
("highest LOCAL_PREF", keep_min(lambda r: -r.local_pref)),
("locally originated", keep_min(lambda r: 0 if r.learned_from is None else 1)),
("shortest AS_PATH", keep_min(lambda r: len(r.as_path))),
("lowest ORIGIN", keep_min(lambda r: ORIGIN_RANK[r.origin])),
("lowest MED", med_step),
("eBGP over iBGP", keep_min(lambda r: 0 if r.ebgp else 1)),
("lowest IGP cost", keep_min(lambda r: r.igp_cost)),
("lowest router ID", keep_min(lambda r: tuple(int(x) for x in r.router_id.split(".")))),
]
def best_path(routes):
"""Return (best route, name of the step that decided)."""
cands = list(routes)
if not cands:
return None, "-"
if len(cands) == 1:
return cands[0], "only route"
for name, step in STEPS:
cands = step(cands)
if len(cands) == 1:
return cands[0], name
return cands[0], "tie"
def rov(prefix, origin, roas):
"""RFC 6811 origin validation against (prefix, maxLength, asn) ROAs."""
net = ip_network(prefix)
covering = [r for r in roas if net.subnet_of(ip_network(r[0]))]
if not covering:
return "not-found"
if any(asn == origin and net.prefixlen <= max_len for _, max_len, asn in covering):
return "valid"
return "invalid"
class AS:
def __init__(self, asn, name):
self.asn, self.name = asn, name
self.neighbours = {} # neighbour ASN -> relationship (what they are to me)
self.rib_in = {} # (prefix, neighbour ASN) -> Route
self.originated = {} # prefix -> Route
self.best = {} # prefix -> (Route, reason)
self.sent = {} # (prefix, neighbour ASN) -> AS_PATH we last sent
self.rov = False
def candidates(self, prefix):
own = [self.originated[prefix]] if prefix in self.originated else []
return own + [r for (p, _), r in self.rib_in.items() if p == prefix]
def decide(self, prefix):
old = self.best.get(prefix, (None, ""))[0]
new, why = best_path(self.candidates(prefix))
if new is None:
self.best.pop(prefix, None)
else:
self.best[prefix] = (new, why)
return old != new
def exportable(route, to_rel):
"""Valley-free (Gao-Rexford) export: customer routes go to everyone,
peer and provider routes go only to customers."""
return route.rel in (None, CUSTOMER) or to_rel == CUSTOMER
class Net:
def __init__(self):
self.ases, self.roas = {}, []
def add(self, asn, name):
self.ases[asn] = AS(asn, name)
def link(self, a, b, a_is_to_b):
"""a_is_to_b: 'provider' means a is b's provider, 'peer' means peers."""
back = {PROVIDER: CUSTOMER, CUSTOMER: PROVIDER, PEER: PEER}[a_is_to_b]
self.ases[b].neighbours[a] = a_is_to_b
self.ases[a].neighbours[b] = back
def originate(self, asn, prefix):
self.ases[asn].originated[prefix] = Route(prefix, (), None, None, local_pref=LOCAL_PREF[CUSTOMER])
self.ases[asn].decide(prefix)
return {(asn, prefix)}
def stop_originating(self, asn, prefix):
del self.ases[asn].originated[prefix]
self.ases[asn].decide(prefix)
return {(asn, prefix)}
def cut(self, a, b):
changed = set()
for x, y in ((a, b), (b, a)):
node = self.ases[x]
node.neighbours.pop(y)
for key in [k for k in node.rib_in if k[1] == y]:
del node.rib_in[key]
if node.decide(key[0]):
changed.add((x, key[0]))
for key in [k for k in node.sent if k[1] == y]:
del node.sent[key]
return changed
def run(self, pending, title, quiet=False):
"""Exchange UPDATE/WITHDRAW messages in synchronous rounds until stable."""
out = (lambda *a: None) if quiet else print
out(f"\n## {title}\n")
rnd = 0
while pending:
rnd += 1
msgs = []
for asn, prefix in sorted(pending):
node = self.ases[asn]
best = node.best.get(prefix, (None, ""))[0]
for nb, rel in sorted(node.neighbours.items()):
path = None
if best and nb != best.learned_from and exportable(best, rel):
path = (asn,) + best.as_path
if node.sent.get((prefix, nb)) == path:
continue
if path is None:
node.sent.pop((prefix, nb), None)
else:
node.sent[(prefix, nb)] = path
msgs.append((asn, nb, prefix, path))
pending = set()
if not msgs:
out(f"Round {rnd}: no messages to send. Converged.\n")
break
out(f"### Round {rnd}: messages\n")
out("| From | To | Message | Prefix | AS_PATH | Receiver does |")
out("|---|---|---|---|---|---|")
for src, dst, prefix, path in msgs:
node = self.ases[dst]
key = (prefix, src)
if path is None:
node.rib_in.pop(key, None)
action, kind, shown = "removes route", "WITHDRAW", "-"
elif dst in path:
action, kind, shown = "rejects: own ASN in path (loop)", "UPDATE", " ".join(map(str, path))
elif node.rov and rov(prefix, path[-1], self.roas) == "invalid":
node.rib_in.pop(key, None)
action, kind, shown = "rejects: RPKI invalid", "UPDATE", " ".join(map(str, path))
else:
rel = node.neighbours[src]
node.rib_in[key] = Route(prefix, path, src, rel, LOCAL_PREF[rel])
action, kind, shown = f"stores (from {rel}, LOCAL_PREF {LOCAL_PREF[rel]})", "UPDATE", " ".join(map(str, path))
if node.decide(prefix):
pending.add((dst, prefix))
action += ", best changed"
out(f"| AS{src} | AS{dst} | {kind} | {prefix} | {shown} | {action} |")
if not quiet:
self.table(f"### Round {rnd}: routing tables")
def table(self, title):
print(f"\n{title}\n")
print("| AS | Prefix | Candidate paths | Best path | Decided by |")
print("|---|---|---|---|---|")
for asn, node in sorted(self.ases.items()):
for prefix in sorted({p for p, _ in node.rib_in} | set(node.originated)):
cands = node.candidates(prefix)
if not cands:
continue
best, why = node.best[prefix]
fmt = lambda r: "(origin)" if not r.as_path else " ".join(map(str, r.as_path))
cs = ", ".join(f"{fmt(r)} [LP {r.local_pref}]" for r in cands)
print(f"| AS{asn} {node.name} | {prefix} | {cs} | **{fmt(best)}** | {why} |")
print()
def lookup(self, asn, dst):
"""FIB lookup: longest-prefix match over this AS's best routes."""
node = self.ases[asn]
hits = [p for p in node.best if ip_address(dst) in ip_network(p)]
if not hits:
return None
prefix = max(hits, key=lambda p: ip_network(p).prefixlen)
return prefix, node.best[prefix][0]
def trace(self, src, dst):
hops, asn = [], src
for _ in range(len(self.ases) + 1):
hit = self.lookup(asn, dst)
if hit is None:
hops.append(f"AS{asn} (no route, dropped)")
break
prefix, route = hit
if route.learned_from is None:
hops.append(f"AS{asn} (delivered, owns {prefix})")
break
hops.append(f"AS{asn} (matches {prefix})")
asn = route.learned_from
print(f"Packet AS{src} -> {dst}: " + " -> ".join(hops) + "\n")
def toy_internet(with_misconfig=False):
net = Net()
for asn, name in [(64500, "ContentCo"), (64510, "TransitA"), (64520, "TransitB"),
(64530, "IndiaISP"), (64540, "CampusNet")]:
net.add(asn, name)
net.link(64510, 64500, PROVIDER) # TransitA sells transit to ContentCo
net.link(64510, 64520, PEER) # the two transit networks peer for free
net.link(64520, 64530, PROVIDER) # TransitB sells transit to IndiaISP
net.link(64500, 64530, PEER) # ContentCo and IndiaISP peer at the IXP
net.link(64530, 64540, PROVIDER) # CampusNet buys from IndiaISP ...
net.link(64520, 64540, PROVIDER) # ... and from TransitB (multihomed)
if with_misconfig:
net.add(64666, "Misconfig")
net.link(64520, 64666, PROVIDER) # Misconfig is another TransitB customer
return net
def worked_examples():
print("\n## Best-path worked examples\n")
p = "198.51.100.0/24"
examples = {
"1. LOCAL_PREF beats a shorter path": [
Route(p, (64520, 64510, 64500), 64520, PROVIDER, local_pref=100),
Route(p, (64530, 64999, 64998, 64500), 64530, CUSTOMER, local_pref=200),
],
"2. MED decides between two links to the same neighbour": [
Route(p, (64510, 64500), 64510, PROVIDER, med=50, router_id="10.0.0.1"),
Route(p, (64510, 64500), 64510, PROVIDER, med=10, router_id="10.0.0.2"),
],
"3. Everything ties until IGP cost (hot potato)": [
Route(p, (64510, 64500), 64510, PEER, ebgp=False, igp_cost=30, router_id="10.0.0.1"),
Route(p, (64520, 64500), 64520, PEER, ebgp=False, igp_cost=5, router_id="10.0.0.9"),
],
"4. Last resort: lowest router ID": [
Route(p, (64510, 64500), 64510, PEER, router_id="192.0.2.7"),
Route(p, (64520, 64500), 64520, PEER, router_id="192.0.2.3"),
],
}
print("| Example | Candidates | Winner | Decided by |")
print("|---|---|---|---|")
for title, routes in examples.items():
best, why = best_path(routes)
show = lambda r: (f"{' '.join(map(str, r.as_path))} (LP {r.local_pref}, MED {r.med}, "
f"{'eBGP' if r.ebgp else 'iBGP'}, IGP {r.igp_cost}, RID {r.router_id})")
print(f"| {title} | {'<br>'.join(show(r) for r in routes)} | {' '.join(map(str, best.as_path))} | {why} |")
print()
if __name__ == "__main__":
PREFIX = "203.0.113.0/24"
net = toy_internet()
net.run(net.originate(64500, PREFIX), f"Scenario A: AS64500 announces {PREFIX}")
net.trace(64530, "203.0.113.10")
net.trace(64540, "203.0.113.10")
assert net.ases[64530].best[PREFIX][0].as_path == (64500,) # peer beats provider
assert net.ases[64540].best[PREFIX][0].as_path == (64530, 64500) # shorter path wins
assert (PREFIX, 64520) not in net.ases[64530].sent # no valley: peer route not sent up
net.run(net.cut(64500, 64530), "Scenario B: the IXP peering link AS64500-AS64530 fails")
net.trace(64530, "203.0.113.10")
net.trace(64540, "203.0.113.10")
assert net.ases[64540].best[PREFIX][0].as_path == (64520, 64510, 64500)
net = toy_internet()
net.run(net.originate(64500, PREFIX), "Scenario C: baseline", quiet=True)
net.run(net.stop_originating(64500, PREFIX), "Scenario C: AS64500 withdraws its own prefix")
net.trace(64540, "203.0.113.10")
assert all(PREFIX not in a.best for a in net.ases.values()) # gone everywhere
# Scenario D: AS64666 wrongly announces a more-specific /25 inside ContentCo's /24.
roa = [(PREFIX, 24, 64500)] # ContentCo's ROA: only AS64500, nothing longer than /24
print("\nROV states:", {p: rov(p, o, roa) for p, o in
[(PREFIX, 64500), ("203.0.113.0/25", 64666), ("192.0.2.0/24", 64666)]})
for label, rov_at in [("no ROV anywhere", []), ("ROV at TransitA only", [64510]),
("ROV at TransitB", [64520])]:
net = toy_internet(with_misconfig=True)
net.roas = roa
for asn in rov_at:
net.ases[asn].rov = True
net.run(net.originate(64500, PREFIX), f"Scenario D ({label}): baseline", quiet=True)
net.run(net.originate(64666, "203.0.113.0/25"), f"Scenario D ({label}): AS64666 announces 203.0.113.0/25")
for src in (64510, 64520, 64530, 64540):
net.trace(src, "203.0.113.10")
net.trace(src, "203.0.113.200")
worked_examples()
assert best_path([Route("x", (1, 2, 3), 1, CUSTOMER, 200), Route("x", (4,), 4, PROVIDER, 100)])[1] == "highest LOCAL_PREF"
assert rov("203.0.113.0/25", 64666, [("203.0.113.0/24", 24, 64500)]) == "invalid"
assert rov("203.0.113.0/24", 64500, [("203.0.113.0/24", 24, 64500)]) == "valid"
assert rov("192.0.2.0/24", 64666, [("203.0.113.0/24", 24, 64500)]) == "not-found"
Things to try:
- Make IndiaISP prefer its provider over its peer: set
LOCAL_PREF[PEER] = 90. Rerun Scenario A and watch IndiaISP's best path and CampusNet's choice change. - Break the export rule: change
exportableto always returnTrue. IndiaISP now sends its peer route up to TransitB, and TransitB starts using IndiaISP as transit to ContentCo. You've simulated a route leak. - Multihome ContentCo: add
net.link(64520, 64500, PROVIDER)intoy_internetso it also buys from TransitB. Rerun Scenario A and see which ASes now reach it directly through TransitB, and how thesentbookkeeping changes. - Set the ROA's maxLength to 25 and rerun Scenario D. The
/25from AS64666 is still invalid (wrong origin), but a/25from AS64500 becomes valid. Now announce the/25with a forged path ending in 64500 and see ROV accept it.
Why Application Engineers Should Care
- Latency optimization: CDNs use BGP anycast. Choosing a CDN means choosing whose BGP relationships determine where your users' traffic terminates.
- DDoS resilience: Services with anycast BGP can absorb volumetric DDoS — Cloudflare absorbs 1.5Tbps+ attacks at the routing layer.
- Multi-cloud routing: AWS Direct Connect, Azure ExpressRoute, GCP Interconnect all use BGP. Understanding BGP is essential for hybrid cloud networking.
- Outage postmortems: Major internet outages almost always involve BGP — a leaked route, a misconfigured AS, or a fiber cut that triggers route withdrawals. Reading BGP-related postmortems is much easier when you understand the protocol.
Correction (2026-11-13)
"1.5Tbps+" is badly out of date as a sense of scale. Cloudflare has publicly reported mitigating attacks above 20 Tbps in 2025. Absorbing them takes anycast (to spread the attack) plus large edge capacity and packet filtering at every site; anycast alone just divides the problem. And "almost always involve BGP" overstates it: many major outages are caused by configuration pushes, DNS, certificates, or software bugs. BGP is involved in many of the network-reachability outages, such as the Facebook and Cloudflare BYOIP incidents, not in most outages overall.
A few more reasons that matter day to day:
- Your DNS provider's routing is your routing. The MyEtherWallet attack hijacked a DNS provider's addresses, not the victim's. Pick providers that sign ROAs and use DNSSEC where you can.
- Sign ROAs for your own address space, if your company holds any. It takes minutes in your RIR's portal and protects against most accidental hijacks of your prefixes.
- Monitor your prefixes from outside. Free alerting from services like bgp.tools or Cloudflare Radar tells you when your prefix appears with a new origin or disappears.
- Cloud networking problems are often BGP problems. "The VPN is up but traffic doesn't flow" is frequently a missing or filtered BGP route, a prefix-count limit, or an asymmetric path.
Quick Reference
| Thing | Value or rule |
|---|---|
| Spec | BGP-4, RFC 4271 |
| Transport | TCP port 179 |
| Messages | OPEN, UPDATE, NOTIFICATION, KEEPALIVE, ROUTE-REFRESH |
| FSM states | Idle, Connect, Active, OpenSent, OpenConfirm, Established |
| Timers | Hold 90 s, keepalive 30 s suggested by RFC 4271 (Cisco: 180/60). MRAI 30 s eBGP / 5 s iBGP suggested |
| Loop prevention | eBGP: reject paths containing your ASN. iBGP: don't re-advertise iBGP routes (full mesh or route reflectors) |
| Best-path order | LOCAL_PREF, locally originated, AS path length, origin, MED, eBGP over iBGP, IGP cost, router ID, neighbour address |
| Forwarding | Longest-prefix match in the FIB, across all prefixes |
| Common max lengths accepted | IPv4 /24, IPv6 /48 |
| Private ASNs | 64512–65534, 4200000000–4294967294 |
| Documentation ASNs | 64496–64511, 65536–65551 |
| Relationships | Customer routes to everyone, peer and provider routes to customers only |
| ROV states | Valid, Invalid, NotFound (RFC 6811) |
| Data-centre design | eBGP everywhere, private ASNs, ECMP (RFC 7938) |
Common Misconceptions
| Misconception | Reality |
|---|---|
| "BGP finds the shortest or fastest path." | BGP finds the path that best fits policy. LOCAL_PREF (money) is checked before AS path length, and BGP never measures latency or bandwidth. |
| "A shorter AS path means lower latency." | One AS hop can span an ocean. Five hops can stay inside one city. AS path length is a weak proxy for distance. |
| "The Active state means the session is working." | Active means the router is trying to connect. Only Established means up. |
| "BGP routes are symmetric." | The forward path is chosen by the sender's networks, the return path by the receiver's. They're independent and often differ. |
| "A higher LOCAL_PREF beats a more-specific prefix." | BGP compares routes per prefix. The FIB picks the longest matching prefix, across prefixes. A /25 always beats a /24 for addresses inside it. |
| "RPKI stops BGP hijacks." | ROV stops origin hijacks for prefixes with ROAs, on networks that enforce it. It doesn't stop forged-origin paths or leaks of valid routes. |
| "The DNS was down" (Facebook 2021). | The DNS servers were running. The routes to them had been withdrawn. |
| "The IXP is an AS in the path." | IXPs and their route servers are normally transparent and don't appear in AS paths. |
| "BGP is only for ISPs." | Data centres, Kubernetes clusters, cloud interconnects and CDNs all run BGP. |
| "Withdrawals clear out instantly." | Path hunting means routers try stale backup paths first. Global convergence can take minutes. |
Interview Questions
1. What problem does BGP solve that OSPF can't?
OSPF assumes one organisation that can see and trust the whole topology, and it optimises one metric. The internet is ~75,000 independently run networks that won't share their internal topology, don't fully trust each other, and choose routes by business policy. BGP exchanges only reachability ("this prefix, via this path"), lets each AS apply its own import and export policy, and scales to about a million prefixes. Inside an AS you still use OSPF or IS-IS. Between ASes you use BGP.
2. How does BGP prevent routing loops?
On eBGP, each AS prepends its ASN to the AS path when it passes a route on. A router that receives a route whose AS path already contains its own ASN rejects it, so a route can never circle back. On iBGP, the AS path doesn't change, so instead iBGP routes aren't re-advertised to other iBGP peers. That requires a full mesh, or route reflectors that use ORIGINATOR_ID and CLUSTER_LIST to prevent loops.
3. Walk through the BGP best-path algorithm.
Discard unusable routes (unreachable next hop, filtered, looped, RPKI-invalid if enforced). Then, in order: highest weight (Cisco, local), highest LOCAL_PREF, locally originated, shortest AS path, lowest origin (IGP before EGP before incomplete), lowest MED (only among routes from the same neighbour AS by default), eBGP over iBGP, lowest IGP cost to the next hop, oldest eBGP route, lowest router ID, shortest cluster list, lowest neighbour address. Stop at the first step that leaves one route.
4. Your company has two ISPs. How do you make one primary for outbound traffic, and for inbound traffic?
Outbound is your choice: set a higher LOCAL_PREF on routes from the primary ISP. Inbound is other networks' choice, so you can only influence it: AS-path prepend your announcements to the backup ISP so they look longer, use the backup ISP's communities to lower its LOCAL_PREF for your routes, or announce more-specific prefixes via the primary (the covering prefix via both). Prepending loses to anyone's LOCAL_PREF, so it's a hint, not a guarantee.
5. Explain customer, peer, and provider relationships and the valley-free rule.
A customer pays a provider for transit. Peers exchange their own and their customers' traffic for free. An AS prefers customer routes (they earn money), then peer routes (free), then provider routes (cost money), usually via LOCAL_PREF. It exports customer routes to everyone, but peer and provider routes only to customers, because carrying traffic between two non-paying neighbours costs money for nothing. The result is that valid paths go up through providers, across at most one peering link, then down through customers. A path that goes down and back up is a valley, and indicates a route leak.
6. How does a BGP hijack work, and why does a more-specific prefix make it worse?
An AS announces a prefix it doesn't hold, and neighbours believe it because BGP has no built-in origin verification. With an exact-prefix hijack, only networks that prefer the hijacker's path by policy and path length are affected. With a more-specific prefix (a /24 inside the victim's /22), longest-prefix match sends traffic for that range to the hijacker everywhere the route is accepted, regardless of BGP attributes. Pakistan/YouTube 2008 and MyEtherWallet 2018 were both more-specific hijacks.
7. What is RPKI ROV, what are its three states, and what doesn't it protect against?
Prefix holders sign ROAs (prefix, maxLength, origin ASN) with certificates chained to the RIRs. Validators verify them and pass the results to routers over RTR. Routers mark each route Valid (a ROA matches origin and length), Invalid (a ROA covers the prefix but nothing matches), or NotFound (no covering ROA), and usually drop Invalid. It doesn't protect against forged paths that end in the correct origin, against leaks of legitimately originated routes, or for prefixes without ROAs. BGPsec and ASPA target path validation.
8. Why do large iBGP networks use route reflectors, and what do they cost?
iBGP routes aren't re-advertised to other iBGP peers, so every router needs a session to every other: n(n−1)/2 sessions, which doesn't scale and makes every new router a change everywhere. A route reflector may re-advertise between its clients, so each client needs only a session or two to the reflectors. The cost is that the reflector advertises only its own best path, chosen from its position, so clients can lose optimal hot-potato exits and path diversity. ADD-PATH and optimal route reflection address that.
9. Why can BGP take minutes to converge after a prefix is withdrawn?
Silent failures are detected only by the hold timer (90 to 180 s) unless BFD is used. MRAI timers batch updates (30 s suggested for eBGP). Worst of all, path hunting: when a router loses its best path, it falls back to the next stored path, which usually also leads to the dead origin but hasn't been withdrawn yet, announces it, and triggers further fallbacks downstream. Each round may cost an MRAI interval, so a router can explore many stale paths before the final withdrawal arrives.
10. How does anycast work, and what are its limits?
Many sites announce the same prefix with the same origin. Every network picks one announcement by normal best-path selection, so users reach the BGP-nearest site, which is usually but not always the geographically nearest. Failover is a withdrawal. Limits: "nearest" is decided by other networks' policies, so users can land far away. Route changes mid-connection can break TCP sessions. A withdrawal at one site shifts all its traffic to neighbours, which must have spare capacity. And a bug that withdraws everywhere at once is a total outage, as Facebook found.
11. Why do some data centres run BGP instead of OSPF between switches?
RFC 7938 describes eBGP on every link of a Clos fabric: one protocol from server to internet, private ASNs per tier, ECMP across the spines, loops prevented by AS paths, simpler failure behaviour (withdrawals travel only along affected paths instead of flooding the whole fabric), easy per-session policy, and mature, scalable implementations. BGP unnumbered (IPv6 link-local sessions, RFC 8950 next hops) removes the need to address every link.
12. What happened to Facebook on October 4, 2021, in BGP terms?
A backbone maintenance command disconnected Facebook's data centres. Its DNS servers, designed to withdraw their BGP announcements when they lose contact with the data centres, all withdrew at once. The prefixes holding Facebook's authoritative DNS vanished from global routing tables, resolvers couldn't reach them, and every Facebook service was unreachable for about six hours. Lessons: withdrawals are as powerful as announcements, health-based withdrawal needs a floor, and out-of-band access must not depend on the production network.
Further Reading
Specifications
- RFC 4271: A Border Gateway Protocol 4 (BGP-4). The protocol itself, including the FSM and decision process.
- RFC 4456: BGP Route Reflection.
- RFC 7938: Use of BGP for Routing in Large-Scale Data Centers.
- RFC 6480 (RPKI architecture) and RFC 6811 (BGP prefix origin validation).
- RFC 7908: Problem Definition and Classification of BGP Route Leaks.
- RFC 8205: BGPsec Protocol Specification.
- RFC 8212: Default External BGP Route Propagation Behavior without Policies.
Incident write-ups and explainers
- Cloudflare, Understanding How Facebook Disappeared from the Internet.
- Facebook Engineering, More details about the October 4 outage.
- Cloudflare, How Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline Today.
- Cloudflare, BGP leaks and cryptocurrencies (the 2018 Route 53 / MyEtherWallet hijack).
- RIPE NCC, YouTube Hijacking: A RIPE NCC RIS case study.
- Cloudflare, Is BGP Safe Yet?: which ISPs enforce RPKI ROV.
- APNIC Blog, Geoff Huston's annual "BGP in 20XX" reports on routing table growth and churn (search the APNIC blog), and his BGP table statistics at bgp.potaroo.net.
Papers
- Gao and Rexford, "Stable Internet Routing Without Global Coordination", IEEE/ACM Transactions on Networking, 2001.
- Labovitz, Ahuja, Bose and Jahanian, "Delayed Internet Routing Convergence", ACM SIGCOMM 2000.
Related Archon pages
- The Network Layer: forwarding vs routing, longest-prefix match, OSPF, IS-IS.
- IP Addressing & Subnetting and NAT.
- DNS Deep Dive: why a routing outage looks like a DNS outage.
- Network Debugging: traceroute, mtr and friends.
- DDoS Response Runbook.
- War Room: Facebook October 2021 and Cloudflare BYOIP 2026.