Support Library
Leaf-and-Spine Data Center Networks
A leaf-and-spine network connects every access switch (leaf) to every core switch (spine), so any two servers are the same small number of hops apart and traffic spreads across many equal paths.
The short answer
Leaf-and-spine is the standard physical design for modern data center networks. Servers plug into leaf switches, often called top-of-rack switches. Every leaf has an uplink to every spine switch, and nothing else connects to the spines except leaves. Because the topology is uniform, traffic between any two servers on different leaves crosses exactly one leaf, one spine, and one more leaf, and the network can use all spines at once with equal-cost multipath routing (ECMP).
Need more server ports? Add a leaf. Need more bandwidth between leaves? Add a spine. Neither change requires replacing a bigger box in the middle of the network.
Why the three-tier design ran out of road
Traditional enterprise data centers used three tiers: access switches for servers, aggregation (or distribution) switches above them, and a pair of large core switches on top. The design was built for north-south traffic, meaning requests that come in from users, hit a server, and go back out. Layer 2 domains stretched across access and aggregation, and Spanning Tree Protocol blocked redundant links to prevent loops, so half of the expensive uplinks often sat idle until something failed.
Modern applications changed the traffic pattern. A single web request can fan out to dozens of microservices, caches, databases, and storage nodes. That east-west traffic between servers now dominates. In a three-tier network, two servers in different pods may cross access, aggregation, core, aggregation, and access again, and the core becomes a bottleneck that can only be fixed by buying a bigger chassis.
| Property | Three-tier | Leaf-and-spine |
|---|---|---|
| Optimized for | North-south traffic | East-west traffic |
| Redundant links | Often blocked by Spanning Tree | All active through ECMP |
| Hops between servers | Varies with location | Same for any two leaves |
| Growth | Upgrade core and aggregation | Add leaves or spines |
| Typical control plane | Layer 2 with Spanning Tree, routing at aggregation or core | Layer 3 routing to the leaf, overlays for Layer 2 needs |
The Clos topology
Leaf-and-spine is a folded Clos network, named after Charles Clos, who described multistage switching networks for telephone exchanges in the 1950s. The idea is to build a large, non-blocking fabric from many small identical switches instead of one huge one. The rules are simple:
Leaves never connect to other leaves for forwarding, and spines never connect to other spines. A packet from a server on Leaf 1 to a server on Leaf 7 takes this path, with the spine chosen per flow:
sends packet
hashes flow
one of N paths
delivers to Server B
The size of the fabric is set by port counts. The number of spines is limited by how many uplinks each leaf has, and the number of leaves is limited by how many ports each spine has. When a fabric outgrows its spines, operators add a third tier, often called super-spines, and connect several leaf-and-spine pods together in the same Clos pattern.
ECMP: using every path at once
Because a leaf learns the same destination through every spine at the same cost, its routing table holds several next hops for each remote prefix. Equal-cost multipath forwarding picks one of them for each packet by hashing header fields, usually the source and destination addresses, protocol, and source and destination ports. All packets of one flow produce the same hash and follow the same spine, so they normally arrive in order, while different flows spread across all spines. When a link or spine fails or returns, the hash is recomputed and some flows move to a new path, which can briefly reorder or drop their packets.
On a leaf running a Linux-based network operating system, a route learned through four spines looks like this:
$ ip route show 203.0.113.0/24
203.0.113.0/24 nhid 52 proto bgp metric 20
nexthop via 192.0.2.0 dev swp51 weight 1
nexthop via 192.0.2.2 dev swp52 weight 1
nexthop via 192.0.2.4 dev swp53 weight 1
nexthop via 192.0.2.6 dev swp54 weight 1ECMP has two well-known limits. Hashing balances flows, not bytes, so a few very large flows can land on the same spine while others sit lightly used. And when a next hop is added or removed, many flows may rehash to a different path. That is harmless for routed traffic between ordinary hosts, but it matters when the next hops are stateful services, as the section on anycast below explains.
Oversubscription
Oversubscription is the ratio between the bandwidth a leaf offers to servers and the bandwidth it has toward the spines. Consider a leaf with 48 server ports at 25 Gbit/s and 6 uplinks at 100 Gbit/s. Downlink capacity is 48 × 25 = 1,200 Gbit/s; uplink capacity is 6 × 100 = 600 Gbit/s. The leaf is 2:1 oversubscribed: if every server sent to another rack at full speed at the same moment, only half of that traffic would fit.
| Ratio | Meaning | Typical use |
|---|---|---|
| 1:1 | Uplinks match server capacity; non-blocking | Storage, AI training, or other heavy east-west workloads |
| 2:1 to 3:1 | Common compromise between cost and headroom | General-purpose compute |
| Higher | Servers rarely all transmit at once | Light or bursty workloads |
Oversubscription is a design choice, not a defect. Servers rarely all transmit at line rate together. The point is to choose the ratio deliberately based on workload, and to remember that losing a spine raises the effective ratio until it returns.
BGP inside the data center
Many large fabrics route at Layer 3 all the way to the leaf and use BGP as their only routing protocol. RFC 7938, Use of BGP for Routing in Large-Scale Data Centers, describes the approach. BGP was chosen because it is simple to run with a single protocol, it scales to very large route counts, and its policy controls make it easy to drain traffic from a switch before maintenance.
In the RFC 7938 design, sessions are external BGP (eBGP) over each point-to-point link. A common numbering scheme gives all spines in a pod the same ASN and each leaf its own, for example spines in AS65536 and leaves in AS65537, AS65538, and so on. Because a spine rejects any path that already contains the shared spine ASN, BGP's normal loop prevention stops routes from bouncing leaf-spine-leaf-spine without extra configuration. Many deployments add Bidirectional Forwarding Detection (BFD) so a failed link is detected in well under a second rather than waiting for the BGP hold timer. For the protocol itself, see BGP Basics: How Internet Routing Works.
Routing to the leaf means each rack is its own subnet, and Layer 2 no longer stretches across the building. Workloads that still need a shared Layer 2 segment across racks, such as virtual machines that migrate between hosts, get it from an overlay instead. VLANs vs VXLAN Overlays covers how VXLAN and EVPN provide that on top of a routed fabric.
Where DNS resolvers and anycast nodes sit
Recursive resolvers inside a data center are ordinary hosts or virtual machines attached to leaves. Because every leaf is equally close to every other, it does not much matter which rack they live in, though operators usually spread them across several leaves so one rack failure does not take them all out.
A common pattern is anycast inside the fabric. Each resolver host runs a small BGP daemon and announces the same service address, such as 192.0.2.53/32, to its leaf. Clients send queries to that one address, and ECMP spreads them across every healthy resolver. A health check on each host withdraws the announcement if the resolver stops answering, so traffic shifts away within seconds. This is the same idea as Internet-scale anycast, described in DNS Anycast Explained, applied to one building.
The rehashing caveat matters here. Plain UDP queries are unaffected when a resolver joins or leaves, but DNS over TCP and encrypted DNS sessions may be sent to a different host mid-connection and reset. Switches that support resilient hashing keep existing flows in place when the next-hop set changes. Where full connection tracking is needed, a load balancer in front of the resolvers is the alternative; Load Balancers: Layer 4 vs Layer 7 compares the options.
Connections to the outside world usually terminate on border leaves, which connect to the WAN or Internet routers. Authoritative DNS servers that face the public Internet, and anycast DNS nodes at a provider's points of presence, often sit on or near these border leaves so their announcements reach upstream networks with as few internal hops as possible.
How DNS Panopticon helps
DNS Panopticon's tools see the public side of a network, not the inside of a data center. Website Recon shows the network and AS number that hold each address a hostname resolves to, which tells you which provider's edge your traffic enters. If you manage internal zones for a fabric, Zone Analyzer parses a zone export locally in the browser so you can filter and review records such as resolver and service addresses without uploading the file elsewhere.
FAQ
Why don't leaf switches connect to each other?
Because every leaf already reaches every other leaf through each spine in exactly two hops. Adding leaf-to-leaf links would create unequal paths and complicate ECMP. The usual exception is a pair of leaves linked for multi-chassis link aggregation so a server can dual-home to both.
What happens when a spine switch fails?
The leaves lose one of their equal-cost paths and rehash flows across the remaining spines. Nothing becomes unreachable, but total capacity between leaves drops by roughly one divided by the number of spines.
Is leaf-and-spine only for large data centers?
No. A small fabric of two spines and a handful of leaves uses the same design and grows by adding leaves. The pattern pays off whenever most traffic flows between servers rather than in and out of the building.