VLANs vs VXLAN Overlays

How 802.1Q VLANs work

An 802.1Q tag is four bytes inserted into the Ethernet header after the source MAC address. It holds a 12-bit VLAN ID, a 3-bit priority field used for quality of service, and a one-bit drop eligibility flag. Of the 4,096 possible IDs, 0 and 4095 are reserved, which leaves 4,094 usable VLANs.

Switch ports come in two kinds. An access port belongs to one VLAN and sends untagged frames to the device plugged into it, so the server never sees a tag. A trunk port carries many VLANs between switches, with each frame tagged so the next switch knows which segment it belongs to. Most trunks also have a native VLAN whose frames travel untagged. Each VLAN is its own broadcast domain: an ARP request on VLAN 20 never reaches hosts on VLAN 30, and traffic between them must go through a router.

VLANs are still the right tool for many jobs, such as separating a guest Wi-Fi network from staff devices, putting management interfaces on their own segment, or splitting a small server room into a few zones.

Where VLANs run out

VLANs struggle in large or shared data centers for three reasons. The first is scale: 4,094 segments sounds like a lot until a cloud provider wants a separate network for every customer. The second is reach: a VLAN is a Layer 2 construct, so stretching one across many racks means extending Layer 2 between switches and relying on Spanning Tree Protocol to block loops. That wastes links and creates large failure domains, where one broadcast storm or loop can affect every rack in the segment. The third is that modern fabrics are routed. In the design described in Leaf-and-Spine Data Center Networks, each leaf is a router and Layer 2 stops at the rack, so there is no shared Layer 2 path for a VLAN to follow.

Yet many workloads still need Layer 2 adjacency across racks: virtual machines that keep their address when they move between hosts, clusters that rely on a shared subnet, or tenants who expect their own private network. Overlays solve that without giving up the routed fabric.

How VXLAN works

VXLAN is MAC-in-UDP encapsulation. A VXLAN Tunnel Endpoint (VTEP), usually the leaf switch or the hypervisor, takes the original Ethernet frame, adds an 8-byte VXLAN header holding a 24-bit VXLAN Network Identifier (VNI), and wraps the result in UDP, IP, and an outer Ethernet header addressed to the remote VTEP. The IANA-assigned UDP destination port is 4789.

The underlay, meaning the physical routed fabric, only sees ordinary UDP packets between VTEP addresses, so it can route them with ECMP like any other traffic. VTEPs set the outer UDP source port from a hash of the inner frame's headers, which gives different inner flows different outer ports and spreads them across all available paths.

On Linux, a VTEP is a network interface. Creating one and inspecting it looks like this:

$ sudo ip link add vxlan100 type vxlan id 100 dstport 4789 local 192.0.2.11 nolearning
$ ip -d link show vxlan100
12: vxlan100: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 3a:5c:1e:8b:20:7f brd ff:ff:ff:ff:ff:ff promiscuity 0
    vxlan id 100 local 192.0.2.11 srcport 0 0 dstport 4789 nolearning ttl auto ageing 300

Note the MTU of 1450. The encapsulation adds about 50 bytes with an IPv4 underlay, so if the physical network keeps a standard 1,500-byte MTU, the inner network must shrink to 1,450. Most fabrics instead raise the underlay MTU, often to around 9,000 bytes, so the overlay can keep a full 1,500.

EVPN: a control plane for VXLAN

VXLAN by itself only defines the packet format. The original RFC 7348 approach learns MAC addresses the way a switch does, by flooding unknown traffic and watching where replies come from, using IP multicast or a list of peer VTEPs to deliver broadcasts. That works, but it floods a lot and scales poorly.

Ethernet VPN (EVPN) replaces flooding with BGP. Defined first in RFC 7432 and adapted to VXLAN in RFC 8365, it has each VTEP advertise the MAC and IP addresses it has learned locally as BGP routes. Every other VTEP then usually knows where each host lives before traffic is sent. EVPN greatly reduces flooding, but broadcast, unknown-unicast and multicast traffic still has to be delivered for hosts that have not been learned yet, and for protocols that need it. That enables several useful features:

• ARP suppression: the local leaf answers ARP requests from its EVPN table instead of flooding them across the fabric.
• Distributed anycast gateway: every leaf can act as the default gateway for a subnet using the same gateway address, so routed traffic leaves the rack at the first hop.
• Multi-homing: a server can connect to two leaves with active links to both.
• Layer 3 routes: IP prefixes can be advertised too, so tenants get routed networks inside the overlay, each in its own VRF.

Because the data center underlay often already runs BGP, EVPN usually rides on the same sessions or on a parallel set. BGP Basics: How Internet Routing Works introduces the protocol.

VLAN vs VXLAN at a glance

What overlays mean for DNS

DNS servers inside an overlay are ordinary hosts, but a few overlay details can cause confusing DNS problems.

MTU mismatches. If the underlay MTU was not raised and hosts still believe they have 1,500 bytes, full-size packets get dropped or fragmented inside the fabric. Small DNS queries keep working, while large UDP answers, such as DNSSEC-signed responses or long TXT records, time out. That pattern is a strong hint to check the path MTU. MTU, Fragmentation and Big DNS Answers and EDNS, Large Responses and TCP Fallback explain the mechanics.

Overlapping tenant address space. In a multi-tenant overlay, two tenants can both use the same private ranges in separate VRFs. Each needs its own resolver path and often its own internal zones, which is a form of the split views described in Split-Brain DNS / Split Horizon.

Reverse DNS and inventory. Overlay addresses move with workloads, so PTR records and internal zones need to follow them, usually through automation rather than hand edits.

How DNS Panopticon helps

Overlays are internal, so DNS Panopticon's public lookups do not see them. Where the tools help is with the zones that describe them. Zone Analyzer parses an internal zone export locally in your browser, and its quick filters can isolate PTR records or narrow by hostname, which makes it easier to spot stale entries for workloads that have moved. For the public side of the same organization, Domain Visualizer checks the externally visible records.