Part 6: VCF Networking with NSX — Deep Dive

NSX in VCF — Why It’s Mandatory

Unlike standalone vSphere deployments where NSX is optional, VMware Cloud Foundation requires NSX as a core component. Every workload domain gets its own NSX deployment, and SDDC Manager handles the full NSX lifecycle — deployment, configuration, updates, and certificate management. This article covers NSX architecture within VCF at a depth suitable for infrastructure architects and senior admins.

NSX Transport Architecture

NSX creates an overlay network on top of your physical underlay. The physical network only needs to provide L3 connectivity between hosts — NSX handles everything above that in software. The key concept is the Transport Node.

Host Transport Nodes

Every ESXi host in a VCF domain is configured as an NSX Host Transport Node. This means:

• The NSX kernel module (nsx-vswitch) is installed on the host as a VIB
• A Tunnel Endpoint (TEP) VMkernel adapter is created on each host with an IP from the TEP IP pool configured in SDDC Manager
• The TEP is used to encapsulate VM traffic in Geneve tunnels (UDP port 6081)
• VM traffic never touches the physical network as raw L2 frames — it’s always encapsulated in Geneve when leaving the host

NSX Edge Transport Nodes

Edge Transport Nodes are dedicated NSX appliances (VM or bare-metal) that handle North-South traffic — traffic between VMs in the overlay and the external physical network. In VCF, Edge nodes are deployed as VMs on the ESXi hosts.

• Edge VM sizes: Small (2 vCPU, 4 GB), Medium (4 vCPU, 8 GB), Large (8 vCPU, 32 GB), X-Large (16 vCPU, 64 GB)
• Production recommendation: Large or X-Large for high-throughput environments
• Edge Cluster: Minimum 2 Edge nodes per cluster for HA (active/standby T0 gateway failover)
• Edge TEP: Edge nodes have their own TEP VMkernel adapters on a dedicated Edge TEP VLAN

NSX Logical Architecture: T0 and T1 Gateways

NSX uses a two-tier routing architecture for overlay traffic:

Tier-0 (T0) Gateway

The T0 gateway is the North-South boundary router. It peers with your physical network (ToR switches or core routers) via BGP or OSPF and redistributes routes between the overlay and physical underlay.

• Active/Standby HA: Two T0 gateways (one active, one standby) running on different Edge nodes. Failover is automatic in ~1 second on link failure detection.
• Active/Active HA: Supported for specific use cases — requires ECMP on the physical routers and careful route advertisement design.
• Uplinks: T0 uplinks connect to the physical router via VLAN-tagged interfaces on the Edge nodes. Each uplink gets a routable IP from your physical subnet.
• BGP peering: T0 advertises overlay segment subnets to physical routers via BGP. Physical routers advertise default routes or specific prefixes back to T0.

Tier-1 (T1) Gateway

T1 gateways sit between overlay segments and the T0 gateway. They provide routing between overlay segments and route advertisements up to the connected T0.

• T1 gateways are purely software-based — they run as distributed router instances on every host transport node, so East-West routing between segments connected to the same T1 is local to each host (no hairpin through Edge nodes)
• T1 gateways are typically created per-application or per-tenant in multi-tenant deployments
• T1 can perform NAT, Load Balancing (L4), and VPN termination

NSX Overlay Segments

An NSX Overlay Segment is a logical L2 network backed by Geneve encapsulation. From a VM’s perspective, it looks like a standard vSphere port group. From the network’s perspective, it’s a Geneve tunnel between hosts.

• Segment creation: Can be done in NSX Manager UI, API, or via Aria Automation. No physical switch configuration needed.
• VXLAN vs Geneve: VCF 9.x uses Geneve exclusively (VXLAN is legacy NSX-V). Geneve supports variable header length, allowing metadata extensions.
• Segment VNI: Each segment has a Virtual Network Identifier (VNI), a 24-bit integer similar to a VLAN ID but with 16 million possible values vs 4096 for VLANs.
• Gateway attachment: Segments are connected to a T1 gateway for routing. A segment without a gateway is L2-only (isolated).

Distributed Firewall (DFW)

The NSX Distributed Firewall is one of VCF’s most powerful security features. Unlike perimeter firewalls, DFW runs in the kernel of every ESXi host — traffic is firewalled at the vNIC of every VM, before it ever leaves the host.

• Zero-Trust enforcement: East-West traffic (VM to VM) is firewalled even on the same host, preventing lateral movement without explicit policy
• Groups (formerly Security Groups): DFW rules are applied to logical groups, not IP addresses. Groups can include VMs, segment ports, tags, or AD groups (if NSX Identity Firewall is configured)
• Stateful inspection: DFW tracks TCP connections and UDP flows for stateful enforcement
• Performance: DFW runs in the kernel at line rate — no performance penalty vs no firewall at hardware speeds above 10 GbE

NSX Manager Cluster

NSX is managed via the NSX Manager — a cluster of 3 VMs providing HA for the management plane. In VCF:

• Each workload domain gets its own NSX Manager cluster (3 VMs, each on a different ESXi host)
• NSX Manager VMs are deployed automatically by SDDC Manager during workload domain creation
• NSX Manager cluster provides a virtual IP (VIP) for single-endpoint access — all API calls and UI access go to the VIP, which floats between the active managers
• NSX Manager data plane (transport nodes, gateways) continues operating if NSX Manager goes down — management plane outage does not affect running workloads

VLAN Segments vs Overlay Segments

VCF’s NSX supports two segment types:

• Overlay Segments: Geneve-encapsulated, no physical VLAN required, can span any host reachable via the TEP network. Used for all workload networks. (Recommended for all new workloads)
• VLAN Segments: Backed by a physical VLAN. Used for management network connectivity, ToR uplinks, and Edge node uplinks. Not routable within NSX overlay — physical router required.

NSX VCF Reference Topology

Broadcom Documentation

• VCF Networking Architecture — TechDocs
• VMware NSX 4.2 Documentation
• NSX Tier-0 Gateway Configuration
• NSX Distributed Firewall Administration