Welcome to part 6 of the VCF-9.1 home lab series. The previous post in this series discussed the bringup process of the management domain. In this post, I will demonstrate setting up network connectivity in the management domain.
If you are not following along, I encourage you to read the earlier parts of this series from the links below:
3: VCF 9.1 Pre-Deployment Planning
4: Setting up VCF 9.1 Offline Depot
5: VCF 9.1 Deploy Management Domain
Why Deploy an Edge Cluster in the Management Domain Even If You Run Nothing There?
The Management Domain Edge Cluster in VCF 9.1 is the network spine of the entire software-defined fabric — it carries transit, external, and inter-VPC traffic even though no workload VMs run on it. Think of it less as a compute resource and more as a dedicated routing/NAT appliance for the whole VCF environment.
The two main reasons why you should deploy an edge cluster in the management domain are:
1. Transit Gateway (TGW) Anchoring
The Transit Gateway in VCF 9.1 is a new construct that interconnects VPCs across workload domains. The TGW and Tier-0 service routers run on the Management Domain Edge Cluster. Without it, cross-VPC and cross-domain routing don’t work.
2. External Connectivity for the Entire VCF Fabric
The External Connection is anchored to the Management Domain. All North-South traffic from workload domains traverses the management domain edges before hitting the physical network. Without an Edge Cluster here, no workload domain gets external connectivity.
With VCF 9.0, the workflow for deployment of the edge cluster has been moved to the vCenter UI. In VCF 9.1, some more capabilities around networking have been added, and I will demonstrate them shortly. Before we start the deployment, let’s collect the networking information for the deployment.
TEP VLAN
| Network Type | VLAN | CIDR | Gateway |
| Edge TEP | 92 | 10.252.92.0/24 | 10.252.92.254 |
TEP Details
| Transport Node | CIDR | IP Pool | Description |
| Mgmt Dom Edge | 10.252.92.0/24 | 10.252.92.209 – 10.252.92.224 | Mgmt Domain Edge TEP |
Mgmt Domain Edge Uplinks
| Node | IP 1 (VL 407) | IP 2 (VL 408) |
| Edge 01 | 10.252.245.52/29 | 10.252.245.60/29 |
| Edge 02 | 10.252.245.53/29 | 10.252.245.61/29 |
Mgmt Domain BGP Details (ToR)
| Device | BGP ASN | BGP VLAN | BGP Address |
| ToR 01 | 65000 | 407 | 10.252.245.49/29 |
| ToR 02 | 65000 | 408 | 10.252.245.57/29 |
| NSX Tier-0 Router | 65006 |
VPC Networking
| Network Type | CIDR |
| External IP Block | 10.252.215.0/24 |
| Private Transit Gateway Subnet | 10.250.0.0/16 |
Now that we have built the networking information, let’s deploy the edge cluster.
To deploy a new edge cluster, select the vCenter server and navigate to the Networks > Transit gateways tab and click “Setup Network Connectivity.”
Select the transit gateway connectivity type. The available modes for gateway connectivity are:
- Centralized Connectivity: In this mode, the Transit Gateway forwards traffic to a Tier-0 gateway, and a virtual router is instantiated on the NSX Edge VMs.
- Distributed Connectivity: In this mode, the Transit Gateway can connect to physical infrastructure via an external VLAN network and doesn’t require any Edge VMs for forwarding traffic.
The table below lists the major differences between the 2 gateway connectivity types:
| Feature | Centralized Connectivity | Distributed Connectivity |
| External IP (1:1 NAT) | No | Yes |
| NAT (SNAT/DNAT) | Yes | No |
| VPC Default Outbound NAT | Yes | No |
| DHCP (distributed) | Yes | Yes |
| E/W Firewall | Yes | Yes |
| N/S Firewall | Yes | No |
| AVI Load Balancer | Yes | Yes |
| VCF Automation | Yes | No |
| Supervisor Integration | Yes | No |
When setting up a centralized external connection for VPCs, the Transit Gateway SR component has the same span as the connected Tier-0 gateway SR. In other words, Transit Gateway SR is distributed across the same NSX Edge VMs that host the Tier-0 Gateway. Each TGW Service Router (SR) is tightly coupled with its corresponding Tier-0 SR.
This deployment can operate in either active/active or active/standby mode. The differences in the service supportability are outlined in the table below.
| Feature | Centralized Active/Standby | Centralized Active/Active |
| External IP (1:1 NAT) | Yes | Yes |
| NAT (SNAT/DNAT) | Yes | Yes |
| VPC Default Outbound NAT | Yes | No |
| DHCP | Yes | Yes |
| E/W Firewall | Yes | Yes |
| N/S Firewall | Yes | No |
| AVI Load Balancer | Yes | Yes |
An important note before we proceed to deploy NSX Edges:
If your ESXi host has only 2 pNICs, then for this configuration to work, the Edge TEP and the ESXi host TEP should be in a different VLAN and should be routable to each other via an external router.
In my lab, I am deploying the centralized connection as I plan to deploy VKS and VCF Automation later.
The wizard provides a checklist for the networking prerequisites. Ensure that you have met them before continuing.
Specify the following configuration values for the edge cluster:
- Edge Cluster name
- Edge VM size
- MTU for overlay traffic (this is auto-selected and is fetched from NSX Manager).
Click the Add button to add the edge VMs to the edge cluster.
Configure the following values for the edge VM:
- Edge Node Name: This must match the edge node’s FQDN configured in the DNS server.
- vSphere Cluster
- Data Store
- IP Assignment: Both DHCP and static are supported.
- Edge VM Management IP
- Default Gateway
- Port Group: Select the infrastructure management port group where you have deployed vCenter, NSX Manager, etc.
- Edge uplink teaming policy.
Provide the Edge TEP VLAN and choose the TEP IP assignment type. If you select IP Pool, you have to create a new one before you can use it.
Specify the IP pool name and click on Set to define the TEP subnet.
Select IP Block or IP Ranges. I am using an IP range in my lab.
Provide the IP ranges, CIDR, gateway IP, etc. Click Add to save the configuration.
Click Save to close the wizard.
Click the Run Check button to validate the TEP VLAN MTU check. Any mismatch in MTU can cause network issues and deployment failures.
To add the second edge node in the cluster, select the first node and click the clone button to copy the common settings.
Provide the information for the second edge node, then click Apply.
The SDDC Manager can auto-generate passwords for the edge VMs. If your environment uses a strict password policy, use the toggle button to disable automatic password generation and set the admin and root credentials for the Edges.
Tier-0 Gateway and BGP Configuration
Provide a name for the Tier-0 gateway and select the HA mode.
If you are planning to deploy VKS in your management domain, set the “High Availability Mode” to Active/Standby.
Under Routing Configuration, select the routing type, and if BGP is selected, specify the local ASN for the Tier-0.
Click on the “Set” button to create the edge uplinks.
Configure the uplink and BGP ASN as per the VLAN/uplink information from your documentation.
Repeat the process to configure the second uplink.
Repeat the process for configuring interfaces on the second edge node.
For the VPC connectivity, you must create an external IP block and a private transit gateway IP block. Two types of IP blocks are required:
- VPC external IP blocks: These are IP ranges owned by VCF and should not overlap with other ranges in the physical network. The Tier-0 gateway will advertise them to external routers via BGP.
- Private Transit Gateway IP Blocks: These IPs are not advertised to the northbound router and are scoped to the transit gateway’s project.
In VCF 9.1, the transit gateway private subnet must be set to /16; otherwise, the VKS deployment will fail. This behavior has changed in 9.1. In VCF 9.0, a /24 works just fine.
Click “Create New” to add the IP blocks.
For the external IP block, ensure that the visibility is set to public.
Repeat the process to create the transit gateway private subnet.
Click Next to go to the review configuration page.
On the review and deploy page, review the configuration and ensure it matches the desired configuration, then click the Deploy button to initiate deployment.
To check the status of the edge VM deployment, navigate to Configure > Networking > Edge Clusters. This option has been added in VCF 9.1.
It takes roughly 20 minutes in a physical lab for the edge VM deployment to complete and report status.
The external connection information can also be viewed under the networking page.
Notice the 2 new options, “VNA Clusters” and “Network Spans”; I will cover them in a later post of this series.
By default, the provider outbound SNAT is disabled. Edit the external connection and enable this option if the workloads need external connectivity.
And that’s it for this post. In the next post of this series, I will discuss the workload domain deployment!!!
I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing.