In my last post on the NSX-T series, I explained how VTEP, MAC & ARP table is constructed. This knowledge is needed to understand packet flow.

In this post, I will demonstrate how packet forwarding is performed for East-West traffic.  

NSX-T has an inbuilt tool called Traceflow, which is very handy when analyzing packet flow within/across segments. This tool is located under Plan & Troubleshoot > Troubleshooting tools > Traceflow in NSX-T UI.

This tool is very easy to use and you just need to select the source vm & destination vm and click on trace to start packet flow analysis.

There are 2 deployment models available for T1 gateways. We can instantiate the T1 gateway on an edge cluster or we can choose not to associate with any edge cluster. If we need stateful services on the T1 gateway, we go with the first deployment model. 

In part-1 of this post, I will demonstrate packet walk when T1 is associated with edge-cluster. Since T1 GW is associated with the edge cluster, we will see both the SR & DR components of T1 on the edge.

There are various cases that I need to discuss here and I will take them one by one. 

My lab topology looks like this as shown below image. 

Note: Before starting the packet walk, I am assuming that TEP, MAC & ARP table has already been formed by NSX-T.

Case 1: VMs are on the same segment and same host

Here is the logical view of when 2 VM’s connected to the same segment.

Here is the sequence of events that takes place when 2 VMs sitting on the same logical segment & same host communicates with each other.

Step 1: App01 VM sends a layer 2 packet to App02 VM. This layer 2 packet has info of source IP/MAC & destination IP/MAC.

Step 2: The packet is received by the logical segment and it examines the headers of the packet to determine where this packet is coming from and where it is headed to. 

Step 3: Mac table associated with that segment is then checked to determine the MAC address of App02 VM is associated with which VTEP.

Step 4: On examining the MAC table, the ESXi host finds that the MAC address of App02 is local to it (as the outer mac (VTEP Mac) is the same for both the VMs).

Step 5: The packet is sent directly to App02 vm.

Case 2: VMs are on the same segment but on different hosts.

Step 1: App01 VM sends a layer 2 packet to App02 VM.

Step 2: Mac table associated with the App-LS segment is checked to find out where App02 vm is sitting. ESXi host discovers that there is no MAC address for App02 vm in its LCP, but the same exists in the table “LCP Remote” and is behind VTEP 172.16.40.12. 

Since the destination Mac address is not local, the packet has to be forwarded to the VTEP of the host which is servicing App02.

Step 3: The packet is then handed over to source host TEP where it is encapsulated and Geneve headers are appended. Geneve headers contain information on source VTEP IP/Mac and destination VTEP IP/Mac.

Step 4: The encapsulated packet is then routed through the Geneve tunnel and it traverses through the physical uplink of the host and finally reaches the VTEP of the destination node.

Step 5: On the destination host, as soon as the packet arrives, it is decapsulated and Geneve headers are stripped and the packet is forwarded directly to the App02 vm.

Case 3: VMs are on a different segment but same host.

Step 1: App01 VM sends a layer 2 packet to Web-01 VM which is connected to a different segment.

Step 2: DR component then check its forwarding table to determine if it has a route to the destination network. 

Note: My web-01 vm is sitting in subnet 192.168.20.0/24

Step 3: If the route to the destination network is present in the forwarding table, the packet from the source vm is sent to the LIF (App-LS).

Step 4: Source LIF then sends the packet to the DR component of the T1 gateway. T1 gateway checks the forwarding table of DR component and determines packet intended to Web-01 vm is associated with subnet 192.168.20.0/24 and interface 5b26944e-fe3d-4954-9d8d-06b37a1d551a

Step 5: T1 GW then forward that packet to the Web-LS interface (5b26944e-fe3d-4954-9d8d-06b37a1d551a)

Step 6: Finally the packet is delivered to the Web-01 vm.

Case 4: VMs are on different segments and different hosts.

The workflow of this case is similar to Case-3. The only difference is packet has to traverse over the physical network after step 5. Before traversing through the physical network, the packet will be encapsulated at the source host and decapsulated at the destination host.

Case 5: VMs on different Tier-1 routers, Same ESXi host But Different segments

Please refer to the below diagram to see the path taken by a packet from source vm to destination vm. Notice that the packet will be traversing through both host TEP and edge TEP since the packet is leaving Tenant-A T1 GW and going all the way up to T0 gateway and then forwarded to Tenant-B T1 gateway. 

 

And that’s it for this post.

I hope you enjoyed reading the post. Feel free to share this on social media if it is worth sharing 🙂