Nsx Alb

Kubernetes

Vmware Vsphere

Nsx

Vsphere With Tanzu

Network Topology — vSphere IaaS control plane (vSphere with Tanzu)- (Part 2)

Navneet Verma

7 min read

·

Jul 5, 2024

--

1

Listen

Share

Introduction

We will discuss the recently introduced topology in Part Two of the series. This topology is a variation of the topology addressed in Part 1 of the series, wherein the load balancing function is offloaded to the AVI Load Balancer. The steps for performing the integration have been discussed in the official documentation and an article by the author here.

Initial configuration

We will start observing the NSX topology towards the end of this stage within the official documentation, where a T0 VLAN-backed uplink segment (VLAN 2135 in this example) has been created, providing North-South connectivity from NSX to the physical infrastructure. The AVI load balancer, NSX, and vCenter integrations have been completed per a previous blog that addressed the setup.

• A T0 gateway has also been created that provides North-South connectivity for the NSX logical networking to the physical infrastructure. T0 Gateway has a Gateway Firewall that, by default, allows ANY/ANY traffic.

AVI Load Balancer specific configuration

As for the AVI Load Balancer specific manual configurations/integrations

• Two T1 gateways are manually created — t1-mgmt and t1-data-dummy — with corresponding segments (avi-mgmt-segment and avi-data-dummy-segment).
• A Gateway firewall with any/any/any rule is configured for these gateways.
• The two T1 gateways have a DHCP server attached that dynamically provides IPs to the management interface of the AVI Service Engines (DHCP range of 192.168.100.10–192.168.100.50 ).
• The t1-data-dummy gateway and avi-data-dummy-segment segment are not used in further steps, so we will no longer display them in the topology diagram.

The topology looks similar to the picture above (Figure 1).

Supervisor enablement

Let us now enable the Supervisor on the Compute cluster using the following configurations (ignoring values that are not relevant) —

• Management network IP range 10.220.7.53–10.220.7.57
• Services CIDR (ClusterIP)10.96.0.0/23
• Tier-0 Gateway nsxt01-t0-tr
• NAT Mode Enabled
• Namespace Network 10.244.0.0/20
• Namespace subnet prefix /28
• Ingress 10.220.3.16/28
• Egress 10.220.3.144/28

Once the Supervisor completes the enablement process, the three Supervisor Control Plane VMs running kube-apiserver will be created. At this stage, the following objects are created within the NSX networking (refer to the topology diagram below — Figure 2) —

Press enter or click to view image in full size

• A T1 Gateway is created that is connected to the Tier-0 Gateway nsxt01-t0-tr .
• An overlay segment is created and attached to the T1 Gateway.
• Based on the Namespace subnet prefix value, a /28 subnet (16 IP addresses) is carved from the Namespace Network network range.
• The 1st IP of this new subnet 10.244.0.1/28 is the gateway address of the new segment.
• The 3 Supervisor Control Plane VMs are attached to the newly created segment, and their IP addresses (workload NICs) are obtained from the 10.244.0.0/28 subnet range.
• Since the Supervisor Control Plane VMs are dual-homed, their management NICs are attached to a standard VDS-backed port group (marked red in the diagram below) and assigned IPs from the Management network IP range of IP address range. They also share a floating IP address within this range of IP addresses.

Additionally, two services are configured on the T1 Gateway. They are -

• A Gateway firewall with Allow ANY/ANY/ANY rule
• With NAT mode enabled (default), 3 NAT rules. They are —
  - A NOSNAT rule for all traffic originating from Namespace Network 10.244.0.0/20 to the Ingress 10.220.3.16/28 pool
  - A NOSNAT rule for all traffic between the Namespace Network 10.244.0.0/20
  - A SNAT rule from any source traffic to any destination traffic to use a translated IP from the Egress 10.220.3.144/28 pool.

AVI Load Balancer specific configuration

With AVI Load Balancer in place to provide the Ingress load balancing, the supervisor enablement process delivers several automated configurations to achieve the load balancing. As shown in the Figure 2,

• With the T1 gateway created in the figure above, an overlay segment, avi-data-segment, is created to connect the AVI Load Balancer service engine’s data traffic interface.
• The avi-data-segment segment is configured for dynamic IP (via DHCP) within the range of 10.65.0.3 and 10.65.0.14
• AVI Service engine(s) are deployed, with the management interface connected to the avi-mgmt-segment and the data interface to the avi-data-segment. In this example, we are only displaying one service engine. Users can configure multiple service engines based on their HA requirements.

Network                                                     Discovered Subnet    Configured Subnet           Static IP Pool  VRF Context

vcf-ako-net-domain-c8:ddffb11d-a734–4442–9050–5356154b50cc  None                 10.220.3.16/28 [1/14]       1                global

• Within the AVI Controller, a new network is created within the Network Profile of the nxt-cloud (created and configured during the AVI controller installation and configuration). This network uses the Ingress 10.220.3.144/28 as the subnet IP pool (see above).

Press enter or click to view image in full size

• The above network has been added as a usable network to the IPAM Profile associated with the nxt-cloud
• With these additional settings, AVI creates two load balancers within the AVI service engines.
  - csi-controller service on ports 2112 and 2113 and
  - kube-apiserver on ports 6443 and 443.

The IP addresses for these load balancers are assigned from the Ingress IP address range.

vSphere Namespace creation

Let us create a new vSphere Namespace namespace1 using default settings. At this stage, the following objects are created within the NSX networking (refer to the topology diagram below — Figure 4) —

Press enter or click to view image in full size

• A T1 Gateway is created that is connected to the Tier-0 Gateway nsxt01-t0-tr .
• An overlay segment is created and attached to the T1 Gateway.
• Based on the Namespace subnet prefix value, the next /28 subnet (16 IP addresses — 10.244.0.16/28) is carved from the Namespace Network network range.
• The 1st IP of this new subnet 10.244.0.17/28 is the gateway address of the new segment.

Additionally, two services are configured on the T1 Gateway. They are -

• A Gateway firewall with Allow ANY/ANY/ANY rule
• With NAT mode enabled (default), 3 NAT rules. They are —
  - A NOSNAT rule for all traffic originating from Namespace Network 10.244.0.0/20 to the Ingress 10.220.3.16/28 pool
  - A NOSNAT rule for all traffic between the Namespace Network 10.244.0.0/20
  - A SNAT rule from any source traffic to any destination traffic to use a newly translated IP from the Egress 10.220.3.144/28 pool.

AVI Load Balancer specific configuration

• With the T1 gateway created in the figure above, a new overlay segment, avi-data-segment, is created to connect the AVI Load Balancer service engine’s data traffic interface.
• The new avi-data-segment segment is configured for dynamic IP (via DHCP) within the range of 10.65.0.3 and 103.65.0.14

Supervisor Service installation

Let us create a new Supervisor Service on the vSphere Namespace namespace1 and expose the service as a type LoadBalancer. At this stage, the following objects are created within the NSX networking (refer to the topology diagram below — Figure 5). To keep the topology diagram simple, we have excluded the connectivity of the Supervisor Control Plane VMs from Figure 5. —

Press enter or click to view image in full size

• As the pods get deployed on the ESXi servers as vSphere pods (a particular type of VM), they are attached to the Overlay segment created previously during the namespace creation process. They get their IP address from the subnet assigned to the overlay segment (10.244.0.16/28 ).

AVI Load Balancer specific configuration

• Since our Supervisor Service consists of a service type Loadbalancer, the AVI Service Engine(s) need connectivity to this T1 network. A new interface is added to the AVI Service Engine(s), and they are connected to the avi-data-segment segment that was created for this vSphere Namespace.
• With these additional settings, AVI creates a new balancer with backing virtual service with an IP from the Ingress pool of IPs.

Workload Cluster creation

For demonstration purposes, let us create a workload cluster in namespace1 . Within that namespace, we already have a Supervisor Service running. We will create the workload cluster using Cluster API. At this stage, the following objects are created within the NSX networking (refer to the topology diagram below — Figure 6) —

Press enter or click to view image in full size

• An overlay segment is created and attached to the T1 Gateway. Note that a new T1 gateway was not created. We only get one T1 Gateway per vSphere Namespace.
• Based on the Namespace subnet prefix value, the next /28 subnet range (16 IP addresses — 10.244.0.32/28) is carved from the Namespace Network network range.
• The 1st IP of this new subnet 10.244.0.33/28 is the gateway address of the new segment.
• This workload cluster’s Control Plane VM(s) and worker node VM(s) are attached to the newly created segment. Their IP addresses are obtained from the 10.244.0.32/28 subnet range assigned to the segment to which they are connected.

AVI Load Balancer specific configuration

• Since our Workload CLuster’s API servers need to be exposed through a load balancer, a new L4 virtual service is created within the AVI Service Engine(s) with an IP from the Ingress pool of IPs.
• Additionally, if there are any service type LoadBalancer that are created within the Workload Cluster, corresponding virtual services are created within the AVI Service Engine(s) with an IP from the Ingress pool of IPs.

Conclusion

This post clarifies the tight integration between NSX, vSphere, and AVI, enabling users to configure and automate networks effectively on vSphere with Tanzu using NSX/AVI.

In the next part of the series, we will explore how the VDS network works with vSPhere with Taznu.