The first two posts in this series covered CSE architecture and NSX ALB deployment/configuration. This post focuses on the steps taken by a service provider to set up a CSE deployment.
You can read the previous posts in this series by clicking on the links provided below.
1: CSE Introduction & Architecture
2: NSX Advanced Load Balancer Configuration & VCD Integration
At this time, it is assumed that the Service Provider has completed the following configurations in VCD:
- vCenter is registered in VCD.
- NSX-T is registered in VCD.
- A Geneve-backed network pool is created in VCD.
- Provider VDC has been created.
The service provider workflow for CSE deployment includes the following tasks:
- Import Tier-0 gateway/VRF that is created for CSE in NSX-T.
- Create an organization in VCD. This is a Service Provider managed organization that hosts the Container Service Extension server and any other extensions in the future. This is known as a Service/Solutions organization.
- An Organization Virtual Datacenter(VDC) under the Service Organization: This Organization VDC must be able to host the Container Service Extension server vApp. The organization VDC must be of type flex.
- Create Catalogs in the Service Organization: These catalogs are used to store and retrieve CSE Server OVAs, as well as to keep a repository of Tanzu Kubernetes Template OVAs. It is recommended to create two separate catalogs, one for storing the CSE vApp template (which should not be shared with any other organization) and the other for storing the TKG template ova’s, which can be shared in read-only mode with the tenant’s organization.
- Create Edge Gateway in the Service/Solutions organization and a routed network configured for outbound (internet) access. This is the network where the CSE server will be deployed.
- Install Kubernetes Container Clusters UI plug-in 4.0 for VCD. This enables the ‘CSE Management’ tab which the service provider uses to configure CSE prerequisites.
- Deploy & Configure the CSE server.
It’s time to dive into the lab and put the above workflow into action.
Step 1: Import Tier-0 Gateway in VCD
In my environment, I am using VRF for CSE and already imported it in VCD by navigating to Resources > Cloud Resources > Tier-0 Gateways.
At the time of the import, you will be asked to specify an IP pool for the gateway, which will be used to sub-allocate IPs to Tenants. Typically, this will be a block of public IP addresses in a production environment.
Because my lab is internal, I have specified an internal IP address range rather than a real public IP address pool.
Step 2: Create Solutions Organization
I have created an organization named “Solution” in my environment.
Ensure that “Catalog Sharing” is enabled for the solutions organization.
Don’t forget to set the vApp runtime and storage lease policy in your newly created organization to never expire.
Step 3: Create Organization VDC
I have created an organization VDC named ‘Solution VDC’ in the solutions org with the below configuration
| Configuration | Specification |
| Allocation Mode | Flex |
| CPU Allocation | 10 GHz |
| Memory Allocation | 30 GB |
| CPU and Memory Guaranteed | 100% |
| Storage Allocation | 200 GB |
| Maximum Provisioned Networks | 10 |
Step 4: Create Catalogs
I have created 2 catalogs named ‘CSE’ & ‘K8-Templates’. I have uploaded the CSE Server vApp into the CSE catalog and the Kubernetes template OVAs in the K8-Templates catalog.
For instructions on how to create a catalog in VCD, please see the product documentation.
Step 5: Create Edge gateway & Routed Network
To create a new edge gateway, navigate to Resources > Cloud Resources > Edge gateways and click on the New button.
5.1: Select the VDC that you have created in step 3.
5.2: Specify the name of the Tier-1 gateway and whether it will be backed up by a dedicated Tier-0 gateway. In my environment, I have a dedicated VRF for the CSE, so this option is enabled.
5.3: Select the Tier-0 gateway/VRF for the edge gateway.
5.4: Select the “edge cluster of the provider Tier-0 gateway” option.
5.5: Specify the primary IP for the edge gateway and sub-allocate the IP pool from the Available IPs (configured on Tier-0 gateway).
5.6 Post edge gateway creation, login to the tenant portal of the Solution organization and navigate to Networking > Networks to create a routed network.
5.7: Make sure to configure the DNS Settings on the routed network. The CSE server will be deployed on this network, and this network should be able to resolve the FQDN of the VCD public address.
5.8: Configure the firewall on the edge gateway for outbound access.
Because this is a lab environment, I have left the firewall wide open. In a production environment, the firewall should typically allow ports 80/443/53.
5.9: Configure the NAT rule for outbound access for the routed network.
Create a SNAT rule translating your routed org network to an external IP address configured for outbound access.
Step 6: Install & Configure Kubernetes Container Clusters UI plug-in 4.0
Download the Kubernetes Container Clusters UI plug-in from here
Navigate to Customize Portal page by clicking on the More > Customize Portal option and click on the upload button and select the plugin file.
Select the scope of publishing the plug-in.
Complete the wizard by clicking on the finish button.
Step 7: Configure CSE
The plugin adds a new feature called “Kubernetes Container Clusters” to VCD. The Service Provider uses this option for CSE configuration, and tenants use it to deploy Kubernetes clusters.
To configure the CSE prerequisites, click on the Kubernetes Container Clusters option and navigate to the ‘CSE Management tab. The CSE Management tab in Kubernetes Container Clusters UI plug-in 4.0 has the following sections:
- Getting Started
- Guidelines
- Server Details
The Getting Started section allows you to set up Container Service Extension in VCD through the Kubernetes Container Clusters UI plug-in 4.0. Through this page, service providers will be able to:
1: Download OVAs: Provides the links to download Required CSE and Tanzu Kubernetes OVAs.
2: Create Catalogs and Upload OVAs: Through this, you will be able to navigate to the Service/Solution Organization and Upload OVAs to Catalogs created under that Organization. You can skip this step if the OVAs are already uploaded to the Catalogs available in the organization.
3: Setup and configure CSE Server: This workflow automatically creates the following:
- Kubernetes Clusters Rights Bundle.
- CSE Admin Role.
- Kubernetes Cluster Author global role.
- VM sizing policies.
The ‘Kubernetes Clusters Rights Bundle’ and ‘Kubernetes Cluster Author’ global roles will be automatically published to all tenants.
In my environment, I have already downloaded the required OVAs and uploaded them to the catalog, so I started with step 3 here.
7.1: Upon clicking the option “Configure Settings for CSE Server“, you can start the configuration by clicking on the Start button which created the required items in the backend.
Under the ‘Configuration Parameters’ page, leave the CAPVCD, CSI, and CPI versions to the default value. You have to specify a GitHub token and select the VM sizing policy for the bootstrap VM which gets created when the tenant deploys a Kubernetes cluster.
Optionally you can configure Proxy and Syslog settings for the CSE server from here. If your environment doesn’t have a proxy server, just leave the fields blank.
7.2: Add VM Sizing Policies to Organization VDCs.
CSE created VM policies can be viewed by navigating to the Resources > Cloud Resources > VM Sizing Policies page.
To add VM sizing policies to an Org VDC, select the Org VDC and navigate to the VM Sizing page and click on the ADD button.
Select the policies that you want to add to the VDC.
7.3: Create a User with the ‘CSE Admin Role‘
To add a new user, go to the Administration > Users page in the provider portal and click on New. Enter the user’s username and password, and then select the “CSE Admin Role” from the Available roles.
7.4: Generate an API Token for the CSE user.
Log out of the VCD portal and login as the newly created CSE user. After successfully logging in, click on the user preferences of the CSE user.
Click on the New button under API Tokens sub-page.
Give the token a name and press the Create button.
Make a note of the API Token because it can only be viewed once after generation. You will need this token when configuring the CSE server.
After the API token generation, log out of the CSE user and log back in as the admin user.
Step 8: Deploy & Configure CSE Server
In the solutions org navigate to the Libraries > vApp Templates and select the CSE Server vApp template and click on the Create vApp option.
Accept the EULA and proceed to the next step.
Give the CSE Server a name and make sure the Runtime and Storage leases are set to Never Expire.
Select the compute resources and storage policy for the CSE server deployment.
Adjust the CPU/Memory allocation for the CSE server as per your environment requirements.
Leave the Hard disk allocation to the default.
Choose the network and IP allocation scheme for the CSE server.
Fill out the relevant information on the custom properties page. Ensure that the org of the CSE service account is set to the ‘System’ org.
Click on the finish button to complete the CSE server deployment wizard.
After deployment, turn on the CSE server VM. And with that, the Service provider workflow is complete. In the next post of this series, I will discuss the Tenant workflow.
I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing.