TKG Cluster Deployment Gotchas with Node Health Check in CSE 4.2

25-02-202425-02-2024 Manish Jha

Recently, I upgraded Container Service Extension to 4.2.0 in my lab and was trying to deploy a TKG 2.4.0 cluster with node health check enabled. The deployment got stuck after deploying one control plane and worker node, and the cluster went into an error state.

Clicking on the Events tab showed the following error:

I checked the CSE log file and the capvcd logs on the ephemeral vm (before it got deleted) and found no error that would make sense to me.

I contacted CSE Engineering to discuss this issue and opened a bug for further analysis of the logs.

Root Cause

CSE Engineering debugged the logs and found that it was a bug in the product version. Here is the summary of the analysis done by Engineering.

This bug is caused by UI plugin 4.2.0 using the wrong API version in the MachineHealthCheck capi yaml object. The UI plugin was using API version v1beta2 is used instead of v1beta1, which is the correct value.

Install Container Service Extension 4.2 in an Airgap Environment

16-02-2024 Manish Jha

Introduction

VMware Container Service (CSE) is an extension of VMware Cloud Director that enables cloud providers to offer Kubernetes as a service to their tenants. CSE helps tenants quickly deploy the Tanzu Kubernetes Grid clusters in their virtual data centers with just a few clicks directly from the tenant portal. Tenants can manage their clusters utilizing Tanzu products and services, such as Tanzu Mission Control, in conjunction with the VMware Cloud Director Container Service Extension.

Until CSE 4.0, the deployment of TKG clusters depended on internet connectivity to get the necessary installation binaries from the VMware public image registry. There was no support for the airgap environment.

With CSE 4.1, VMware introduced support for deploying CSE in an Airgap environment. Before diving into the nitty-gritty of configuring CSE, let’s look at the CSE airgap architecture.

CSE Airgap Architecture

The image below is from the CSE product documentation and depicts the high-level architecture and service provider workflow of CSE in an airgap setup.… Read More

Container Service Extension 4.0 on VCD 10.x – Part 4: Tenant Operations

22-03-202322-03-2023 Manish Jha

In the previous post in this series, I discussed the CSE configuration options that a service provider can use to provide Kubernetes-as-a-service to their tenants. In this post, I’ll go over how tenants can use the Container Service Extension plugin for Kubernetes cluster deployment in a self-service manner.

If you haven’t read the previous posts in this series, you can do so by clicking on the links provided below.

1: CSE Introduction & Architecture

2: NSX Advanced Load Balancer Configuration & VCD Integration

3: Container Service Extension Configuration by Service Provider

Log in to the tenant’s org to deploy a Kubernetes cluster. The user should be assigned the “Kubernetes Cluster Author” role. To begin with the cluster creation wizard, navigate to Home > More > Kubernetes Container Clusters and click the New button.

Select the Kubernetes runtime for the cluster. CSE 4.0 only supports Tanzu Kubernetes Grid runtime.

Choose the Kubernetes version and give the Kubernetes cluster a name.… Read More

Container Service Extension 4.0 on VCD 10.x – Part 3: Service Provider Configuration

03-02-2023 Manish Jha

The first two posts in this series covered CSE architecture and NSX ALB deployment/configuration. This post focuses on the steps taken by a service provider to set up a CSE deployment.

You can read the previous posts in this series by clicking on the links provided below.

1: CSE Introduction & Architecture

2: NSX Advanced Load Balancer Configuration & VCD Integration

At this time, it is assumed that the Service Provider has completed the following configurations in VCD:

vCenter is registered in VCD.
NSX-T is registered in VCD.
A Geneve-backed network pool is created in VCD.
Provider VDC has been created.

The service provider workflow for CSE deployment includes the following tasks:

Import Tier-0 gateway/VRF that is created for CSE in NSX-T.
Create an organization in VCD. This is a Service Provider managed organization that hosts the Container Service Extension server and any other extensions in the future. This is known as a Service/Solutions organization.

Container Service Extension 4.0 on VCD 10.x – Part 2: NSX Advanced Load Balancer Configuration

27-01-202302-02-2023 Manish Jha

In part 1 of this blog series, I discussed Container Service Extension 4.0 platform architecture and a high-level overview of a production-grade deployment. This blog post is focused on configuring NSX Advanced Load Balancer and integrating it with VCD.

I will not go through each and every step of the deployment & configuration as I have already written an article on the same topic in the past. I will discuss the configuration steps that I took to deploy the topology that is shown below.

Let me quickly go over the NSX-T networking setup before getting into the NSX ALB configuration.

I have deployed a new edge cluster on a dedicated vSphere cluster for traffic separation. This edge cluster resides in my compute/workload domain. The NSX-T manager managing the edges is deployed in my management domain.

On the left side of the architecture, you can see I have one Tier-0 gateway, and VRFs carved out for NSX ALB and CSE networking.… Read More

Container Service Extension 4.0 on VCD 10.x – Part 1: Introduction & Architecture

26-01-202327-01-2023 Manish Jha

Introduction

VMware Container Service is an extension to VMware Cloud Director which enables cloud providers to offer Kubernetes-as-a-Service to their tenants. CSE helps tenants quickly deploy the Tanzu Kubernetes Grid clusters in their virtual data centers in just a few clicks directly from the tenant portal. By using VMware Cloud Director Container Service Extension, customers can also use Tanzu products and services such as Tanzu Mission Control to manage their clusters.

Container Service Extension (CSE) has come a long way and with each release, the product is getting better and better. Folks who have worked on the older versions of CSE knows how painful it the setup process was and involved too many manual steps. With CSE 4.0, the provider workflow is simplified and the installation can be done in less than 30 minutes. Kudos to the CSE engineering team.

CSE 4.0 Benefits

I want to list a few benefits that CSE 4.0 offers before getting into the architecture.… Read More

Resizing TKGm Cluster in VCD

28-12-202127-02-2022 Manish Jha

This blog post explains how to resize (horizontal scale) a CSE provisioned TKGm cluster in VCD.

In my lab, I deployed a TKGm cluster with one control plane and one worker node.

[root@cse ~]# kubectl get nodes
NAME        STATUS   ROLES                  AGE   VERSION
mstr-4fxl   Ready    control-plane,master   21h   v1.21.2+vmware.1
node-cstl   Ready    <none>                 21h   v1.21.2+vmware.1

[root@cse ~]# kubectl get nodes

NAME STATUS ROLES AGE VERSION

mstr-4fxl Ready control-plane,master 21h v1.21.2+vmware.1

node-cstl Ready <none> 21h v1.21.2+vmware.1

To resize the cluster through the VCD UI, go to the Kubernetes Container Clusters page and select the TKGm cluster to resize. Click on the Resize option.

Select the number of worker nodes you want in your TKGm cluster and click the Resize button.… Read More

Error Deploying Container Service Extension 3.1.1 – No module named ‘_sqlite3’

28-12-202127-02-2022 Manish Jha

Container Service Extension 3.1.1 was released a few days back with new enhancements. The release announcements were made here and here.

Although the deployment procedure hasn’t changed much, mine was not smooth and I faced a couple of hiccups. This blog post discusses the problem I experienced and how I resolved it.

After installing VCD-CLI using pip, I was unable to execute any VCD command. The command was throwing an error as shown below:

[root@cse ~]# vcd version

Traceback (most recent call last):
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 18, in <module>
    from pysqlite2 import dbapi2 as sqlite3
ModuleNotFoundError: No module named 'pysqlite2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/bin/vcd", line 5, in <module>
    from vcd_cli.vcd import vcd
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/vcd.py", line 121, in <module>
    from vcd_cli import login  # NOQA
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/login.py", line 24, in <module>
    from vcd_cli import browsercookie
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 20, in <module>
    import sqlite3
  File "/usr/local/lib/python3.7/sqlite3/__init__.py",

Unable to delete TKGm clusters in VCD

28-12-202127-02-2022 Manish Jha

I encountered an issue while playing with Container Service Extension 3.1.1 in my lab where I was unable to construct TKGm clusters. During troubleshooting, I discovered that the Rights Bundle “cse:nativeCluster Entitlement” was missing certain critical rights that are newly added with CSE 3.1.1.

On attempting to delete the failed clusters, the clusters stuck in the state “DELETE:IN_PROGRESS”.

On attempting to delete the failed cluster via vcd-cli, the operation failed with the error “RDE_ENTITY_NOT_RESOLVED“

[root@cse ~]# vcd cse cluster delete mj-tkc01 --yes

Usage: vcd cse cluster delete [OPTIONS] [NAME]
Try 'vcd cse cluster delete -h' for help.

Error: [ 0870ce65-3c19-4faa-a483-9ff52f4e9f06 ] RDE_ENTITY_NOT_RESOLVED [ 556ce860-1b72-4365-b327-3b38af622eca ]

[root@cse ~]# vcd cse cluster delete mj-tkc01 --yes

Usage: vcd cse cluster delete [OPTIONS] [NAME]

Try 'vcd cse cluster delete -h' for help.

Native Kubernetes in VCD using Container Service Extension 3.0

20-09-202127-02-2022 Manish Jha

Introduction

VMware Container Service is an extension to Cloud Director which enables VCD cloud providers to offer Kubernetes-as-a-Service to their tenants. CSE integration with VCD has allowed CSPs to provide true developer-ready cloud offering to VCD tenants. Tenants can quickly deploy the Kubernetes cluster in just a few clicks directly from the VCD portal.

Cloud Providers upload customized Kubernetes templates in public catalogs which tenants leverage to deploy K8 clusters in self-contained vApps. Once the K8 cluster is available, developers can use their native Kubernetes tooling to interact with the cluster.

To know more about the architecture and interaction of CSE components, please see my previous blog on this topic.

Container Service Extension 3.x went GA earlier this year and brought several new features/enhancements and one of them is supporting Tanzu Kubernetes Grid multi-cloud (TKGm) for K8 deployments and thus unlocking the full potential of consistent upstream Kubernetes in their VCD powered clouds.… Read More

Container Service Extension

TKG Cluster Deployment Gotchas with Node Health Check in CSE 4.2

Like this:

Install Container Service Extension 4.2 in an Airgap Environment

Introduction

CSE Airgap Architecture

Like this:

Container Service Extension 4.0 on VCD 10.x – Part 4: Tenant Operations

Like this:

Container Service Extension 4.0 on VCD 10.x – Part 3: Service Provider Configuration

Like this:

Container Service Extension 4.0 on VCD 10.x – Part 2: NSX Advanced Load Balancer Configuration

Like this:

Container Service Extension 4.0 on VCD 10.x – Part 1: Introduction & Architecture

Introduction

CSE 4.0 Benefits

Like this:

Resizing TKGm Cluster in VCD

Like this:

Error Deploying Container Service Extension 3.1.1 – No module named ‘_sqlite3’

Like this:

Unable to delete TKGm clusters in VCD

Like this:

Native Kubernetes in VCD using Container Service Extension 3.0

Introduction

Like this:

Spread the Love

Like this:

Introduction

CSE Airgap Architecture

Spread the Love

Like this:

Spread the Love

Like this:

Spread the Love

Like this:

Spread the Love

Like this:

Introduction

CSE 4.0 Benefits

Spread the Love

Like this:

Spread the Love

Like this:

Spread the Love

Like this:

Spread the Love

Like this:

Introduction

Spread the Love

Like this: