TKG Cluster Deployment Gotchas with Node Health Check in CSE 4.2

Posted on 25-02-2024 by Manish Jha

Recently, I upgraded Container Service Extension to 4.2.0 in my lab and was trying to deploy a TKG 2.4.0 cluster with node health check enabled. The deployment got stuck after deploying one control plane and worker node, and the cluster went into an error state.

Clicking on the Events tab showed the following error:

I checked the CSE log file and the capvcd logs on the ephemeral vm (before it got deleted) and found no error that would make sense to me.

I contacted CSE Engineering to discuss this issue and opened a bug for further analysis of the logs.

Root Cause

CSE Engineering debugged the logs and found that it was a bug in the product version. Here is the summary of the analysis done by Engineering.

This bug is caused by UI plugin 4.2.0 using the wrong API version in the MachineHealthCheck capi yaml object. The UI plugin was using API version v1beta2 is used instead of v1beta1, which is the correct value.

How to Integrate TMC Self-Managed 1.0 with VCD

Posted on 23-02-2024 by Manish Jha

Introduction

VMware Tanzu Mission Control is a centralized hub for simplified, multi-cloud, multi-cluster Kubernetes management. It helps platform teams take control of their Kubernetes clusters with visibility across environments by allowing users to group clusters and perform operations, such as applying policies, on these groupings.

VMware launched Tanzu Mission Control Self-Managed last year for customers running their Kubernetes (Tanzu) platform in an air-gapped environment. TMC Self-Managed is designed to support organizations that prefer to maintain complete control over their multi-cluster management hub for Kubernetes to take full advantage of advanced capabilities for cluster configuration, policy management, data protection, etc.

The first couple of releases of TMC Self-Managed only supported TKG clusters that were running on vSphere. Last month, VMware announced the release of the VMware Cloud Director Extension for Tanzu Mission Control, which allows installing TMC Self-Managed in a VCD environment to manage TKG clusters deployed through the VCD Container Service Extension (CSE).… Read More

Install Container Service Extension 4.2 in an Airgap Environment

Posted on 16-02-2024 by Manish Jha

Introduction

VMware Container Service (CSE) is an extension of VMware Cloud Director that enables cloud providers to offer Kubernetes as a service to their tenants. CSE helps tenants quickly deploy the Tanzu Kubernetes Grid clusters in their virtual data centers with just a few clicks directly from the tenant portal. Tenants can manage their clusters utilizing Tanzu products and services, such as Tanzu Mission Control, in conjunction with the VMware Cloud Director Container Service Extension.

Until CSE 4.0, the deployment of TKG clusters depended on internet connectivity to get the necessary installation binaries from the VMware public image registry. There was no support for the airgap environment.

With CSE 4.1, VMware introduced support for deploying CSE in an Airgap environment. Before diving into the nitty-gritty of configuring CSE, let’s look at the CSE airgap architecture.

CSE Airgap Architecture

The image below is from the CSE product documentation and depicts the high-level architecture and service provider workflow of CSE in an airgap setup.… Read More

VCD (10.5) Service Crashing Continuously in CSE Environment

Posted on 14-02-2024 by Manish Jha

After updating my lab’s Container Service Extension to version 4.2.0, I observed that the VMware VCD service was frequently crashing. Restarting the cell service did not help much, as the VCD user interface (UI) died again after five minutes. The cell.log was throwing below exception

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /opt/vmware/vcloud-director/logs/java_pid11530.hprof ...

1 2	java.lang.OutOfMemoryError: Java heap space Dumping heap to /opt/vmware/vcloud-director/logs/java_pid11530.hprof ...

You will find similar log entries in the cell-runtime.log file.

2024-02-12 15:25:07,873 | FATAL    | pc-activity-pool-13       | UncaughtExceptionHandlerStartupAction | Uncaught Exception. Originating thread: Thread[pc-activity-pool-13,5,main]. 

Message: Java heap space | activity=(com.vmware.vcloud.vimproxy.internal.impl.PCEventProcessingActivity,urn:uuid:5d072602-9517-4ce9-b00c-9e12a46e03bd)
java.lang.OutOfMemoryError: Java heap space

2024-02-12 15:25:07,873 | FATAL | pc-activity-pool-13 | UncaughtExceptionHandlerStartupAction | Uncaught Exception.

Container Service Extension 4.0 on VCD 10.x – Part 4: Tenant Operations

Posted on 22-03-2023 by Manish Jha

In the previous post in this series, I discussed the CSE configuration options that a service provider can use to provide Kubernetes-as-a-service to their tenants. In this post, I’ll go over how tenants can use the Container Service Extension plugin for Kubernetes cluster deployment in a self-service manner.

If you haven’t read the previous posts in this series, you can do so by clicking on the links provided below.

1: CSE Introduction & Architecture

2: NSX Advanced Load Balancer Configuration & VCD Integration

3: Container Service Extension Configuration by Service Provider

Log in to the tenant’s org to deploy a Kubernetes cluster. The user should be assigned the “Kubernetes Cluster Author” role. To begin with the cluster creation wizard, navigate to Home > More > Kubernetes Container Clusters and click the New button.

Select the Kubernetes runtime for the cluster. CSE 4.0 only supports Tanzu Kubernetes Grid runtime.

Choose the Kubernetes version and give the Kubernetes cluster a name.… Read More

Container Service Extension 4.0 on VCD 10.x – Part 3: Service Provider Configuration

Posted on 03-02-2023 by Manish Jha

The first two posts in this series covered CSE architecture and NSX ALB deployment/configuration. This post focuses on the steps taken by a service provider to set up a CSE deployment.

You can read the previous posts in this series by clicking on the links provided below.

1: CSE Introduction & Architecture

2: NSX Advanced Load Balancer Configuration & VCD Integration

At this time, it is assumed that the Service Provider has completed the following configurations in VCD:

vCenter is registered in VCD.
NSX-T is registered in VCD.
A Geneve-backed network pool is created in VCD.
Provider VDC has been created.

The service provider workflow for CSE deployment includes the following tasks:

Import Tier-0 gateway/VRF that is created for CSE in NSX-T.
Create an organization in VCD. This is a Service Provider managed organization that hosts the Container Service Extension server and any other extensions in the future. This is known as a Service/Solutions organization.

Container Service Extension 4.0 on VCD 10.x – Part 2: NSX Advanced Load Balancer Configuration

Posted on 27-01-2023 by Manish Jha

In part 1 of this blog series, I discussed Container Service Extension 4.0 platform architecture and a high-level overview of a production-grade deployment. This blog post is focused on configuring NSX Advanced Load Balancer and integrating it with VCD.

I will not go through each and every step of the deployment & configuration as I have already written an article on the same topic in the past. I will discuss the configuration steps that I took to deploy the topology that is shown below.

Let me quickly go over the NSX-T networking setup before getting into the NSX ALB configuration.

I have deployed a new edge cluster on a dedicated vSphere cluster for traffic separation. This edge cluster resides in my compute/workload domain. The NSX-T manager managing the edges is deployed in my management domain.

On the left side of the architecture, you can see I have one Tier-0 gateway, and VRFs carved out for NSX ALB and CSE networking.… Read More

Container Service Extension 4.0 on VCD 10.x – Part 1: Introduction & Architecture

Posted on 26-01-2023 by Manish Jha

Introduction

VMware Container Service is an extension to VMware Cloud Director which enables cloud providers to offer Kubernetes-as-a-Service to their tenants. CSE helps tenants quickly deploy the Tanzu Kubernetes Grid clusters in their virtual data centers in just a few clicks directly from the tenant portal. By using VMware Cloud Director Container Service Extension, customers can also use Tanzu products and services such as Tanzu Mission Control to manage their clusters.

Container Service Extension (CSE) has come a long way and with each release, the product is getting better and better. Folks who have worked on the older versions of CSE knows how painful it the setup process was and involved too many manual steps. With CSE 4.0, the provider workflow is simplified and the installation can be done in less than 30 minutes. Kudos to the CSE engineering team.

CSE 4.0 Benefits

I want to list a few benefits that CSE 4.0 offers before getting into the architecture.… Read More

Resizing TKGm Cluster in VCD

Posted on 28-12-2021 by Manish Jha

This blog post explains how to resize (horizontal scale) a CSE provisioned TKGm cluster in VCD.

In my lab, I deployed a TKGm cluster with one control plane and one worker node.

[root@cse ~]# kubectl get nodes
NAME        STATUS   ROLES                  AGE   VERSION
mstr-4fxl   Ready    control-plane,master   21h   v1.21.2+vmware.1
node-cstl   Ready    <none>                 21h   v1.21.2+vmware.1

[root@cse ~]# kubectl get nodes

NAME STATUS ROLES AGE VERSION

mstr-4fxl Ready control-plane,master 21h v1.21.2+vmware.1

node-cstl Ready <none> 21h v1.21.2+vmware.1

To resize the cluster through the VCD UI, go to the Kubernetes Container Clusters page and select the TKGm cluster to resize. Click on the Resize option.

Select the number of worker nodes you want in your TKGm cluster and click the Resize button.… Read More

Error Deploying Container Service Extension 3.1.1 – No module named ‘_sqlite3’

Posted on 28-12-2021 by Manish Jha

Container Service Extension 3.1.1 was released a few days back with new enhancements. The release announcements were made here and here.

Although the deployment procedure hasn’t changed much, mine was not smooth and I faced a couple of hiccups. This blog post discusses the problem I experienced and how I resolved it.

After installing VCD-CLI using pip, I was unable to execute any VCD command. The command was throwing an error as shown below:

[root@cse ~]# vcd version

Traceback (most recent call last):
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 18, in <module>
    from pysqlite2 import dbapi2 as sqlite3
ModuleNotFoundError: No module named 'pysqlite2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/bin/vcd", line 5, in <module>
    from vcd_cli.vcd import vcd
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/vcd.py", line 121, in <module>
    from vcd_cli import login  # NOQA
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/login.py", line 24, in <module>
    from vcd_cli import browsercookie
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 20, in <module>
    import sqlite3
  File "/usr/local/lib/python3.7/sqlite3/__init__.py",

Spread the Love

Introduction

Spread the Love

Introduction

CSE Airgap Architecture

Spread the Love

Spread the Love

Spread the Love

Spread the Love

Spread the Love

Introduction

CSE 4.0 Benefits

Spread the Love

Spread the Love

Spread the Love