How to make NSX ALB 21.1.3 work with TKGm 1.5.1

Posted on 14-03-2022 by Manish Jha

To test TKGm 1.5.1 against the latest version of nSX ALB, I upgraded my ALB deployment to 21.1.3. The deployment of the TKG management and workload cluster went smoothly.

However, when I deployed a sample load balancer application that uses a dedicated SEG and VIP network, the service was waiting for an external IP assignment.

root@linjump [ ~ ]# kubectl get all
NAME                                   READY   STATUS    RESTARTS   AGE
pod/http-deployment-785df98964-dhvqj   1/1     Running   0          104s
pod/http-deployment-785df98964-xf6hk   1/1     Running   0          104s

NAME             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/l4-svc   LoadBalancer   100.70.116.153   <pending>     80:31082/TCP   103s

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/http-deployment   2/2     2            2           104s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/http-deployment-785df98964   2         2         2       104s

root@linjump [ ~ ]# kubectl get all

NAME READY STATUS RESTARTS AGE

pod/http-deployment-785df98964-dhvqj 1/1 Running 0 104s

pod/http-deployment-785df98964-xf6hk 1/1 Running 0 104s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/l4-svc LoadBalancer 100.70.116.153 <pending> 80:31082/TCP 103s

NAME READY UP-TO-DATE AVAILABLE AGE

deployment.

NSX ALB Signed Certificates and TKGm Installation Gotcha

Posted on 13-03-2022 by Manish Jha

The Problem

I recently replaced the self-signed NSX-ALB certificates with a CA-signed (Microsoft CA) certificate, which caused a new unanticipated issue with TKGm deployment.

The TKGm installer wizard was complaining about the certificate validity. I knew there was nothing wrong with the certificate validity on NSX ALB because it was replaced just a few hours ago. Nonetheless, I double-checked the certificate expiration date, which is set to 2024.

After some jiggling, I investigated the bootstrap machine CLI terminal, where I issued the tanzu management-cluster create command, and spotted the main problem right away.

This is the error shown in the CLI.

E0313 12:55:20.639848 1549280 avisession.go:666] Client error for URI: login. Error: Post "https://alb.vstellar.local/login": x509: certificate signed by unknown authority

1	E0313 12:55:20.639848 1549280 avisession.go:666] Client error for URI: login. Error: Post "https://alb.vstellar.local/login": x509: certificate signed by unknown authority

Since the certificate is not signed by a Public CA, the bootstrapper machine has no idea about the CA server who signed this cert.… Read More

Replacing NSX ALB Certificates with Signed Certificates

Posted on 13-03-2022 by Manish Jha

In this post, I will walk through the steps of replacing NSX ALB self-signed certificates with a CA-signed certificate. For the purpose of this demonstration, I am using Active Directory Certificate Service in my lab. I have a windows server 2019 deployed and additional roles configured for AD integrated Certificate Service.

Please follow the below procedure for replacing NSX ALB certificates.

Step 1: Generate Certificate Signing Request (CSR)

CSR includes information such as domain name, organization name, locality, and country. The request also contains the public key/private key, which will be associated with the certificate generated. A CSR can be generated directly from the NSX ALB portal, but that requires configuring a Certificate Management Profile or using the OpenSSL utility.

To generate a CSR via the NSX ALB portal, go to Templates > Security > SSL/TLS Certificates and click on the Create button, then select controller certificate from the drop-down menu.… Read More

Backup and Restore TKG Clusters using Velero Standalone

Posted on 06-03-2022 by Manish Jha

In my last post, I discussed how Tanzu Mission Control Data Protection can be used to backup and restore stateful Kubernetes applications. In this tutorial, I’ll show you how to backup and restore applications using Velero standalone.

Why Use Velero Standalone?

You might wonder why you need to use Velero standalone when TMC Data Protection exists to make the process of backing up and restoring K8 applications easier. The answer is pretty simple. Currently, TMC does not have the ability to restore data across clusters. Backup and restore are limited to a single cluster. This is not an ideal solution from a business continuity point of view.

You may simply circumvent this situation if you use Velero standalone. In this approach, Velero is installed separately in both the source and target clusters. Both clusters have access to the S3 bucket where backups will be kept. If your source cluster is completely lost due to a disaster, you can redeploy the applications by downloading the backup from S3 and then restoring it.… Read More

Backing Up Stateful Applications using TMC Data Protection

Posted on 05-03-2022 by Manish Jha

Introduction

Kubernetes is frequently thought of as a platform for stateless workloads because the majority of its resources are ephemeral. However, as Kubernetes grows in popularity, enterprises are deploying more and more stateful apps. Because stateful workloads require permanent storage for application data, you can no longer simply reload them in the event of a disaster.

As businesses invest extensively in Kubernetes and deploy more and more containerized applications across multi-clouds, providing adequate data protection in a distributed environment becomes a challenge that must be addressed.

Data Protection in Tanzu Mission Control (TMC) is provided by Velero which is an open-source project. Velero backups typically include application and cluster data like config maps, custom resource definitions, secrets, and so on, which would then be re-applied to a cluster during restoration. The resources that are using a persistent volume, are backed up using Restic.

In this post, I’ll show how to backup and recover a stateful application running in a Tanzu Kubernetes cluster.… Read More

Using Custom S3 Storage (MinIO) with TMC Data Protection

Posted on 01-03-2022 by Manish Jha

Introduction

Data protection in TMC is provided by Velero which is an open-source project that came with the Heptio acquisition.

When data protection is enabled on a Kubernetes cluster, the data backup is stored external to the TMC. TMC supports both AWS S3 and Custom S3 storage locations to store the backups. Configuring the AWS S3 endpoint is pretty simple as TMC provides a CloudFormation script that does all the backend tasks such as creating S3 buckets, assigning permissions, etc.

AWS S3 might not be a suitable solution in some use cases. For instance, a customer has already invested heavily in an S3 solution (MinIO, Cloudian, etc). TMC allows customers to bring their own self-provisioned AWS S3 bucket or S3-compatible on-prem storage locations for their Kubernetes clusters.

In this post, I will be talking about how you can use on-prem S3 storage for storing Kubernetes backups taken from TMC Data Protection.… Read More

Tanzu Kubernetes Grid 1.4 Installation in Internet-Restricted Environment

Posted on 30-12-2021 by Manish Jha

An air gap (aka internet-restricted) installation method is used when the TKG environment (bootstrapper and cluster nodes) is unable to connect to the internet to download the installation binaries from the public VMware Registry during TKG install, or upgrades.

Internet restricted environments can use an internal private registry in place of the VMware public registry. An example of a commonly used registry solution is Harbor.

This blog post covers how to install TKGm using a private registry configured with a self-signed certificate.

Pre-requisites of Internet-Restricted Environment

Before you can deploy TKG management and workload clusters in an Internet-restricted environment, you must have:

An Internet-connected Linux jumphost machine that has:
- A minimum of 2 GB RAM, 2 vCPU, and 30 GB hard disk space.
- Docker client installed.
- Tanzu CLI installed.
- Carvel Tools installed.
- A version of yq greater than or equal to 4.9.2 is installed.
An internet-restricted Linux machine with Harbor installed.

Resizing TKGm Cluster in VCD

Posted on 28-12-2021 by Manish Jha

This blog post explains how to resize (horizontal scale) a CSE provisioned TKGm cluster in VCD.

In my lab, I deployed a TKGm cluster with one control plane and one worker node.

[root@cse ~]# kubectl get nodes
NAME        STATUS   ROLES                  AGE   VERSION
mstr-4fxl   Ready    control-plane,master   21h   v1.21.2+vmware.1
node-cstl   Ready    <none>                 21h   v1.21.2+vmware.1

[root@cse ~]# kubectl get nodes

NAME STATUS ROLES AGE VERSION

mstr-4fxl Ready control-plane,master 21h v1.21.2+vmware.1

node-cstl Ready <none> 21h v1.21.2+vmware.1

To resize the cluster through the VCD UI, go to the Kubernetes Container Clusters page and select the TKGm cluster to resize. Click on the Resize option.

Select the number of worker nodes you want in your TKGm cluster and click the Resize button.… Read More

Error Deploying Container Service Extension 3.1.1 – No module named ‘_sqlite3’

Posted on 28-12-2021 by Manish Jha

Container Service Extension 3.1.1 was released a few days back with new enhancements. The release announcements were made here and here.

Although the deployment procedure hasn’t changed much, mine was not smooth and I faced a couple of hiccups. This blog post discusses the problem I experienced and how I resolved it.

After installing VCD-CLI using pip, I was unable to execute any VCD command. The command was throwing an error as shown below:

[root@cse ~]# vcd version

Traceback (most recent call last):
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 18, in <module>
    from pysqlite2 import dbapi2 as sqlite3
ModuleNotFoundError: No module named 'pysqlite2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/bin/vcd", line 5, in <module>
    from vcd_cli.vcd import vcd
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/vcd.py", line 121, in <module>
    from vcd_cli import login  # NOQA
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/login.py", line 24, in <module>
    from vcd_cli import browsercookie
  File "/root/.local/lib/python3.7/site-packages/vcd_cli/browsercookie/__init__.py", line 20, in <module>
    import sqlite3
  File "/usr/local/lib/python3.7/sqlite3/__init__.py",

Unable to delete TKGm clusters in VCD

Posted on 28-12-2021 by Manish Jha

I encountered an issue while playing with Container Service Extension 3.1.1 in my lab where I was unable to construct TKGm clusters. During troubleshooting, I discovered that the Rights Bundle “cse:nativeCluster Entitlement” was missing certain critical rights that are newly added with CSE 3.1.1.

On attempting to delete the failed clusters, the clusters stuck in the state “DELETE:IN_PROGRESS”.

On attempting to delete the failed cluster via vcd-cli, the operation failed with the error “RDE_ENTITY_NOT_RESOLVED“

[root@cse ~]# vcd cse cluster delete mj-tkc01 --yes

Usage: vcd cse cluster delete [OPTIONS] [NAME]
Try 'vcd cse cluster delete -h' for help.

Error: [ 0870ce65-3c19-4faa-a483-9ff52f4e9f06 ] RDE_ENTITY_NOT_RESOLVED [ 556ce860-1b72-4365-b327-3b38af622eca ]

[root@cse ~]# vcd cse cluster delete mj-tkc01 --yes

Usage: vcd cse cluster delete [OPTIONS] [NAME]

Try 'vcd cse cluster delete -h' for help.

Spread the Love

Spread the Love

Spread the Love

Spread the Love

Introduction

Spread the Love

Introduction

Spread the Love

Spread the Love

Spread the Love

Spread the Love

Spread the Love