Introduction

VMware Container Service is an extension to VMware Cloud Director which enables cloud providers to offer Kubernetes-as-a-Service to their tenants. CSE helps tenants quickly deploy the Tanzu Kubernetes Grid clusters in their virtual data centers in just a few clicks directly from the tenant portal. By using VMware Cloud Director Container Service Extension, customers can also use Tanzu products and services such as Tanzu Mission Control to manage their clusters.

Container Service Extension (CSE) has come a long way and with each release, the product is getting better and better. Folks who have worked on the older versions of CSE knows how painful it the setup process was and involved too many manual steps. With CSE 4.0, the provider workflow is simplified and the installation can be done in less than 30 minutes. Kudos to the CSE engineering team.

CSE 4.0 Benefits

I want to list a few benefits that CSE 4.0 offers before getting into the architecture.

1: Provider Specific Workflow: CSE 4.0 ships with Kubernetes Container Clusters UI Plugin (4.0) for CSE.  This plugin enables the ‘CSE management tab’ in VCD UI and it provides a guided workflow for the service provider to prepare the environment for tenant users to create clusters. Using this new plugin, the service provider can onboard tenants in a matter of minutes.

2: Multi-node control planes for the Kubernetes cluster: The previous release of CSE supported a single-node control plane for the Kubernetes (TKGm) clusters deployed by tenants. CSE 4.0 allows creating tkgm clusters using multiple control plane nodes for the high-availability.

3: Kubernetes clusters enhanced life-cycle management: You can now carry out cluster life-cycle management operations including creating, upgrading, resizing and deleting Kubernetes clusters directly using the Kubernetes Container Clusters UI plugin.

These are some of the enhancements that stand out in this release of CSE. For the full list of what’s new in CSE 4.0, please see CSE 4.0 release notes. 

CSE 4.0 Platform Architecture

The 4.0 release of CSE requires deploying the CSE server (now available as a vApp template) in one of the organization owned by the service provider. This is the organization where the service provider deploys the add-ons for VCD. For simplicity let’s call this organization a ‘solution organization‘. CSE Server is deployed in one of the VDC created inside this organization and is connected to a routed org network configured for outbound access. CSE Server talks to VCD provider APIs using the public API endpoint configured in VCD.

Tenants deploy tkgm clusters on routed networks in their respective organizations. The network that is chosen for the Kubernetes cluster deployment must be configured for outbound access (internet). It is important to note that the Container Service Extension server is involved only during the creation or deletion of Tanzu Kubernetes Clusters, once the Tanzu Kubernetes Clusters are created, these clusters are self-managed. I will discuss this in greater detail in the next blog of this series.

The deployed Kubernetes cluster comes with pre-installed Tanzu core packages. Tenants can enhance cluster functionality by installing additional tanzu packages such as:

• Contour – Provides layer 7 ingress control to deployed HTTP(S) applications.
• Fluent Bit – Collects data and logs from different sources, unifies them, and sends them to multiple destinations.
• Prometheus – Provides out-of-the-box health monitoring of Kubernetes clusters. The Tanzu Kubernetes Grid implementation of Prometheus includes an Alert Manager. You can configure Alert Manager to notify you when certain events occur.
• Grafana – Provides monitoring dashboards for displaying key health metrics of Kubernetes clusters.
• Harbor Image Registry – Provides a centralized location to push, pull, store, and scan container images used in Kubernetes workloads. It supports storing artifacts such as Helm Charts and includes enterprise-grade features such as RBAC, retention policies, automated garbage clean up, and docker hub proxying.

Based on the NSX ALB Service Engine design, Service Engines are consumed/deployed in the compute cluster and accommodate the Virtual Service and VIP created for the tenant Kubernetes cluster.

CSE Deployment Architecture

The below architecture is a recommended architecture for a production-grade deployment of Container Service Extension. A big shout out to my colleague @Ranjith for working tirelessly on finalizing this design.

The next blog posts in this series are written keeping the above architecture in mind. I will discuss the design in greater detail. Stay tuned!!!

Summary

CSE 4.0 addresses the opportunity for Cloud Providers to extend their potent VCD based multi-tenant constructs to offer Kubernetes as a Service (KaaS) in addition to the existing Infrastructure as a Service (IaaS) for multi-cloud and private cloud, as businesses all over the world increase their adoption of container technology to run applications.

Tenants can now unlock additional opportunities by integrating their Kubernetes cluster with Tanzu Ecosystem. The clusters can be registered in Tanzu Mission Control (TMC) which provides full control on day-2 operations. Tanzu Service Mesh can be enabled (through TMC) on the registered Kubernetes clusters to provide service mesh features for the applications deployed in the clusters. Tenants can also leverage the VMware Cloud Director extension for VMware Data Solutions for delivering caching, messaging, and database services at scale. 

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing.