In a vSphere with Tanzu environment, when you enable Workload Management, the Supervisor cluster that gets deployed operates as the management cluster. After supervisor cluster is deployed, you can provision two types of workload clusters
- Tanzu Kubernetes clusters.
- Clusters based on a ClusterClass (aka Classy Cluster).
TKG on vSphere 8 provides different sets of APIs to deploy a TKC or a Classy cluster:
- v1alpha3 API
- v1beta1 API
When you deploy a cluster using v1beta1 API, you get a Classy Cluster or TKG 2.0 cluster which is based on a default ClusterClass definition.
By default workload cluster don’t trust any self-signed certificates. Prior to TKG 2.0 clusters, the easiest way to make Tanzu Kubernetes Clusters (TKCs) to trust any self-signed CA certificate was to edit tkgserviceconfigurations and define your Trusted CAs there. This setting was then enforced on any newly deployed TKCs.
For TKG 2.0 clusters, the process has changed a bit and in this post I will walk through configuring the same. I am demonstrating example of trusting self-signed certificate of internal harbor registry in this example.
Step 1: Create a Kubernetes secret definition YAML file
Create a yaml file to store your kubernetes secret for the additional CA certificates that your workload cluster should trust.
|
1 2 3 4 5 6 7 8 9 |
apiVersion: v1 data: harbor-ca: | TFMwdExTMUNSVWRKVGlCRFJWSlV..........TVVpKUTBGVVJTMHRM kind: Secret metadata: name: svc-tkc-user-trusted-ca-secret namespace: tmcsm type: Opaque |
You have to customize the yaml file as per below:
- namespace: This is the name of the vSphere Namespace in which you will deploy your workload cluster.
- name: This is a very important field. Any mistake in the name will lead to the cluster deployment failure. The name of the secret should include your workload cluster name followed by a fix value ‘user-trusted-ca-secret’. For e.g. if the name of your workload cluster that you will deploy is tkc01, the corresponding secret name will be ‘tkc01-user-trusted-ca-secret’
- data: This name can be any user defined name whose value is a double base64-encoded certificate.
Important: If the contents of the data map value are not double base6-encoded, the resulting PEM file cannot be processed.
So what do we mean by double encoding here? It simply means that you have to encode your CA certificate twice. You can run a simple command to do double encoding: cat ca.crt | base64 | base64 where ca.crt is the file that contains certificate contents of your root CA.
Step 2: Switch the context to the vSphere namespace and create the secret.
|
1 2 3 |
# kubectl config use-context <context name> # kubectl apply -f harbor-ca.yaml |
Step 3: Create a TKG 2.0 cluster deployment yaml
You can refer to the TKGS product documentation to see the examples listed for TKG 2.0 cluster creation. A sample yaml file is shown below for reference.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: svc-tkc namespace: tmcsm spec: clusterNetwork: services: cidrBlocks: ["198.52.100.0/12"] pods: cidrBlocks: ["192.168.100.0/16"] topology: class: tanzukubernetescluster version: v1.23.8---vmware.2-tkg.2-zshippable controlPlane: replicas: 3 workers: machineDeployments: - class: node-pool name: np01 replicas: 3 variables: - name: vmClass value: best-effort-large - name: storageClass value: vsan-default-storage-policy - name: defaultStorageClass value: vsan-default-storage-policy - name: trust value: additionalTrustedCAs: - name: harbor-ca |
Important: Name of the additional trusted CA should match with the name that you have specified under the field ‘data’ in the Kubernetes secret definition file created in step1.
Step 4: Create Workload cluster
|
1 |
# kubectl apply -f svctkc.yaml |
Step 5: Create a registry secret for kapp-controller
Kapp-controller is set up automatically when you deploy TKG 2.0 workload clusters in Supervisor running on vSphere 8. The kapp-controller must also trust the CA certificate of the internal harbor repo in order to deploy packages and applications from it. To activate trust for kapp, follow the instructions below.
5.1: Login to the TKG 2.0 workload cluster
|
1 2 3 4 5 |
# kubectl vsphere login --server=<supervisor control plane vip> --vsphere-username administrator@vsphere.local --insecure-skip-tls-verify=true --tanzu-kubernetes-cluster-name <cluster name> --tanzu-kubernetes-cluster-namespace <namespace name> Example: kubectl vsphere login --server=172.14.24.3 --vsphere-username administrator@vsphere.local --insecure-skip-tls-verify=true --tanzu-kubernetes-cluster-name svc-tkc --tanzu-kubernetes-cluster-namespace tmcsm # kubectl config use-context svc-tkc |
5.2: Get the configMap name of the kapp-controller
|
1 2 3 |
# kubectl get cm -n tkg-system | grep kapp kapp-controller-config 5 27h |
5.3: Create secret for kapp-controller
Before executing the below command, create a file named ‘registry_certs‘ and paste the certificate contents of your harbor CA certificate.
Important: Don’t use any other name for the certificate file, otherwise the operation would fail.
|
1 2 3 |
# kubectl create secret generic <kapp configMap name> --namespace tkg-system --from-file caCerts=registry_certs Example: kubectl create secret generic kapp-controller-config --namespace tkg-system --from-file caCerts=registry_certs |
Now that the workload cluster has been installed, you can deploy applications and tanzu packages from your private repository.
I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing.