Introduction

Tanzu Kubernetes Grid can be configured with a private registry for the rapid deployment of K8 workloads. Although there are a variety of container and artifact registries out there, Harbor has drawn attention because of its accessibility and ease of use, and rich feature set.

Although public registries are out there on the internet, they might contain everything you are looking for. In that case, you can create a custom Harbor registry to push custom K8 images to be used within your organization. A standalone Harbor registry is a perfect use case for an air-gapped TKG deployment.

In my last post, I have documented the steps of deploying a private Harbor registry for TKG. This post will show how you can leverage the registry to push/pull images for your K8 deployment. 

I have created a new project (named manish) in Harbor and I will be pushing images in that custom project.

I have also deployed a CentOS 7 VM where I have docker client installed and I have pulled some images from the docker hub to my local docker instance.

To push images to your harbor instance, you have to login to the harbor instance which you deployed. 

# docker login <harbor-fqdn> -u <username> -p <passed>

The next step is to tag the images that you already have in your local docker instance. Without tagging the image, you won’t be able to push the image.

An example is shown below to push the Yelb application instance to Harbor.

Once the images have been pushed, verify the images by running the docker images command again.

Setup Trust b/w TKG and Harbor

By default, TKG doesn’t trust any self-signed certificate, and if you try to pull images from a custom repository, it will error out complaining about the authenticity of the harbor certificate. 

A kubectl describe on the pod, reveals the real reason for the failure. Kubectl event for the failed pod clearly shows that certificate authenticity can’t be verified.

Workarounds

There are a couple of workarounds in this case:

1: Copy the ca.crt file from the harbor node and pasted the cert manually on all worker nodes that were part of the TKG workload cluster and  execute the below command:

# openssl x509 -in /etc/ssl/certs/myca.pem -text >> /etc/pki/tls/certs/ca-bundle.crt

2: Disable https access to harbor node: When configuring harbor, comment the HTTPS option in the harbor.yml file and proceed with the installation. 

For Lab/POC environments, you can apply the workarounds, but you should know the right way to do this when it comes to a Prod environment. 

The Right Procedure

The right way to add custom CA certificates in TKG workload nodes is by using a ytt overlay file to enable the cluster nodes to pull images from a container registry that uses self-signed certificates. 

For vSphere environments, the ytt overlay file is present in the directory $HOME/.tanzu/tkg/providers/infrastructure-vsphere/ytt. Look for a file named vsphere-overlay.yaml in this directory. 

Note: The above mentioned directory is created when you install the tanzu cli utility on a machine. 

Modify the yaml file using the below code:

Show More

Also copy the ca.crt file from the harbor node, and save it as tkg-custom-ca.pem in the directory /etc/ssl/certs/

When a new workload cluster is deployed, the above overlay code is executed and it adds the custom CA certificates to all nodes so that containerd and other tools start trusting the harbor certificate.

Note: If you already have workload clusters deployed, follow method#1 mentioned in the workaround section of this post.

Now when you deploy any application from your private Harbor registry, you will not see any failures during the deployment. 

And that’s it for this post.

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing.