Table of Contents
In first Post of this series, I talked about high level architecture of CSE infrastructure. I also discussed about various components that makes up the CSE platform. In this post I will walk through steps of installing & configuring CSE server.
CSE Installation Prerequisites
Before starting with CSE server installation, make sure following requirements are met:
1: VCD installed & configured: For Lab/POC environment, single node VCD installation is sufficient. For production environment 3 or more nodes (configured behind lb) is recommended.
2: Organization & Catalog for CSE: Dedicated Org created in VCD for CSE consumption. This org should have a Routed Org Network which has outbound connectivity to internet. Also this org should have a catalog created in advance. This catalog holds the K8’s ready vApp templates and will be shared to tenants for consumption.
3: AMQP broker configured in VCD: To extend VCD Public API, AMQP broker needs to be configured beforehand.
4: Dedicated Service Account for CSE (with System Admin Rights)
5: Dedicated Exchange and Service Account for CSE in AMQP. This needs to be different than what is configured in VCD.
6: Cent OS (or any other linux distro) VM with 1 CPU, 4 GB Mem and 25 GB disk.
Below slideshow depicts the above requirements configured in my lab.
Yum setup & Python Installation
Once above prerequisites are met, connect to the linux vm designated for CSE installation and run following commands:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# yum update -y # yum install open-vm-tools wget curl net-tools vim yum-utils sqlite sqlite-devel gcc openssl-devel bzip2-devel libffi-devel -y # yum groupinstall -y development # wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz # tar xzf Python-3.7.4.tgz # cd Python-3.7.4 # ./configure --enable-optimizations # make altinstall # python3.7 -V # yum install python3-pip -y # pip3.7 install --user --upgrade pip |
Install CSE Modules
|
1 2 3 4 5 6 7 8 9 |
# pip3.7 install --user container-service-extension # export PATH="/root/.local/bin:$PATH" # cse version CSE, Container Service Extension for VMware vCloud Director, version 2.6.1 # vcd version vcd-cli, VMware vCloud Director Command Line Interface, 23.0.0 |
Enable CSE Client
vcd version command creates a file named .vcd-cli/profile.yaml in the directory from where this command was run. Edit profile.yaml file and add below line at the end of the file (as shown in screenshot)
extensions:
– container_service_extension.client.cse
CSE Server Configuration File
CSE server installation is controlled by a yaml configuration file that needs to be filled before performing installation. You can generate a sample using command:
# cse sample -o config.yaml
This file has various sections that needs to be filled as per your infrastructure details. These sections are explained in greater detail here
This is how my config.yaml file looks like
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
amqp: exchange: cse-exchange host: mgmt-rmq01.vstellar.local password: VMware1! port: 5672 prefix: vcd routing_key: cse ssl: false ssl_accept_all: false username: cse-amqp vhost: / vcd: api_version: '33.0' host: vcd.vstellar.local log: true password: VMware1! port: 443 username: svc_cse verify: false vcs: - name: 172.16.10.110 password: VMware1! username: administrator@vsphere.local verify: false service: enforce_authorization: false listeners: 5 log_wire: false telemetry: enable: true broker: catalog: CSE default_template_name: ubuntu-16.04_k8-1.17_weave-2.6.0 default_template_revision: 1 ip_allocation_mode: pool network: CSE-Routed-NW org: cse_org remote_template_cookbook_url: https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template.yaml storage_profile: 'vSAN Default Storage Policy' vdc: CSE-VDC01 |
CSE Server Installation
Starting with CSE 2.6.0, CSE server commands will accept only encrypted configuration files by default. Run below command to generate encrypted config file:
# cse encrypt config.yaml –output encrypted-config.yaml
Also we need to change the permission of encrypted file so that only current user can modify it.
# chmod 600 encrypted-config.yaml
Before installing CSE, ensure that your encrypted config file is valid. This validation is done by running command:
# cse check encrypted-config.yaml
Note: Above command ask for decryption password which you set while encrypting your original config file and then performs number of checks.
|
1 2 3 4 5 6 7 8 9 |
Password for config file decryption: Decrypting 'encrypted-config.yaml' Validating config file 'encrypted-config.yaml' Connected to AMQP server (mgmt-rmq01.vstellar.local:5672) InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. Connected to vCloud Director (vcd.vstellar.local:443) Connected to vCenter Server '172.16.10.110' as 'administrator@vsphere.local' (172.16.10.110:None) Config file 'encrypted-config.yaml' is valid |
Now we are ready to install CSE server.
# cse install -c encrypted-config.yaml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
Password for config file decryption: Decrypting 'encrypted-config.yaml' Validating config file 'encrypted-config.yaml' Connected to AMQP server (rmq01.mgmt.vmw:5672) InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. Connected to vCloud Director (vcd.cloud.vmw:443) Connected to vCenter Server 'vcenter.rp1.vmw' as 'administrator@vsphere.local' (192.168.101.30:443) Config file 'encrypted-config.yaml' is valid Installing CSE on vCloud Director using config file 'encrypted-config.yaml' Connected to vCD as system administrator: vcd.cloud.vmw:443 Checking for AMQP exchange 'cse-exchange' AMQP exchange 'cse-exchange' is ready Updated cse API Extension in vCD Right: CSE NATIVE DEPLOY RIGHT added to vCD Right: CSE NATIVE DEPLOY RIGHT assigned to System organization. Right: PKS DEPLOY RIGHT added to vCD Right: PKS DEPLOY RIGHT assigned to System organization. Found catalog 'CSE' |
cse install command will start installing the templates as per templates defined in templates.yaml file (in config.yaml). To know more about templates available in current release of VSE, please check out CSE Template Announcements
You can also list available templates by running command:
# cse template list -d remote -c encrypted-config.yaml
Note: CSE installation takes a bit of time as templates are pulled from internet and then uploaded into CSE catalog.
Once the templates are uploaded, CSE deploys a temporary vm (per template) and install the Kubernetes inside those vm’s and then re-export modified vm’s as templates in CSE catalog. These new template will then be used by CSE to create Kubernetes clusters when tenants invokes vcd-cli to create K8 clusters.
Note: Customization of temp vm’s are carried out by various scripts that are located in directory .cse_scripts/<template-name>/
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
[root@vcd-cse ~]# cd /root/.cse_scripts [root@vcd-cse .cse_scripts]# ls -l drwxr-xr-x. 3 root root 104 Jun 23 22:03 photon-v2_k8-1.14_weave-2.5.2_rev2 drwxr-xr-x. 3 root root 104 Jun 23 20:51 ubuntu-16.04_k8-1.15_weave-2.5.2_rev3 drwxr-xr-x. 3 root root 104 Jun 23 20:06 ubuntu-16.04_k8-1.16_weave-2.6.0_rev1 drwxr-xr-x. 3 root root 123 Jun 23 19:44 ubuntu-16.04_k8-1.17_weave-2.6.0_rev1 [root@vcd-cse .cse_scripts]# cd ubuntu-16.04_k8-1.15_weave-2.5.2_rev3 [root@vcd-cse ubuntu-16.04_k8-1.16_weave-2.6.0_rev1]# ls -l drwxr-xr-x. 2 root root 116 Jun 23 20:04 cluster-upgrade -rw-------. 1 root root 3222 Jun 23 20:06 cust.sh -rw-------. 1 root root 444 Jun 23 20:04 init.sh -rw-------. 1 root root 631 Jun 23 20:04 mstr.sh -rw-------. 1 root root 228 Jun 23 20:04 nfsd.sh -rw-------. 1 root root 208 Jun 23 20:04 node.sh |
Validate CSE Installation: Once CSE is installed, we can use cse check command to validate it has been installed correctly or not.
# cse check -i -s
|
1 2 3 4 5 6 7 8 |
--- snipped output--- Config file 'config.yaml' is valid Validating CSE installation according to config file AMQP exchange 'cse-exchange' exists CSE on vCD is currently enabled Found catalog 'CSE' CSE installation is valid |
Run CSE Service: Once K8’s templates are installed in VCD, we can run CSE server service by invoking command:
# cse run -c encrypted-config.yaml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
--- snipped output--- Validating CSE installation according to config file AMQP exchange 'cse-exchange' exists CSE on vCD is currently enabled Found catalog 'CSE' CSE installation is valid Started thread 'MessageConsumer-0 (140157509945088)' Started thread 'MessageConsumer-1 (140157501552384)' Started thread 'MessageConsumer-2 (140157493159680)' Started thread 'MessageConsumer-3 (140157484766976)' Started thread 'MessageConsumer-4 (140157476374272)' Container Service Extension for vCloud Director Server running using config file: /root/encrypted-config.yaml Log files: cse-logs/cse-server-info.log, cse-logs/cse-server-debug.log waiting for requests (ctrl+c to close) |
Note: If you press ctrl+c here, CSE service is stopped, so we need to find a way to control CSE service automatic startup/shutdown. This is achieved by creating a script file that systemd process can reference to control the service using regular systemctl command.
Controlling CSE service with systemctl
Create a script file as shown below
# vim cse.sh
|
1 2 3 4 5 |
#!/usr/bin/env bash export CSE_CONFIG_PASSWORD='VMware1!' /root/.local/bin/cse run -c /root/encrypted-config.yaml |
# chmod +x cse.sh
Create cse.service file so that systemd can control it.
# vim /etc/systemd/system/cse.service
|
1 2 3 4 5 6 7 8 9 10 |
[Service] ExecStart=/bin/sh /root/cse.sh Type=simple User=root WorkingDirectory=/root Restart=always [Install] WantedBy=multi-user.target # systemctl daemon-reload |
# systemctl enable cse
Created symlink from /etc/systemd/system/multi-user.target.wants/cse.service to /etc/systemd/system/cse.service.
# systemctl start cse
Now if we run the systemctl status cse command, we will see service is now running and since we have enabled this service, it will be persistent across system reboot.
And that completes the CSE server installation & configuration. In next post of this series, I will walk through CSE client setup and K8 cluster deployment.
I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing 🙂