In this post we will learn how to patch/update a Multi cell vCD 10.x environment.

Note: Above steps are for updating vCD from one build to another (patch release) within same version. Please do not confuse this with upgrading a vCD deployment where we jump from one major version to another.

I will breakup this post in 4 sections:

• Pre Update Checks.
• vCD Update Process.
• Post Update Checks.
• Post Update Tasks.

Pre Update Checks

There are number of checks which must be performed before attempting to update a vCD environment.

1: vCD Health Check & Primary Node identification: Before updating multi cell vCD environment, please ensure all vCD cells in server group are healthy and functioning correctly.

Also we need to identify vCD primary node as we will be needing this info later when we will be taking backup of the embedded database. To fetch this info we can make use of vCD Appliance API as shown below:

Method: GET

URL: https://<vcd-fqdn>:5480/api/1.0.0/nodes

Headers: x-vcloud-authorization: auth-token

Response Output:

From the above output we can infer that all 3 vCD nodes are healthy here, cluster health is also healthy and node “vcd1” is the current primary node.

Same can be fetched via CLI command also. Login to vCD cell over SSH and run this command: su – postgres -c ‘/opt/vmware/vpostgres/current/bin/repmgr cluster show’

Note: If any of the node status reports as “Unreachable”, then we should fix the issue first before proceeding to next state.

2: Verify vCenter-vCD Connectivity: Before starting vCD patching, we need to ensure that vCenter is in connected state in vCD. Why this is important? 

This is important because during vCD update process, cells will be quiesced sequentially and any new incoming tasks will be sent over to remaining cells (that are still active). If vCenter is in disconnected state in vCD, then those tasks will never reach to vSphere level for execution and will start piling in vCD until it times out.

If you try to quiesce and vCD cells that have lingering tasks on it, it might take long time for quiesce process to complete and you will have longer downtime.

vCenter connectivity can be checked by logging into vCD via system admin and navigating to vSphere Resources > vCenters

If you are an API lover and looking for automating this process, below API can help you.

2a: Get VIM Server Reference

Method: GET

URL: https://<vcd-fqdn>/api/admin/extension/vimServerReferences

Headers: x-vcloud-authorization, Accept: application/*+xml;version=32.0

Above API call will return a vimserver uuid n response which we have to use in next call.

Response Output:

2b: Fetch vCenter Status

Method: GET

URL: https://<vcd-fqdn>/api/admin/extension/vimServer/<vimserver uuid>

Headers: x-vcloud-authorization, Accept: application/*+xml;version=32.0

Sample Output: I have truncated the output for better readability, but basically we have to look for below items in output. 

3: Root password expiry: Before kicking vCD update, please ensure appliance root password has not expired. You can use standard Linux command to check this. 

vCD Update/Patch Process

Once pre updates checks have passed, we can start updating vCD by following below steps:

1: Download vCD Update Bundle: vCD update bundle is distributed as an tar file with a name of the form VMware_vCloud_Director_v.v.v.v- nnnnnnnn_update. tar.gz, where v. v. v. v represents the product version and nnnnnnnn the build number. Update bundle for vCD 10.x can be downloaded from here

2: Upload the update bundle to /opt/vmware/vcloud-director/data/transfer directory using winscp or similar utility. 

3: Extract the update bundle:

# mkdir -p /opt/vmware/vcloud-director/data/transfer/vcd-update-package

# tar -zxf VMware_vCloud_Director_10.0.0.4649-15450333_update.tar.gz -C /opt/vmware/vcloud-director/data/transfer/vcd-update-package

4: Set vcd-update-package directory as the update repository

# vamicli update –repo file:///opt/vmware/vcloud-director/data/transfer/vcd-update-package

5: Check for updates to verify that you established the update repository correctly.

# vamicli update –check

If updates are available you will see output as shown in below screenshot.

6: Quiesce vCD cell service: vCD cells are quiesced using cell management tool command which accepts username and password fields in plain text and are recorded into vCD logs. To avoid using username and password with cell-management tool, we can use below one liner:

/opt/vmware/vcloud-director/bin/cell-management-tool cell -i cat /var/run/vmware-vcd-cell.pid -t

Note: Thanks to Tomas for helping me out with this one liner. You can read more about this command in his blog

6a: Quisce cell service

# /opt/vmware/vcloud-director/bin/cell-management-tool cell -i cat /var/run/vmware-vcd-cell.pid -q

Once cell is quiesced, make sure running Job count on cell is 0

6b: Make sure there are no active task on cell which is quiesced

# /opt/vmware/vcloud-director/bin/cell-management-tool cell -i cat /var/run/vmware-vcd-cell.pid -t

Job count = 0
Is Active = false
In Maintenance Mode = false

7: Shutdown vCD Cell service

# /opt/vmware/vcloud-director/bin/cell-management-tool cell -i cat /var/run/vmware-vcd-cell.pid -s

8: Snapshot vCD cells: Take snapshot of vCD appliance once cell service is stopped.

Note: Don’t include memory in snapshot.

9: Install available update: Install update package using below command:

# vamicli update –install latest

Important: Repeat step 4-8 on remaining cells of the vCD server group.

10: Perform vCD Embedded DB Backup: DB backup needs to be taken only from primary cell.

# /opt/vmware/appliance/bin/create-db-backup

11: Perform database update: DB update can be kicked from any node.

# /opt/vmware/vcloud-director/bin/upgrade –unattended-upgrade

12: Perform rolling reboot of vCD nodes: Reboot all nodes one by one and make sure cell service is initialized on all nodes of server group.

vCD Post-Update Checks

1: Verify vCD has been updated to the intended build: cat /opt/vmware/vcloud-director/etc/global.properties|grep product.version

2: vCD Cell  & DB Cluster Health Status: Use API/Command from Pre-update check section.

3: vCenter-vCD Connectivity:  Use GUI/API option from pre-update check.

vCD Post Update Tasks

Once post update checks have passed, delete snapshot on all vCD nodes (as per your snapshot retention policy).

And that’s it for this post.

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing