vSphere 6.0 brought many enhancements with it and one of the most significant among them was VMware Certificate Authority which is VMware’s own CA and it eases the pain of certificate management in vSphere 6.

VMCA is itself a fully functional CA and can be used to issue certificates to all vSphere 6 components (vCenter and ESXi hosts) in your environment. VMCA dont have any graphical interface like Microsoft CA and is totally command line driven.

VMCA is part of Platform services controller and there are various deployment model available for configuring VMCA including:

• VMCA as Root CA
• VMCA as Subordinate CA to an External Enterprise CA
• External CA
• Hybrid mode

Derek Seamen has explained about these deployment options in greater detail here

By default, the VMCA self-signs its own certificate which is used by vCenter server and Esxi hosts. If  your organization policy don’t allow using self-signed certs then you can replace the certs generated by VMCA and sending them to an enterprise CA for signing. The signed certificates can be then imported back into VMCA.

In this mode VMCA is acting as a subordinate CA to your enterprise CA and any certificate signed by the VMCA, can then be validated by clients with the root CA and VMCA certificates installed.

Lab setup

• I have a Root CA running in server 2012 R2 (casrv01.alex.local)
• I have a Platform Services Controller 6.0 U3 Appliance (psc02.alex.local)
• I have a vCenter Server 6.0 U3 Appliance (vcentersrv02.alex.local)

Create the Certificate Templates

We need to create 2 certificates templates as per VMware KB-2112009. I have listed the steps for doing so in this blog post. 

Generate VMCA Certificate Signing Request (CSR)

Note: Before proceeding with certs replacements, make sure to snapshot the PSC/vCenter server appliance.

SSH to the Platform Services Controller (or vCenter Server if using vCenter with Embedded PSC) and enable the BASH shell and set it to the default shell 

Launch the certificate-manager utility from /usr/lib/vmware-vmca/bin/certificate-manager and choose option 2 “Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates.”

Certificate-manager will ask a question about generating all certificates using configuration files. Press Y to proceed and provide the sso administrator credentials

Provide the details to populate certool.cfg file

Important

The certificate issued based on this certool.cfg configuration file is going to be the certificate used to secure the vSphere Web Client UI and this is the certificate that the end user will see when he clicks the padlock in the web browser.

The fields that are the most important to us are the Name field and the Hostname one. The text we type in the Name field will actually be the common name in the certificate, and the name that we provide in the Hostname field will be the Subject Alternative Name attribute of the certificate.

If you have a deployment with an external PSC you will type the FQDN of your PSC in both of these fields, and if you have a deployment with an embedded PSC you will type the FQDN of your vCenter server.

Select option 1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate and provide an output directory name to save the generated certs file.

The certificate-manager will then generate the vmca_issued_csr.csr and vmca_issued_key.key in the location you specified earlier.

psc02:~/certs # ll
-rw——- 1 root root 1159 Nov 4 07:12 vmca_issued_csr.csr
-rw——- 1 root root 1707 Nov 4 07:12 vmca_issued_key.key

Send vmca_issued_csr.csr file to your Enterprise CA for signing.

Obtain the signed Certificates from a Microsoft CA

1. Open a browser to your certificate authority web interface. i.e. http://casrv01.alex.local/certsrv

2. Select ‘Request a certificate‘ then select ‘advanced certificate request‘

3. Open the vmca_issued_csr.csr in a text editor like notepad++ and copy and paste the entire contents into the saved request text field and choose your VMCA Root certificate template from Certificate Template drop down and then hit Submit

Select base 64 encoded format and click on download certificate.

Now we need root certificate of the CA server. To obtain so, go to home page of your CA server and click on “Download a CA certificate, certificate chain or CRL“

Rename the downloaded file to vmca_root_signed.cer or any other name which you like.

Select Base 64 and click on Download CA certificate

Rename the downloaded .cer file to CARoot.cer

Now open a new file on Notepad++ and first copy the contents of vmca_root_signed.cer followed by contents of CARoot.cer and save the file as vmca-final.cer. 

Upload vmca-final.cer file on the PSC appliance via winscp

Return to certificate manager utility and click on Continue to importing Custom certificates and keys for VMCA Root Signing certificate

Provide the path to where your signed cert and key file is stored and Press Y to continue

Let the process complete. It will roughly take 7-10 minutes.

If you are using vCenter with embedded PSC, your job is done and you should now have a trusted certificate in the browser when accessing the vSphere Web client.

For vCenter with external PSC, follow the next steps.

Replace the Machine SSL Certificate on the vCenter Server from the new VMCA

1: SSH to your vCenter appliance and stop and start all services 

Post all services have been restarted, launch Certificate Manager on the vCenter appliance and choose the third option “Replace Machine SSL certificate with VMCA Certificate“

Provide SSO administrator credentials

Provide IP of PSC server for the infrastructure server IP and press Enter.

Provide required information to populate certool.cfg file

Note: For the Name and Hostname filed, use IP/FQDN of the vCenter server and not PSC appliance

Press Y to continue replacing the certs on vCenter appliance and wait for the process to complete.

Wait for web client to initialize and now when you login to Web Client you will see the vCenter connection as secure and certs displayed in green. 

Now if open the properties of this cert file, we can see its issued to vcenter server by CA server and cert is valid for 2 years. 

Also if we navigate to certificate path, we can see the list of CA’s who have signed this cert. In my case I have no intermediate CA between VMCA and my Enterprise root CA.

Replace the Solution User Certificates on the vCenter Server from the new VMCA

Replacing solution user certificate is completely optional as they are not directly exposed to the end user. 

Launch the Certificate-Manager /usr/lib/vmware-vmca/bin/certificate-manager and select Option 6 Replace Solution user certificates with VMCA certificates

Press Y for generating all certificates using configuration files. 

Provide the sso administrator credentials and supply PSC IP as valid infrastructure IP

Provide info to populate certool.cfg file.

Press Y to regenerate Solution User Certificates and wait for process to complete.

The machine, vsphere-webclient, vpxd and vpxd-extension solution user certificates will be replaced with new ones issued by the VMCA.

Final Thoughts

1: Issues with certs replacement in vSphere 6.0 U2

There is a known issue in 6.0 U2 where the CSR created for the VMCA as a Subordinate does not have the required attributes. Most Microsoft CA’s will force the inclusion of the attributes based on the certificate template used.

Check the certificate you just created from your CA by running the following OpenSSL Command:

openssl x509 -in vmca.cer -noout -text

The attributes we need present are CA:TRUE, Digital Signature, Certificate Sign, CRL Sign

If these attributes are missing in the cert, then you will need to create the CSR by following the steps listed in VMware 2145544

2: In vsphere 6.0 U1 or earlier, Add new Esxi host to vCenter 24 hours after cert replacement on vCenter server. 

If you are on vSphere 6.0 U1 or earlier, VMware documentation suggests that you replace your VMCA certificate first before adding your ESXi hosts to the vCenter Server Inventory. The reason for this is that the VMCA will replace vCenter certificates as well as ESXi certificates.

When adding a new ESXi host with a self-signed certificate to the vCenter Server Inventory, the VMCA will sign a new SSL certificate for the “new” ESXi host as part of the process. The problem is, that VMware have decided to backdate all new ESXi certificates by 24-hours to “avoid time-sync” issues. 

If you have generated a new CSR and then get it signed immediately by your root CA and install it straight away to the VMCA, the VMCA will use that certificate that was only signed a few minutes ago, to sign a new SSL certificate for ESXi, however, the ESXi certificate will be backdated by 24 hours, which means that it would seem to have been signed by a root CA at a date and time at which the signing VMCA certificate itself didn’t even exist.

Any attempt to add new Esxi host will error out with error “A general system error occurred: Unable to get signed certificate for host: esxi_hostname. Error: Start Time Error (70034)“. VMware KB-2123386 explains about this error.

Note: This behavior is changed in VMware vCenter 6.0 Update 2 and later with the advanced setting vpxd.certmgmt.certs.minutesBefore

Edit this key and change the value from 1440 to 10

The workaround with vSphere 6.0 U1 is to add your ESXi hosts to the new vCenter Server Inventory first and then replace the VMCA root certificate.

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable