Retry Failed Bringup with Modified Input Spec in VCF

Those who are experienced with VCF might have seen this situation in past where a SDDC Bringup task fails because of an incorrect parameter in the json/xls file used for bringup.

Cloud Builder UI don’t provides a mechanism to go back and change the input file with correct values and retry bringup with modified file. Don’t loose heart if you are caught in this situation as it is possible to do this via API and in this post I will walk through steps of doing so.

By default cloud builder places the sddc-bringup json in /opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-ems/ directory by name vcf-ems.json.

Note: Even if you are using xls to feed cloud builder, it is internally converted in json file.

You can either modify this json directly and fix the incorrect value that was supplied in your xls/json file or you can generate a new json for bringup. 

Step 1: Login to cloud builder vm via ssh and switch to root user.Read More

Troubleshooting vRSLCM Deployment Failure in VCF

Last week while working in my VCF lab, I faced an issue with vRSLCM deployment. Deployment failed in step where SDDC Manager tries to configure vRSLCM NTP settings.

Started my troubleshooting with checking domainmanager.log on sddc manager appliance. I saw following log entries:

Read More

Dealing With MyVMware Credentials shenanigan in VCF

Recently while working in my lab, I deployed a proxy server and configured my SDDC Manager to use that proxy server to talk to internet.

SDDC Manager was happily talking to internet, but when I tried configuring MYVMware credentials to download product binaries, operation was failing with error “Updating depot user credentials failed”

Tried configuring credential via API (as UI sometime acts weird), but operation failed again. 

curl http://localhost/lcm/depot/user -H ‘Content-Type: application/json’ -X PUT -d ‘{“userId”:”xxx@vmware.com”, “password”:”<PASSWORD>”}’

and got error 500 in response

On further investigation I found that the issue was with proxy server itself as it was not trusting certificate returned by depot.vmware.com  and thus causing lcm-bundle-transfer to fail.

Google search returned nothing for this issue and I had to reach out to VCF engineering team.Read More

Deploy Edge Cluster With Custom Form Factor in VCF via API

Recently while working in my lab, I was facing an issue with edge cluster deployment in workload domain. Edge Cluster deployment was failing with below error:

Since my lab is a nested lab, I bumped cpu on Esxi hosts to 10 and retriggered the task, but deployment was still failing with same error.Read More

How to Reuse VCF Cloud Builder For Another Bringup

Those who are working on VCF are aware of the fact that, after a successful bringup, the only option you get is to launch SDDC Manager. There is no option in UI to go back to cloud builder initial screen where you can upload configuration workbook xls/json to start a fresh bringup.

sddc-bringup

One work around to this problem is to take snapshot of cloud builder before bringup and revert snapshot after succesful bringup to start again.

But this task seems very tedious to me. If you are running a huge infrastructure and performs bringup almost every day, this will not be an ideal situation.

In this post I will explain how to get rid of this annoying problem.

This can be achieved by running an internal one liner API call on cloud builder vm as root user.

Just refresh the cloud builder ui after running above API call and job done. Read More

How To Verify if SDDC Bringup is AVN Enabled in VCF?

Recently I saw an interesting thread on slack where one of the consultant was seeing deploy button grayed out for VRSLCM deployment. Consultant was not aware if the management domain bringup was AVN enabled or not. 

AVN was first introduced in VCF 3.9 and it enabled deploying vRealize components on logical segments instead of VLAN backed portgroup. AVN brought many enhancements with Cloud Foundation. Please read this Article to know more about AVN.

Since AVN enablement is optional and you can chose not to use it during SDDC bringup This is controlled via a parameter in VCF configuration workbook and ca be enabled later also.

vcf-avn

Now the question is how to verify whether or not a SDDC bringup is AVN enabled bringup. There are couple of ways to verify this.

1: Through PowerVCF

PowerVCF is a PowerShell module to interact with SDDC Manager and the VCF stack. If you have PowerVCF is installed, you can connect to sddc-manager using below command:

# Connect-VCFManager <sddc-manager-fqdn> <sso-credentials>

and then run command: Get-VCFApplicationVirtualNetwork

Read More

Deploying vSphere with Kubernetes via VCF 4.0

In this post I will walk through how to deploy a Kubernetes cluster in a workload domain in VCF. This is  a new feature that is introduced in VCF 4.0. vSphere with Kubernetes is also known as Project Pacific and Cormac Hogan did a great job in explaining nuances of Kubernetes in his Article

Before deploying a Kubernetes cluster, there are few prerequisites that must be met:

1: A NSX-T backed workload domain deployed.

2: Dedicated Edge Cluster deployed for workload domain. I have covered steps of deploying edge cluster Here

3: All Esxi hosts that are part of workload domain are licensed with “VMware vSphere 7 Enterprise Plus with Add-on for Kubernetes” license.

4: Subnets for Kubernetes cluster egress/ingress traffic created on your ToR.

Once above prerequisites, we are good to go with deployment. Let’s jump into lab and walk through deployment steps.

To deploy Kubernetes cluster, login to SDDC Manager and navigate to Home > Solutions.Read More

How To Safely Remove Edge Cluster from Workload Domain in VCF

Recently I deployed an edge cluster in my lab for NSX-T based workload domain. In VCF 4.0, a new feature was added for automated deployment of edge cluster. If you haven’t tried this yet, you can follow my previous Post to learn more about this.

My edge cluster encountered some routing issues because of incorrect BGP information passed during edge cluster deployment workflow and I was looking for how to remove it cleanly from sddc manager. 

While searching on internet, I came across VMware KB-78635 which lists the instructions of doing so. In this post I will walk through steps from the above KB.

Edge Cluster removal workflow is not baked into sddc manager yet and is carried out by python scripts provided by VMware. May be in next release of VCF, we will see this option added to sddc manager 😉

Edge cluster python script is named “edge_cluster_cleaner” and you will find this script as tar file in attachment section of KB-78635.Read More

Automated NSX-T Edge Cluster Deployment in VCF 4.0

NSX-T integration with VCF is there for quite some time. Till VCF 3.9, SDDC Manager was not deploying NSX-T edge cluster in workload domains as part of VI WLD creation. Process of edge cluster deployment was manual and very well documented Here

Also this article by Cormac Hogan is worth checking to understand things in greater details.

In VCF 4.0, edge cluster deployment task is automated and can be kicked directly from SDDC Manager itself. No more juggling around configuring various things in NSX-T.

Below are high level steps for Automated Edge Cluster Deployment in VCF 4.0.

Step 1: Login to SDDC Manager and navigate to Workload Domains tab and click on environment where you want to deploy NSX-T Edge Cluster. 

Form Actions Menu, click on Add Edge Cluster 

WLD-EC02

Step 2: A list of prerequisites will pop up. Make sure your infrastructure has met all these requirements before moving ahead.

Word of caution here, make sure VLAN used for Host and Edge TEP’s are different.Read More

Change NSX-T Deployment Form Factor in VCF

Recently while playing with VCF, I noticed one annoying thing about NSX-T deployment for workload domain. By default NSX-T managers are deployed in large form factor with 48 GB of memory and all 48 GB reserved.

Also SDDC Manager checks for 144 GB of free memory at custer level where NSX-T managers will be deployed.

For production environments this setup works fine as we have plenty of resources, but for Lab/POC purpose 48 GB per node seems too big in my opinion (I am open to discussion on this)

In this post we will learn how to deploy NSX-T for workload domains with custom size. 

NSX-T deployment form factor can be controlled via /opt/vmware/vcf/domainmanager/config/application-prod.properties file. To use custom form factor, login to sddc manager as root user and edit this file to add below properties:

Read More