I had a working HCX setup in my lab, and I was making some modifications to my setup and tried chopping off my interconnect networking configuration on the HCX Cloud side. Deletion of the interconnect configuration was failing for me with the below error

Let me first explain how I landed in this situation. 

I deleted the interconnect appliances from my on-prem to show a demo to my peers on how the interconnects are deployed via the HCX plugin in the vSphere webclient. During the demo, I did not notice that the site pairing between my on-prem HCX and cloud side HCX was broken (due to a vCenter upgrade on the cloud side, a cert mismatch issue occurred).

When I kicked the CGW appliance removal, the backing VM got deleted, and the appliance configuration disappeared from on-prem. But when I checked on the cloud side, the peer CGW appliance and the Mobility Agent host were still intact.

That is when I discovered that communication between the two sites was broken. Since cloud-side CGW appliances are automatically deleted when the deletion is called from on-prem, I couldn’t find any option in the HCX Cloud UI to delete the same.

Frustrated by this, I deleted the mobility agent host and CGW manually from the cloud side vCenter. However, the interconnect configuration was still intact in the HCX Cloud UI. Since this is a lab setup, I could not open SR with VMware, and hence I decided to do some R&D on this, and finally, I was able to fix the issue.

Disclaimer: The below resolution is not supported by VMware, and it is an outcome of R&D in the lab setup. Don’t perform these steps in your production without engaging VMware support.

HCX configuration data is saved in MongoDB, and since UI was not helping me, I jumped into DB and examined a few tables. 

1: Connect to HCX DB: Connect to HCX appliance over putty and run command: mongo hybridity

2: List tables: To list the tables in the database, run the command: show collections

This command gives you a long list of tables. The important tables related to interconnect configuration are:

• FleetAppliance
• FleetConfigIP
• FleetConfig
• VirtualNetwork

I found details of the offending IP in the above tables. To examine the contents of a table in Mongo, you need to run the command: db.<table-name>.find().pretty()

Examples are shown below

Once I verified the details in the above table, I dropped them using the below commands

> db.FleetConfig.drop ()

> db.FleetConfigIP.drop ()

> db.FleetAppliance.drop()

> db.VirtualNetwork.drop()

After dropping these tables, I refreshed the HCX Cloud UI, and the stubborn interconnect configuration disappeared from there.

I hope you find this post informative. Feel free to share it on social media if it is worth sharing.