Lessons from the Field: Recovering from Orphaned Avi Controllers in VCF-9

Manish Jha

In a VCF environment, lifecycle operations are expected to be performed through SDDC Manager/VCF Operations. However, in real-world scenarios, operational mistakes occur—especially when components are modified directly in vCenter Server rather than through the VCF control plane.

In this blog, I will walk through a failure scenario involving NSX Advanced Load Balancer (Avi) in a VCF 9 environment, where Avi controller VMs were accidentally deleted from vCenter, leaving behind stale inventory references in SDDC Manager. I will cover the issue, impact, recovery approach, and key lessons.

The Problem Scenario

In a VCF-9 environment, Avi was deployed via SDDC Manager (integrated with NSX) and was running fine. During a troubleshooting session, Avi controller nodes were deleted directly from vCenter, leaving stale entries in the SDDC Manager UI and thus preventing the redeployment of Avi.

Result: SDDC Manager still believed Avi was deployed, and NSX integration references remained. The deletion option was grayed out in the SDDC manager UI.

This created a split-brain condition between the actual infrastructure state (Avi missing) and the VCF inventory state (Avi still present).

Root Cause

VCF maintains strict lifecycle state tracking via SDDC Manager. When components are altered outside their control:

  • Inventory drift occurs.
  • Internal metadata (database + APIs) becomes inconsistent.
  • Dependent services (NSX, VCF Automation) retain stale references.

In this case, deleting Avi directly from vCenter bypassed SDDC Manager’s lifecycle workflows.

Resolution: Clean Up Stale Entries in SDDC Manager

Removing a stale Avi instance from the SDDC Manager inventory may involve the following:

  • API calls
  • Database edits.
  • Guided VMware KB procedures (custom scripts)

The end goal is to ensure the SDDC Manager no longer believes Avi is deployed.

1: API Calls: The API call to remove Avi from SDDC Manager was failing as the API was invoking a workflow, and the workflow failed because it couldn’t reach Avi controllers.

However, the API to remove Avi references from NSX Manager was successful. The following API calls were executed:

2: Database Edit: Since the SDDC manager API call failed, the next step was to investigate stale entries in the database and remove them.

Disclaimer: Don’t attempt the following steps in a prod environment and without Broadcom support, as modifying the database can be catastrophic and can lead to inconsistent states and outages.

2.1 Connect to the SDDC Manager database

2.2: Search Avi controller entries

2.3: Clean Avi controller entries

The commands below remove the Avi instance details as well as Avi ‘admin’ credentials and certificates.

After that, I refreshed the SDDC Manager UI, and “Deploy Avi Load Balancer” was available. And that’s when I thought the job was done. But there was more to it than meets the eye.

The redeploy attempt failed at step “Create NSX Service users for Avi Load Balancer”

The error was pretty evident in the domainmanager.log

Once again, I had to modify the database to remove the service account entry.

Reattempted the deployment, and again it failed with the same error. On further investigation, I found that the problem is not on the SDDC manager side, but on the NSX side. The SDDC manager was trying to create the service account ‘svc-vcf9-avi-vcf9-nsx’, but it already exists on the NSX manager.

First, I listed the local user via API, but it failed, complaining that the user doesn’t exist.

Next, I listed the role-bindings and found the service account instance.

But when I tried deleting the role-binding, it errored out.

The error “Role binding for local user can not be deleted” (error code 71004) is a deliberate NSX restriction. Here’s what’s happening and how to work around it:

Why does the DELETE fail?

NSX does not allow you to delete a role binding directly for local users via the /api/v1/aaa/role-bindings/{id} endpoint. Local user accounts in NSX have restrictions — you cannot simply add or delete role bindings for them through the standard role-binding API. The role binding for a local user is managed through the user account itself, not through the role-bindings endpoint.

The /api/v1/aaa/role-bindings DELETE endpoint is intended for external/LDAP/VIDM users, not local users. For local users, manage roles via the user account APIs or the UI.

Since the user was not available in the GUI/API output, I decided to clean the entries from the NSX corfu database.

Interacting with NSX Corfu Database

NSX 9.0 uses /opt/vmware/bin/corfu_tool_runner.py — a Python wrapper — instead of the old Java JAR commands to interact with the corfu database.

List the RoleBinding table

Delete Avi Service Account

3: Delete the underlying local user account

4: Verify the role binding is gone

After cleaning up NSX, I restarted the Avi deployment task, and it completed without any further issues.

Conclusion

In a production environment, it’s not recommended to modify product databases directly. The best is to open a Broadcom support case. Broadcom might have custom scripts/commands to deal with stale entries cleanup.

That’s it for this post. I hope you enjoyed reading it. Feel free to share it on social media if it’s worth sharing.