Today while working in production, I came across an issue where the edge VM’s backing the edge gateway were not present in vCenter (no idea how they got deleted).
Due to this I was not able to delete the Org network from vCD. Any attempt to delete the Org network was failing with error
1 2 3 4 5 6 7 |
[ f3fcd1fd-cf1c-4a57-a920-504204dceba7 ] Cannot delete organization VDC network DMZ (0dff6592-df7c-4e02-a106-5f0ed722d601) Cannot update edge gateway "urn:uuid:f41b9f5a-0389-4781-ae0d-4b9cd19e9756" java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive. Cannot update edge gateway "urn:uuid:f41b9f5a-0389-4781-ae0d-4b9cd19e9756" java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive. |
This is what I was seeing in vCloud Director
On further investigation I found that the virtual wires associated with the problematic network were present in vCenter but on selecting the virtual machine tab for the virtualwire (distributed portgroup), I was not seeing any VM connected. Typically an edge VM should appear in the list.
That is when I found that edge VM’s were gone from vCenter.
Tried to re-deploy edge gateway and it also failed. Got same error about edge VM’s not being present
1 2 3 4 5 |
[ 8580da07-1ce4-440f-b126-0e5c8cca6e3e ] Cannot redeploy edge gateway XYZ (urn:uuid:f41b9f5a-0389-4781-ae0d-4b9cd19e9756) com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive. - com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive. - VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive. |
In this situation there is nothing much you can do. You need to hack your VCD DB and remove the stale entries from the database which would cause those entries to disappear from vCD UI.
First of all delete all virtual wires corresponding to the stale Org network from vCenter.
Secondly you need to analyze the database to look into the tables where info about those Org networks is stored.
Update: 13-07-2017: Before performing below steps, please check if the issue is similar to this. DB hack should be last weapon for any administrator dealing with this issue
Typically you have to use following queries:
1: Find Org vDC ID where your stale networks exists
select id ,vdc_id ,name FROM vdc_logical_resource where lr_type= ‘NETWORK’ AND name = ‘Org_nw_name’;
Output (Assuming your org network name is Test-123)
1 2 3 |
id vdc_id name 0xAAAAAA 0xBBBBBB5D Test-123 |
2: List all Org Networks in your Org VDC (In case you have more than one stale network)
Select id, name from vdc_logical_resource where vdc_id = 0xBBBBB AND lr_type = ‘NETWORK’;
In this example I am assuming that we have 3 stale Org network namely Test-123,Test-456 and Test-789
1 2 3 4 5 6 7 |
id name 0xAAAAAAA Test-123 0xNNNNNNN Test-456 0xTTTTTTT Test-789 |
Note: In case you have several org networks associated with your edge gateway but there is only one problematic (stale) org network, then you can modify the above query as shown below.
Select id, name from vdc_logical_resource where vdc_id = 0xBBBBBB5D lr_type = ‘NETWORK’ AND name = ‘Org_nw_name’;
This query will specifically return record of the stale network and thus will prevent accidental deletion of any healthy org network
3: Find Logical network id and gateway id
select display_name, id, gateway_id, logical_network_id from gateway_interface where display_name = ‘Test-123’;
1 2 3 |
display_name id gateway_id logical_network_id Test-123 0xF41B9F 0x8EEB1C 0x9FCD1Y |
4: Query for link_lnet_id
select id,link_lnet_id FROM logical_network WHERE name = ‘ABC’
1 2 3 |
id link_lnet_id 0x8EEB1C757916483893A633FF3E459ECE NULL |
Once you have all the information in place, use following delete queries to remove the records of the stale networks from vCD UI
delete FROM vdc_logical_resource WHERE id = 0xAAAAAAA;
delete FROM gateway_assigned_ip WHERE gateway_interface_id = 0xF41B9F;
delete FROM gateway_interface WHERE logical_network_id = 0x9FCD1Y;
delete FROM logical_network WHERE name = ‘Test-123’;
I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable