Handling Outage with Double Mix - vCenter Server 5.5 and Webclient

Hi,

Hope you have read my earlier post on recovering vCenter Server from the unexpected Outage in the Datacenter and how to revive vCenter Server and gain access to the inventory properly.

Now lets go little deeper in to another situation and with vSphere 5.5 now a days a normal vSphere environment looks something like this.

vCenter Server 5.5 with latest updates installed on a Virtual Machine with Latest Hardware version 10

Inventory Server on a separate  virtual machine with Hardware Version 10

Running Nexus 1000v VDS / VMware VDS for all nature of Traffic including Management, VM, vMotion, iSCSI, Control, Packet and Management (for Nexus 1000v), NFS, FT etc. etc.

ESXi host Management Network is on VDS as well so keep this in mind.

Now lets assume an outage occurred and you were able to connect to the ESXi host and also you were able to trace down the vCenter Server. Oh man, that was pain. I know that feeling, and hence I posted earlier about the available ways on just how to recover vCenter Server from an outage. So lets continue.

Now if you verify the VM traffic is going through VDS and the vCenter Server VM has one of the dvPortGroups selected. As you restored a standard vSwitch on ESXi host though DCUI and now you have connectivity to the ESXi host using vSphere client.

vCenter Service is not starting due to the reason that the Inventory Service is running on some other virtual machine. Luckily you found that VM and registered the VM on the same host as the vCenter Server.

But still both VMs - vCenter Server and Inventory Server cant connect to each other as both using dvPortGroup of VDS.

 Now you have the vSwitch on the ESXi host which you can use temporarily but how to change the settings of the vNIC on the VM.

This is where the trouble starts

You can't edit the Virtual Machine settings as you are connected through vSphere client to the ESXi host

In order to change the vNIC Settings you need to connect using WebClient for Hardware Version 10 Virtual machine but as the vCenter Service is not starting as it depends on the Inventory Service which is running on another Virtual machine so you are now stuck in a loop. Its a chain reaction within which you cant make vCenter Server and Inventory Server talk to each other.

Now by design you can't edit the HW 10 virtual machine settings if not using Web Client.

So here is the trick on how to fix this.

First of all shut down both vCenter Server and Inventory Server virtual machine and remove them from Inventory. Right click the virtual machine and select "Remove from Inventory" option.

Lets assume you have a standard portgroup created on the Standard vSwitch called "Test".

Now you need to open SSH access to the ESXi host and connect through putty or you can use DCUI too.

Go to the virtual machine directory

cd /vmfs/volumes/datastore-name/vcenter-vm/

Now you can use the "vi" command to modify the .vmx file of the vCenter Server virtual machine and also the Inventory Server virtual machine

Go to the line where you see the hardware version - go in to edit mode by pressing a in vi.

Hardware = 10

Change the value from 10 to 8 - save the file :wq! and you will be back at the root prompt of ESXi host.

Now go back to vSphere client and browse the respective datastore one at a time and register both vCenter Server virtual machine and also the Inventory Server virtual machine in to the Inventory.

Now you can Edit the Settings using vSphere Client and select the standard portgroup "Test" for both VMs.

Power on both VMs and verify that the vCenter Service is running.

Once verified you can open WebClient and connect to vCenter Server or you can use the vSphere client as well to connect to the vCenter Server. Make sure you have all the hosts and cluster settings as they were before the outage.

Now you should be back in Business with minimal down time.

Hope you will find this information useful when hit by the limitation of Hardware Version 10. Hopefully this should be addressed in future products where you have the ability to change the settings of the virtual machine without any specific requirement. With ESXi 5.5 U2 you can now edit the setting of the VM with Hardware Version 10 using vSphere client. So update your ESXi host in order to get this benefit and also there are other issues resolved too with this release.

Please share and care !!

Enjoy !!

Recovering vCenter Server after the major Outage

Hi,

I have discussed this issue so many time with my colleagues and friends who experienced the situation where the whole Datacenter had an Outage due to X reason and then depending on the products running in the environment, it is getting difficult to restore everything back to normal.

The products which can add complexity in the restoration process can be Nexus 1000v, SRM, VMware VDS. Now let me emphasize here that there is nothing wrong with these products but to recover vCenter Server which is a Critical/Key component in the whole infrastructure.

I am going to explain few scenarios on how to recover vCenter Server and get everything up and running which may take time from minutes to few hours and sometimes a day or two (depends on the size and inventory of  the Datacenter).

1) If running VMware VDS, Nexus 1000v VDS

First of all one needs to find out where the vCenter Server was running lastly.

Now lets assume you are running fewer number of ESXi hosts in the cluster/Datacenter (lets say 1-20) then if you have the location documented if the vCenter was tied up on certain hosts only using the DRS rules then you can connect directly (assuming network connection is working or else we need to get that working first) to that particular ESXi host (using putty or DCUI and later using vSphere client) and find out if the vCenter Virtual Machine is still registered on that host or not. (If running WebClient then I will be covering that in an upcoming blog post).

vmware-cmd -l

The above command will tell us which VMs are still registered on the host and you can power on the vCenter Server through command line or from vSphere Client. Once logged in you can open the console of the vCenter and login using the local administrator account. Now here there is another assumption going here that if the vCenter Database is residing on another VM or a physical machine than there should be network connectivity available between the DB machine and vCenter Server.

Now lets take a situation where the management Network of ESXi host is configured on VDS (VMware) then if needed you need to restore the Standard Switch on the ESXi host so that atleast you can connect to the Management interface. Then you can try accessing vCenter Server from the console right on the vCenter server VM using localhost option and you can see all the inventory within the VC virtual machine. Once you power on all the ESXi hosts you should be able to see all the VMs and other inventory items.

If the VMs are showing inaccessible or orphaned then you can simply unregister the VMs and register them again. (This needs the information documented somewhere about the names of the virtual machines, network they are using / connected to, Datastores they are using or configured on).

If running Nexus 1000v then make sure both VSMs are powered on and can reach to vCenter Server and ESXi host/s. If needed you can register the VSMs on the same host as vCenter Server to make things easy. You can put them back on to the host where they belong as per the DRS rules (if any specified for Nexus
VSMs) set for them once everything is in working order.

2) Where you have large number of Hosts in the cluster (more than 20-50) so atleast you need the name of the Datastore where the vCenter Server files are residing. (Assuming no Storage vMotion occurred before the outage happened) so it will be a good practice to designate a specific datastore for vCenter Server virtual machine files so that it will be easy to just connect to that datastore using ssh from one of the ESXi host which is using as a shared datastore.

You can setup ssh to the host or use local ESXi shell option from DCUI and browse to the datastore and register the vCenter Server on that ESXi host.


vmware-cmd -s register .vmx

or

vim-cmd solo/reistervm /vmfs/volumes/datastore_name/VM_directory/VM_name.vmx

Once you see the vCenter entry in the inventory then just power it on. Make sure the vNIC is connected. Now if the VM was part of VDS then it must be connected to one of the dvPortGroup on the VDS which is not accessible as vCenter Service is not available yet. So you need to create a Standard vSwitch on the ESXi host where you registered the vCenter Server VM. Provide at least one uplink which can carry the same VALN traffic (if there was VLAN configured for the vCenter network) so get the connectivity. Now again you need to see if you have a spare NIC which you can use on VSS and if not (assuming all the NICs are used by VDS) then you just need to use one on VSS which was assigned to VDS before the outage.

3) Now assume you don't know the Datastore / ESXi host name where the vCenter Server was residing and running lastly and you have more than 50/100 hosts in the cluster. 

Here's comes the real part of this post so be patient and read on.

The question is how to find the vCenter Server virtual machine directory.

There are few methods you can go with

a) Run the PowerCLI script across all the ESXi hosts which definitely is a time consuming task as you need to connect to each host individually and run the command.

I can update the post here if someone comes with a PowerCLI one liner so please leave a comment.

b) If you are running SQL Database for vCenter Server database then you need to find out that VM first.
Login to the SQL VM (with local admin or using Domain Admin account (if DC/AD machine is available and accessible). Then login to the SQL Database using the Administrator Account. Run the following query against the VC Database.

First query only returns a host ID

select HOST_ID from VPX_VM where DNS_NAME like '%vcenter%'

You need to use the valued derived in the 2nd query. You just need to replace "vcenter' in the above query to the actual name of your vCenter Server VM name.

select * from VPX_ENTITY where ID='x'

The above will give you the result with the ESXi host where it was lastly registered and running.

Hopefully you can use the same query on Oracle Database as well but not sure so if someone is Oracle Expert then please leave a comment and I will modify the post with actual Query for Oracle database.

The above methods are having certain assumptions such as connectivity, login information to vCenter Server, SQL Server, ESXi host etc. etc. which are needed in the whole restore process.

4) Now the last situation where you dont know the name of the vCenter Server/ESXi Host/Datastore name then you just cross the fingers, pray to God and start digging for the VM on each and every direction possible and make a resolution first that you will DOCUMENT everything about your virtual Inventory going forward. Not joking here as seen instances like this too.

Let me know if you feel to add/update the existing information and I will be happy to do it. Just need your comment through any available medium.

Please share and care !!

Thanks for your time.

Simulating Management Network Redundancy

Hello Readers,

First of all let me thank you, for your time, you will be investing in reading this post. And as this is a long post, I appreciate your patience as the information documented here is very important and helps you Design/Configure/Troubleshoot/Maintain your VMware environment accordingly.


So many times during discussion I came across a situation where power outage happened in the environment and everything went down completely or partially. Now this really depends on the configuration and design you have opted on the Network side of things when you lost the connectivity to your Management Network (either to Service Console - ESX 3.x, ESX 4.x or to vmkernel post ESXi 5.x).

As this is a long post I appreciate your patience as the information documented here is very important and helps you design your environment accordingly.

The purpose of this article is to simulate a condition with which you can test the redundancy of your service console/vmkernel used for carrying the Management Network.

Assumptions made before we proceed:

1) Here I am making an assumption that the configuration on the ESX/ESXi host with atleast two uplinks for redundancy either in NIC Teaming or only have one uplink or more than one uplinks on one vSwitch/VDS or having a second vSwitch/dvportgroup on VDS configured wither either 1 or 2 uplinks.

2) Both NICs are either connected to one physical switch or if connected to two separate physical switches then both uplink switches are connected at L2 using any workable technologies such as vPC, MLAG, SMTL etc. etc.

3) vSwitch port group or VDS dvportgroup are configured with correct Load Balancing policies which corresponds to the configuration on the uplink physical switches

4) Environment is not running any Production workload and it could be Dev/Test/Lab so that no impact on the running Production VMs.

5) The default gateway is reachable from ESX/ESXi

6) If more than one isolation address is required on the Cluster/Host then necessary configuration is done on ESX/ESXi

7) Physical connectivity is configured properly for each ESX/ESXi host irrespective of hardware and other components necessary.

8) ESX/ESXi host is configured to use any of the available tools such as SSH (putty)/DCUI/ILO/DRAC/RSA/KVM etc.

9) All the network cables which are used for providing an uplink/s to ESX/ESXi host are identified properly and marked/documented properly on both ESX/ESXi and physical Switch/es Side.


10) Familiarity with vSphere Networking and use of the necessary commands where not listed.

11) Make changes at the vSwitch level only when you a separate vSwitch configured for Management Network with a different uplink or if using VDS you should have a different dvportgroup configured for Management Network.


1st Method

You can go into VSS port group settings and go to NIC teaming Tab where you can see the two uplinks in Active Active, check mark the box Override failover order, select one of the NICs from the Tea, select the arrow Move Down and click it twice so the NIC will be moved to Unused state from Active state

At this time the vSwitch/VDS will still shows both NICs as Active/Active state. Once you move the NIC into unused state you will see the Management Network will have only one uplink and you will received the warning under the Summary Tab "Host currently has no management network redundancy".

Now HA may also gives you an error but related information is not under the scope of this article and user should be aware about the impact and able to recover if needed.

Note: If the warning continues to appear, disable and re-enable VMware High Availability/FDM (depending on the version of ESX/ESXi in the Cluster.

To set das.ignoreRedundantNetWarning to true (for ESX 3.x and 4.x):

   1. From the VMware Infrastructure Client, right-click on the cluster and click Edit Settings.
   2. Select vSphere HA and click Advanced Options.
   3. In the Options column, enter das.ignoreRedundantNetWarning
   4. In the Value column, type true.

      Note: Steps 3 and 4 create a new option.
   5. Click OK.
   6. Right-click the host and click Reconfigure for vSphere HA. This reconfigures HA.

To set das.ignoreRedundantNetWarning to true in the vSphere 5.1 Web Client:

   1. From the vSphere Web Client, right click on the cluster.
   2. Click on the Manage tab for the cluster, then under Settings click vSphere HA.
   3. Click on the Edit button in the top right corner.
   4. Expand the Advanced Options section, and click Add.
   5. In the Options column, type das.ignoreRedundantNetWarning.
   6. In the Value column, type true.
   7. Click OK.
   8. Right-click the host and click Reconfigure for vSphere HA.

2nd Method

If you do not have access through vSphere client or Web Client then you can set the NIC to unused state using command line through ssh session to the ESX/ESXi hos. Now this command will move the NIC into unsed state for the whole vSwitch or VDS and will NOT make change at the particular port group (VSS) or dvport group (VDS) level.

If using vSwitch (VSS)

#esxcfg-vswitch -U vmnicX vSwitchX

(Note: Replace X with the actual NIC number and vSwitch Number)

For ESXi 5.x

Remove network card (known as vmnics) to or from a Standard vSwitch using these commands:

#esxcli network vswitch standard uplink remove --uplink-name=vmnic --vswitch-name=vSwitch 

Change the link status of one of the uplinks vmnic with one of these commands:

#esxcli network nic down -n vmnicX

If using VDS

To get the dvport ID for the dvportgroup

#esxcfg-vswitch -l | more

To unlink the uplink from VDS

#esxcfg-vswitch -Q vmnicX -V dvPort_ID_of_vmnic dvSwitch

3rd Method

For ESX/ESXi 3.x/4.x, ESXi 5.0 you can change the VLAN property on the VSS/VDS which carries the Management Network though SSH/DCUI

#esxcfg-vswitch -v -p “Service Console” vSwitch0

(Note: for ESXi 5.1 the rollback feature won't allow you to make such change due to which the ESXi host may get disconnected from vCenter server but you can still try the command as we have one uplink still available)

4th Method

You can change the load balancing policy using the command line as well for the vSwitch where the Management Network is configured using Service Console or vmkernel and we are using only one vSwitch with one uplink and have a 2nd vSwitch configured separately with 2nd Management Network (Service Console/VMkernel) with one uplink.

Here you have to use a different teaming policy than the physical switch which results in to loss of Network connectivity on the uplink/s used for Management Network VSS/dvportgroup.

Run the command on ESX/ESXi 3.x/4.x:

#vim-cmd /hostsvc/net/vswitch_setpolicy --nicteaming-policy='loadbalance_ip' vSwitch0

To change the load balancing policy on an ESXi 5.x host, run this command:

#esxcli network vswitch standard policy failover set -l iphash -v vSwitch0

* To change the load balancing policy to a route based on the originating virtual port ID, run this command:

#vim-cmd /hostsvc/net/vswitch_setpolicy --nicteaming-policy='loadbalance_srcid' vSwitch0

* To change the load balancing policy to a route based on the MAC hash, run this command:

#vim-cmd /hostsvc/net/vswitch_setpolicy --nicteaming-policy='loadbalance_srcmac' vSwitch0

Or

#Set load balancing policy. The options are:

    *      Port ID = loadbalance_srcid
    *      IP Hash = loadbalance_ip
    *      MAC = loadbalance_srcmac
    *      Failover Only = failover_explicit

Here is an example command:

#vimsh -n -e "hostsvc/net/vSwitch_setpolicy --nicteaming-policy loadbalance_iphash vSwitch#"

Refresh network settings with the command:
#vimsh -n -e "/internalsvc/refresh_network"

Restart the management service with the command:

#service mgmt-vmware restart

On ESXi 5.x,

* To change the load balancing policy to a route based on the originating virtual port ID, run this command:

#esxcli network vswitch standard policy failover set -l portid -v vSwitch0

* To change the load balancing policy to a route based on the MAC hash, run this command:

#esxcli network vswitch standard policy failover set -l mac -v vSwitchX

* To change the load balancing policy to a route based on the IP hash, run this command:

#esxcli network vswitch standard policy failover set -l iphash -v vSwitchX

Restart the management service with the command:

#/etc/init.d/hostd restart

5th Method

Using Cisco CDP protocol find out the uplink switch port (assuming the uplink switch/es are Cisco branded), access the uplink switch using SSH and go to the port configuration for one of the NIC which is used as an uplink for Management Network, and just shut the port down administratively.

You can click on the small Blue icon right beside the vmnic to find out the CDP information as shown in the attachment.





You will see red X against the NIC which is disconnected in vSphere client which means the link is down.

Using the command "esxcfg-nics -l" will show you the Link is down for that particular NIC as well.

6th Method

By changing the MTU for the portgorup/dvportgorup used for Service Console/vmkernel Management Network

For ESX/ESXi 3.x/4.x if the MTU value selected is Default of 1500 then use the command and use the commands if the configuration is vice versa.

Find out the vmkernel or Service Console first using the command/s.

#esxcfg-vswif -l

#esxcfg-vmknic -l

Find out the port group/dvportgroup name from the out put of the above command/s.

# esxcfg-vmknic -m 9000 portgroup_name

Run this command for ESXi 5.x:

# esxcli network ip interface set -m 9000 -i vmk_interface

7th Method (Need more testing)

Once you connect to the ESX/ESXi host using SSH (putty) or any other tools and use the following command which will blink the NIC for 5-6 seconds.

#ethtool -p  vmnicX

8th Method

You can actually pull the cable from the back of the physical ESX/ESXi host which is carrying the Management Traffic so you need to have the information about the port mapping and related cable used for port/s used for Management Network on VSS/VDS.

9th Method

Warning: Using vsish interface which is currently NOT SUPPORTED by VMware so be careful when using it.

To block a particular link for a certain NIC of the vmkernel, run this command:

vsish -e set /net/pNics/vmnicX/block 1

All the above methods basically isolating the uplink connectivity for one NIC which in turn should break the redundancy status for the Management Network and the host should stay connected inside Virtual Center Server/ vCenter Server.

Enjoy the information and test out your Management Network properly and DO NOT use Production Host/s to test out the above methods.

Share and Care !!

Cheers !!

How to verify VMFS Heartbeat Region Corruption

Recently I was discussing the issue when you see some messages in the logs where the HB offset is noticed in place of the actual location of the bytes.

Upon research found out that the there is a heartbeat region corruption occurred on the VMFS volume due to the power outage or some other reasons.

You will see the following messages in the logs:


cpu2:2816)WARNING: HBX: 533: Volume 4f7217c1-3589352d-625d-001b22248a7a ("1TB-LAB") may be damaged on disk. Corrupt heartbeat detected at offset 3751936: [HB state 0 offset 0 gen 0 stampUS 0 uuid 00000000-00000000-00

cpu5:2980)WARNING: HBX: 533: Volume 4f7217c1-3589352d-625d-001b22248a7a ("1TB-LAB") may be damaged on disk. Corrupt heartbeat detected at offset 3751936: [HB state 0 offset 0 gen 0 stampUS 0 uuid 00000000-00000000-00


Now on your ESXi 5.x host run the following command on ESXi console using SSH or DCUI:

hexdump -s 22626304 -n 2048 -C /vmfs/devices/disks/

The output will be similar to this:

hexdump -s 22626304 -n 2048 -C /vmfs/devices/disks/

01594000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01594200  01 ef cd ab 00 42 39 00  00 00 00 00 de 00 00 00  |.....B9.........|
01594210  00 00 00 00 cd 31 82 11  00 00 00 00 00 00 00 00  |.....1..........|
01594220  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
01594230  0e 00 00 00 36 00 00 00  00 00 00 00 00 00 00 00  |....6...........|
01594240  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01594400  01 ef cd ab 00 44 39 00  00 00 00 00 00 00 00 00  |.....D9.........|
01594410  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01594600  01 ef cd ab 00 46 39 00  00 00 00 00 00 00 00 00  |.....F9.........|
01594610  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01594800

Now if you look at the bold section in RED above you see that the 1st section is missing some value out of the 4.

After fixing the heartbeat region you can see the value as follows once you run the same command again.


hexdump -s 22626304 -n 2048 -C /vmfs/devices/disks/

01594000  01 ef cd ab 00 40 39 00  00 00 00 00 00 00 00 00  |.....@9.........|
01594010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01594200  01 ef cd ab 00 42 39 00  00 00 00 00 de 00 00 00  |.....B9.........|
01594210  00 00 00 00 cd 31 82 11  00 00 00 00 00 00 00 00  |.....1..........|
01594220  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
01594230  0e 00 00 00 36 00 00 00  00 00 00 00 00 00 00 00  |....6...........|
01594240  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01594400  01 ef cd ab 00 44 39 00  00 00 00 00 00 00 00 00  |.....D9.........|
01594410  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01594600  01 ef cd ab 00 46 39 00  00 00 00 00 00 00 00 00  |.....F9.........|
01594610  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01594800

As you can see in the above output with all Bold in RED, all 4 sections have some HEX values which is a sign of healthy VMFS heartbeat region. If you do have issue viewing the file for any VM before and it was giving an error, try again you should be able to view the file from the affected datastore.
Now to fix the same issue I would suggest to contact VMware Support if not dealing with Lab/Test environment or you are not sure how to fix the corruption. This information is just to give you an idea how to verify if there is indeed a corruption on the VMFS volume or not.

For more information refer to KB 1012036 and also refer to the blog by @VMwareStorage on VOMA which talks about the metadata corruption.


Hope you find this information useful, if yes then do share.