Handling Outage with Double Mix - vCenter Server 5.5 and Webclient

Hi,

Hope you have read my earlier post on recovering vCenter Server from the unexpected Outage in the Datacenter and how to revive vCenter Server and gain access to the inventory properly.

Now lets go little deeper in to another situation and with vSphere 5.5 now a days a normal vSphere environment looks something like this.

vCenter Server 5.5 with latest updates installed on a Virtual Machine with Latest Hardware version 10

Inventory Server on a separate  virtual machine with Hardware Version 10

Running Nexus 1000v VDS / VMware VDS for all nature of Traffic including Management, VM, vMotion, iSCSI, Control, Packet and Management (for Nexus 1000v), NFS, FT etc. etc.

ESXi host Management Network is on VDS as well so keep this in mind.

Now lets assume an outage occurred and you were able to connect to the ESXi host and also you were able to trace down the vCenter Server. Oh man, that was pain. I know that feeling, and hence I posted earlier about the available ways on just how to recover vCenter Server from an outage. So lets continue.

Now if you verify the VM traffic is going through VDS and the vCenter Server VM has one of the dvPortGroups selected. As you restored a standard vSwitch on ESXi host though DCUI and now you have connectivity to the ESXi host using vSphere client.

vCenter Service is not starting due to the reason that the Inventory Service is running on some other virtual machine. Luckily you found that VM and registered the VM on the same host as the vCenter Server.

But still both VMs - vCenter Server and Inventory Server cant connect to each other as both using dvPortGroup of VDS.

 Now you have the vSwitch on the ESXi host which you can use temporarily but how to change the settings of the vNIC on the VM.

This is where the trouble starts

You can't edit the Virtual Machine settings as you are connected through vSphere client to the ESXi host

In order to change the vNIC Settings you need to connect using WebClient for Hardware Version 10 Virtual machine but as the vCenter Service is not starting as it depends on the Inventory Service which is running on another Virtual machine so you are now stuck in a loop. Its a chain reaction within which you cant make vCenter Server and Inventory Server talk to each other.

Now by design you can't edit the HW 10 virtual machine settings if not using Web Client.

So here is the trick on how to fix this.

First of all shut down both vCenter Server and Inventory Server virtual machine and remove them from Inventory. Right click the virtual machine and select "Remove from Inventory" option.

Lets assume you have a standard portgroup created on the Standard vSwitch called "Test".

Now you need to open SSH access to the ESXi host and connect through putty or you can use DCUI too.

Go to the virtual machine directory

cd /vmfs/volumes/datastore-name/vcenter-vm/

Now you can use the "vi" command to modify the .vmx file of the vCenter Server virtual machine and also the Inventory Server virtual machine

Go to the line where you see the hardware version - go in to edit mode by pressing a in vi.

Hardware = 10

Change the value from 10 to 8 - save the file :wq! and you will be back at the root prompt of ESXi host.

Now go back to vSphere client and browse the respective datastore one at a time and register both vCenter Server virtual machine and also the Inventory Server virtual machine in to the Inventory.

Now you can Edit the Settings using vSphere Client and select the standard portgroup "Test" for both VMs.

Power on both VMs and verify that the vCenter Service is running.

Once verified you can open WebClient and connect to vCenter Server or you can use the vSphere client as well to connect to the vCenter Server. Make sure you have all the hosts and cluster settings as they were before the outage.

Now you should be back in Business with minimal down time.

Hope you will find this information useful when hit by the limitation of Hardware Version 10. Hopefully this should be addressed in future products where you have the ability to change the settings of the virtual machine without any specific requirement. With ESXi 5.5 U2 you can now edit the setting of the VM with Hardware Version 10 using vSphere client. So update your ESXi host in order to get this benefit and also there are other issues resolved too with this release.

Please share and care !!

Enjoy !!

Recovering vCenter Server after the major Outage

Hi,

I have discussed this issue so many time with my colleagues and friends who experienced the situation where the whole Datacenter had an Outage due to X reason and then depending on the products running in the environment, it is getting difficult to restore everything back to normal.

The products which can add complexity in the restoration process can be Nexus 1000v, SRM, VMware VDS. Now let me emphasize here that there is nothing wrong with these products but to recover vCenter Server which is a Critical/Key component in the whole infrastructure.

I am going to explain few scenarios on how to recover vCenter Server and get everything up and running which may take time from minutes to few hours and sometimes a day or two (depends on the size and inventory of  the Datacenter).

1) If running VMware VDS, Nexus 1000v VDS

First of all one needs to find out where the vCenter Server was running lastly.

Now lets assume you are running fewer number of ESXi hosts in the cluster/Datacenter (lets say 1-20) then if you have the location documented if the vCenter was tied up on certain hosts only using the DRS rules then you can connect directly (assuming network connection is working or else we need to get that working first) to that particular ESXi host (using putty or DCUI and later using vSphere client) and find out if the vCenter Virtual Machine is still registered on that host or not. (If running WebClient then I will be covering that in an upcoming blog post).

vmware-cmd -l

The above command will tell us which VMs are still registered on the host and you can power on the vCenter Server through command line or from vSphere Client. Once logged in you can open the console of the vCenter and login using the local administrator account. Now here there is another assumption going here that if the vCenter Database is residing on another VM or a physical machine than there should be network connectivity available between the DB machine and vCenter Server.

Now lets take a situation where the management Network of ESXi host is configured on VDS (VMware) then if needed you need to restore the Standard Switch on the ESXi host so that atleast you can connect to the Management interface. Then you can try accessing vCenter Server from the console right on the vCenter server VM using localhost option and you can see all the inventory within the VC virtual machine. Once you power on all the ESXi hosts you should be able to see all the VMs and other inventory items.

If the VMs are showing inaccessible or orphaned then you can simply unregister the VMs and register them again. (This needs the information documented somewhere about the names of the virtual machines, network they are using / connected to, Datastores they are using or configured on).

If running Nexus 1000v then make sure both VSMs are powered on and can reach to vCenter Server and ESXi host/s. If needed you can register the VSMs on the same host as vCenter Server to make things easy. You can put them back on to the host where they belong as per the DRS rules (if any specified for Nexus
VSMs) set for them once everything is in working order.

2) Where you have large number of Hosts in the cluster (more than 20-50) so atleast you need the name of the Datastore where the vCenter Server files are residing. (Assuming no Storage vMotion occurred before the outage happened) so it will be a good practice to designate a specific datastore for vCenter Server virtual machine files so that it will be easy to just connect to that datastore using ssh from one of the ESXi host which is using as a shared datastore.

You can setup ssh to the host or use local ESXi shell option from DCUI and browse to the datastore and register the vCenter Server on that ESXi host.


vmware-cmd -s register .vmx

or

vim-cmd solo/reistervm /vmfs/volumes/datastore_name/VM_directory/VM_name.vmx

Once you see the vCenter entry in the inventory then just power it on. Make sure the vNIC is connected. Now if the VM was part of VDS then it must be connected to one of the dvPortGroup on the VDS which is not accessible as vCenter Service is not available yet. So you need to create a Standard vSwitch on the ESXi host where you registered the vCenter Server VM. Provide at least one uplink which can carry the same VALN traffic (if there was VLAN configured for the vCenter network) so get the connectivity. Now again you need to see if you have a spare NIC which you can use on VSS and if not (assuming all the NICs are used by VDS) then you just need to use one on VSS which was assigned to VDS before the outage.

3) Now assume you don't know the Datastore / ESXi host name where the vCenter Server was residing and running lastly and you have more than 50/100 hosts in the cluster. 

Here's comes the real part of this post so be patient and read on.

The question is how to find the vCenter Server virtual machine directory.

There are few methods you can go with

a) Run the PowerCLI script across all the ESXi hosts which definitely is a time consuming task as you need to connect to each host individually and run the command.

I can update the post here if someone comes with a PowerCLI one liner so please leave a comment.

b) If you are running SQL Database for vCenter Server database then you need to find out that VM first.
Login to the SQL VM (with local admin or using Domain Admin account (if DC/AD machine is available and accessible). Then login to the SQL Database using the Administrator Account. Run the following query against the VC Database.

First query only returns a host ID

select HOST_ID from VPX_VM where DNS_NAME like '%vcenter%'

You need to use the valued derived in the 2nd query. You just need to replace "vcenter' in the above query to the actual name of your vCenter Server VM name.

select * from VPX_ENTITY where ID='x'

The above will give you the result with the ESXi host where it was lastly registered and running.

Hopefully you can use the same query on Oracle Database as well but not sure so if someone is Oracle Expert then please leave a comment and I will modify the post with actual Query for Oracle database.

The above methods are having certain assumptions such as connectivity, login information to vCenter Server, SQL Server, ESXi host etc. etc. which are needed in the whole restore process.

4) Now the last situation where you dont know the name of the vCenter Server/ESXi Host/Datastore name then you just cross the fingers, pray to God and start digging for the VM on each and every direction possible and make a resolution first that you will DOCUMENT everything about your virtual Inventory going forward. Not joking here as seen instances like this too.

Let me know if you feel to add/update the existing information and I will be happy to do it. Just need your comment through any available medium.

Please share and care !!

Thanks for your time.

Download vSphere 5.5 with other Suites

vSphere 5.5. downloads are available from today so after the long wait since the Announcement was made at VMworld


 ESXi / vCenter Server

• ESXi 5.5
• vCenter Server 5.5
  
• vSphere App HA 1.0
  
• vCenter Certificate Automation Tool 5.5

Networking 

• VMware vCloud Networking and Security 5.5.0

• Cisco Nexus 1000v for 5.5
  Note: Please refer the compatibility guide on Cisco's for your vCenter if running SRM and other on Nexus 1000v
     

vCOPS

• VMware vCenter Operations Manager Standard 5.7.2

SRM, VDP & VR

• VMware vCenter Operations Manager Standard 5.7.2
  
• vSphere Data Protection 5.5.1
  
• vSphere Replication 5.5

CLI

• vSphere CLI 5.5
  
• vSphere PowerCLI 5.5
  
• vSphere Management Assistant 5.5
  

vCO

• vCenter Orchestrator Appliance 5.5
  

 vCloud

• VMware vCloud Automation Center 5.2.0
  
• VMware vCloud Director 5.5.0

Please choke the Internet by downloading the bits :-) and Happy upgrading your production systems and also your favorite Home Labs.

Share and Care !!

Controlling Upgrade Chaos of VMware Environment

Lot of times depending on the VMware components you are using its always better to check which component to upgrade 1st whenever there is a newer version available.

Now this is a big question for any Organization as its affects the whole chain of Maintenance, downtime, change management requirements etc. etc.

If the downtime is involved in cost and other losses then such upgrade process must be thought out carefully before proceeding further.

Depending on the products in use sometimes people have the ability/luxury of having a standby/Dev/Test/Lab environment where they can do such upgrade first and verify if its affecting anything in such environment and if everything successfully upgraded then you can roll out the same steps in your Production environment.


Now look at the following Table which will clear up most of your doubts

Now as you can see the VCD (vCloud Director) is the 1st one in the process of upgrade. As the upgrade starts with the lowest number First you go with the same recommendation for all the products and related upgrade path (whether you run only few, all or some of them) and you should not be having any issues.

I would also suggest to search http://kb.vmware.com/ and the Release Notes for each products involved, for any known issues/gotchas during the upgrade so that you can take care of them before you proceed with the Upgrade.

Now here if you are using Cisco Nexus 1000v then you need to make sure before you upgrade your ESXi/vCenter Server, please make sure that the VSM/VEM version are supported/Compatible otherwise after upgrading your vCenter server you may end up in a dead water with NO connectivity on the VMs/ESXi if hosts if everything is managed through Nexus 1000v. With your existing version of VSM you can check up the Cisco's website for the Compatibility of ESXi/VC.

As Nexus 1000v is a Cisco product the above Matrix is not reflecting the same but it will be definitely grab a spot after vCenter/ESXi host upgrade paths.

There is a published KB 2037630 available too for more information which you can refer if you are running SRM/VR and VDP solution in your environment and which will guide you on the steps of overall upgrade.

And suppose you are running vCenter Server Appliance then the above KB will not help but instead refer the KB 2031331.

Enjoy upgrading VMware Environment !!

Share and care !

Cheers.

vSphere VSS & VDS - Cisco Nexus1000v Feature Comparison

I have been asked or requested, so many times before, while talking to the end user or with some one about the features offered by VMware VSS (Virtual Standard Switch), VDS (VMware Distributed Switch) and Cisco Nexus 1000v VDS and I was thinking to gather as much information as I could and blog about it. So here it is.

I believe that any advanced user, working with VMware environment for few years, is aware of the fact that Nexus 1000v is Cisco's product and it will appear as a VDS inside the vSphere GUI. So by looking at the GUI if the naming convention is not used properly for the VDS than you may interpret as a regular VMware VDS so to avoid the confusion one can check the "Summary" Tab of the VDS to verify the vendor of that VDS and also the version of that VDS.

IBM has released Nexus 5000 VDS about which I have not heard much discussion till date. So in this comparison I am not including IBM VDS.

Have a look at the following Table, which I have tried including possibly every feature offered by the VMware VSS, VMware VDS and Nexus VDS. If I miss something then do leave the comment or get in touch with me and I will update the article.

Feature	VMware vSphere 4.x VSS	VMware vSphere 5.5 VDS	Cisco Nexus 1000V 4.2 (1)-SV2(2.2)
Switching Features
Layer 2 Forwarding	YES	YES	YES
IEEE 802.1Q VLAN tagging	YES	YES	YES
Multicast Support (IGMP V2 and V3 support)	YES	YES	YES
IGMP V3 Snooping	-	-	YES
VMware vMotion	YES	YES	YES
Network VMware vMotion	-	YES	YES
Multi-Nic vMotion Support	-	YES	YES
Physical Switch Connectivity
Virtual Mac Pinning	YES	YES	YES
EtherChannel	YES	YES	YES
Virtual Port Channels	-	-	YES
Link Aggregation Control Protocol (LACP)	-	YES	YES
Static LAG	-	YES	YES
Dynamic LAG	-	YES	YES
Load – Balancing Algorithms
Virtual Port ID	YES	YES	YES
Source MAC Address	YES	YES	YES
Source and destination IP Address	YES	YES	YES
Source MAC Address	-	YES	YES
Additional hashing Options	-	YES	YES
Load Based Teaming	-	YES	-
Source and Destination port IP	-	YES	YES
Advanced Port channel	-	YES	YES
IP Hash	YES	YES	YES
Traffic Management Features
Transmit-rate (from Virtual Machine) limiting	YES	YES	YES
Receive-rate (to Virtual Machine) limiting	-	YES	YES
ISCSI Mutipathing	YES	YES	YES
Unicast Flooding Control	-	YES	YES
Quality-of-Service (QoS) marking
Differentiated Serviecs Code Point (DSCP)	-	YES	YES
Type of Service	-	-	YES
Class of Service	-	YES	YES
802.1Q	-	YES	YES
Network IO control (NIOC)	-	YES	YES
Transmit-rate (from Virtual Machine) limiting	YES	YES	YES
Receive-rate (to a Virtual Machine) limiting	-	YES	YES
802.1p	-	YES	YES

Security Features
Port Security	YES	YES	YES
VMware VMsafe Compatible	YES	YES	YES
Private VLANs (PVLANs) 512	-	YES (no limit)	YES(512)
Local PVLAN enforcement	-	YES	YES
PVLAN with Promiscuous Trunk	-	YES	YES
Access Control List (ACLs)	-	YES	YES
Virtual Service Domain	-	-	YES
DHCP Snooping	-	-	YES
IP source Guard	-	-	YES
Dynamic ARP Inspection	-	-	YES
MAC ACL	-	YES	YES
VXLAN	-	YES (no limit)	YES (2048)

Management Features
VMware vCenter Support	YES	YES	YES
VMware vCloud Director Support	YES	YES	YES
vCloud Director Automation Center support	YES	YES	YES
RESTful API	YES	YES	YES
Third-party-Accessible APIs	YES	YES	YES
Network Policy Groups	YES	YES	YES
Multitier Policy Groups	-	-	YES
Packet Capture and Analysis	-	YES	YES
RADIUS and TACACS+	-	-	YES
LLDP	-	YES	-
Network CLI	-	-	YES
Server CLI	YES	YES	-
Configuration and Management Console Interface	vSphere Client	vSphere Web Client/vSphere Client	vCenter and Cisco CLI
Graphical UI	YES	YES	-
Config Backup and Restore	-	YES	YES
Network Rollback and Recovery	-	YES	-
IPv6 for Management	YES	YES	YES
Monitoring and Troubleshooting
VMware Port Mirroring (promiscuous)	YES	YES	YES
Switched Port Analyzer (SPAN)	-	YES	YES
Encapsulated Remote SPAN (ERSPAN)	-	YES	YES
NetFlow ver. 9	-	-	YES
NetFlow ver. 10 (Ipv6, VXLAN flows)	-	YES	-
Network Health Check	-	YES	-

Simple Network Management Protocol (SNMP) V3 Read and Write (V1,V2C)	-	YES	YES
Cisco Discovery Protocol (CDP) v1 and v2	YES	YES	YES
Syslog *	YES	YES	YES
ACL Logging	-	YES	YES
SNMP ACLs	-	-	YES
Network Virtualization
VXLAN support with Multicast	-	YES	YES
VXLAN support without Multicast	-	-	YES
ARP suppression for VXLAN	-	-	YES
L3 Gateway for NV	-	YES	YES
Site-to-Site IPSec VPN	-	YES	YES
Remote Access SSL VPN	-	YES	-
Scalability
Hosts per Switch	500	500	128
Switches per management system (VC)	128	128	32
VXLAN segments	-	10000 (VCNS)	2048
VLAN (no vxlan)	4096	4096	2048
Port Groups/Profiles per Switch	4096	10000	2048
Virtual Ports per host	4096	4096	300
Virtual Ports per Switch	10000	60000	4096
Max Active Virtual ports per Host	1016	1016	300
Max MAC Addresses per Host	No limit	No limit	32000

(NOTE: Above Table is updated for VDS 5.5 so some features which are not shown as supported on VDS 5.5 are fully supported for VDS with NSX so please verify the same.)

* Syslog information is exported and included with VMware ESX/ESXi server events

So you can select the option depending on the features you need. Both VMware and Cisco VDS requires Enterprise + License. Now if you see that I have not included the column for VMware VDS 4.1. The reason behind it, as most of the features available with VDS 5.5 and there are some more features offered with the latest version of VDS so I encourage the reader to use the latest version anyway. Even the Cisco Nexus VDS version is 4.2 (1) - SV2(2.2) which is the latest one.

For configuration of VDS you can refer the online documentation available on www.vmware.com and for Cisco Nexus 1000v VDS you can refer the documentation page here.

I will update the Sheet with NSX 6.0.5 once I get some time so please be waited till then and DO NOT leave the comment for the requesting the same Update :-) but I appreciate any Feedback or comments regarding any features (if missed) or needs an update to the above Table.

Share and Care !!

Enjoy !!