Recovering vCenter Server after the major Outage

Hi,

I have discussed this issue so many time with my colleagues and friends who experienced the situation where the whole Datacenter had an Outage due to X reason and then depending on the products running in the environment, it is getting difficult to restore everything back to normal.

The products which can add complexity in the restoration process can be Nexus 1000v, SRM, VMware VDS. Now let me emphasize here that there is nothing wrong with these products but to recover vCenter Server which is a Critical/Key component in the whole infrastructure.

I am going to explain few scenarios on how to recover vCenter Server and get everything up and running which may take time from minutes to few hours and sometimes a day or two (depends on the size and inventory of  the Datacenter).

1) If running VMware VDS, Nexus 1000v VDS

First of all one needs to find out where the vCenter Server was running lastly.

Now lets assume you are running fewer number of ESXi hosts in the cluster/Datacenter (lets say 1-20) then if you have the location documented if the vCenter was tied up on certain hosts only using the DRS rules then you can connect directly (assuming network connection is working or else we need to get that working first) to that particular ESXi host (using putty or DCUI and later using vSphere client) and find out if the vCenter Virtual Machine is still registered on that host or not. (If running WebClient then I will be covering that in an upcoming blog post).

vmware-cmd -l

The above command will tell us which VMs are still registered on the host and you can power on the vCenter Server through command line or from vSphere Client. Once logged in you can open the console of the vCenter and login using the local administrator account. Now here there is another assumption going here that if the vCenter Database is residing on another VM or a physical machine than there should be network connectivity available between the DB machine and vCenter Server.

Now lets take a situation where the management Network of ESXi host is configured on VDS (VMware) then if needed you need to restore the Standard Switch on the ESXi host so that atleast you can connect to the Management interface. Then you can try accessing vCenter Server from the console right on the vCenter server VM using localhost option and you can see all the inventory within the VC virtual machine. Once you power on all the ESXi hosts you should be able to see all the VMs and other inventory items.

If the VMs are showing inaccessible or orphaned then you can simply unregister the VMs and register them again. (This needs the information documented somewhere about the names of the virtual machines, network they are using / connected to, Datastores they are using or configured on).

If running Nexus 1000v then make sure both VSMs are powered on and can reach to vCenter Server and ESXi host/s. If needed you can register the VSMs on the same host as vCenter Server to make things easy. You can put them back on to the host where they belong as per the DRS rules (if any specified for Nexus
VSMs) set for them once everything is in working order.

2) Where you have large number of Hosts in the cluster (more than 20-50) so atleast you need the name of the Datastore where the vCenter Server files are residing. (Assuming no Storage vMotion occurred before the outage happened) so it will be a good practice to designate a specific datastore for vCenter Server virtual machine files so that it will be easy to just connect to that datastore using ssh from one of the ESXi host which is using as a shared datastore.

You can setup ssh to the host or use local ESXi shell option from DCUI and browse to the datastore and register the vCenter Server on that ESXi host.


vmware-cmd -s register .vmx

or

vim-cmd solo/reistervm /vmfs/volumes/datastore_name/VM_directory/VM_name.vmx

Once you see the vCenter entry in the inventory then just power it on. Make sure the vNIC is connected. Now if the VM was part of VDS then it must be connected to one of the dvPortGroup on the VDS which is not accessible as vCenter Service is not available yet. So you need to create a Standard vSwitch on the ESXi host where you registered the vCenter Server VM. Provide at least one uplink which can carry the same VALN traffic (if there was VLAN configured for the vCenter network) so get the connectivity. Now again you need to see if you have a spare NIC which you can use on VSS and if not (assuming all the NICs are used by VDS) then you just need to use one on VSS which was assigned to VDS before the outage.

3) Now assume you don't know the Datastore / ESXi host name where the vCenter Server was residing and running lastly and you have more than 50/100 hosts in the cluster. 

Here's comes the real part of this post so be patient and read on.

The question is how to find the vCenter Server virtual machine directory.

There are few methods you can go with

a) Run the PowerCLI script across all the ESXi hosts which definitely is a time consuming task as you need to connect to each host individually and run the command.

I can update the post here if someone comes with a PowerCLI one liner so please leave a comment.

b) If you are running SQL Database for vCenter Server database then you need to find out that VM first.
Login to the SQL VM (with local admin or using Domain Admin account (if DC/AD machine is available and accessible). Then login to the SQL Database using the Administrator Account. Run the following query against the VC Database.

First query only returns a host ID

select HOST_ID from VPX_VM where DNS_NAME like '%vcenter%'

You need to use the valued derived in the 2nd query. You just need to replace "vcenter' in the above query to the actual name of your vCenter Server VM name.

select * from VPX_ENTITY where ID='x'

The above will give you the result with the ESXi host where it was lastly registered and running.

Hopefully you can use the same query on Oracle Database as well but not sure so if someone is Oracle Expert then please leave a comment and I will modify the post with actual Query for Oracle database.

The above methods are having certain assumptions such as connectivity, login information to vCenter Server, SQL Server, ESXi host etc. etc. which are needed in the whole restore process.

4) Now the last situation where you dont know the name of the vCenter Server/ESXi Host/Datastore name then you just cross the fingers, pray to God and start digging for the VM on each and every direction possible and make a resolution first that you will DOCUMENT everything about your virtual Inventory going forward. Not joking here as seen instances like this too.

Let me know if you feel to add/update the existing information and I will be happy to do it. Just need your comment through any available medium.

Please share and care !!

Thanks for your time.

#Web-Scale Infrastructure by #Nutanix

virtualpatel.blogspot.com: Web-Scale Infrastructure by Nutanix: Hello, For last few weeks I was reading this term a lot on social media and kept wondering what exactly Web-Scale is. Nutanix has relea...

Web-Scale Infrastructure by Nutanix

Hello,

For last few weeks I was reading this term a lot on social media and kept wondering what exactly Web-Scale is. Nutanix has released their Hyper converged solution 3 years back to meet the needs of varying industries and which can match variety of requirements too which can range from SMB to Mid Size, or from Enterprise level to Federal customers. Their NX- range of solutions are available to meet varying range of requirements.

Their main components which build up as part of Webscale can be pointed out as follows.

Hyper Convergence on x86 nodes which allows you the Freedom to scale as required. One server at a time.

Now for the organization who has budget for certain department but if its growing then to keep up with the growth, it gives the ability to add the Nutanix Node (server) one at a time. Now this is giving you freedom of not doing a large investment at the same time but you can spread it over period of time in future years to come. Accounting wise and Business Management perspective this is very crucial as you are not stuck with the Investment and ROI aspects of it. Depreciation also counts at some point but Finance Dept. will be happy as they dont have the deal with this kind of situation.

Intelligence - this is provided by their Nutanix's own kernel developed which will run on individual node known as CVM and two CVMs always in connection with each other all the time using the backend network connection. Even if one CVM goes down due to the underlying hardware failure or some other reasons, still the functionality wont be affected in any of the features and operation served by the nodes.

Everything will keep functioning as they supposed to work.

Distributed Architecture - This is another plus point where all the infrastructure related information is stored in the distributed fashion across the Nutanix nodes and such information can contain mostly all the items including data, metadata, operations occurring between virtual machines etc. etc. this information is getting stored within whole cluster so if one component fails still there wont be any information loss. By having this feature you can easily plan ahead and make the infrastructure more scalable without even worrying about the limitation. This scalability gives you the possibility to plan appropriately without wasting resources and time (eventually it involves this factor as we need time to setup/implement and to maintain it later on too) which is very crucial for any organization from small to large size.

Self-healing System : Nodes are built up in such a way that the failures of the components are not becoming responsible for system crash or operational impact of production workload. Fault Isolation and Automatic Recovery allows this facility which keeps the nodes active and keep running the overall system.

API based automation and Analytics : These two features are the pillars where automation in place and if one is using any system monitoring to achieve any data-driven efficiency and REST based programmable interface for the Datacenter management. This is a big one as going forward you can use the feature when working with various cloud based systems which can be Private Cloud, Public Cloud or Hybrid Cloud.

Multi-application support :  On Nutanix Nodes you can run various type of applications working as part of VDI, Big Data, Private Cloud, Enterprise Branch Office etc. etc.

And the following is an extract I have taken from one of the post on Nutanix

• Web-scale system enables a non-disruptive approach to disruptive tasks, such as rolling or forklift upgrades, expandable clusters, always-on clusters, and all workflows always done online. Examples of Nutanix scalability include ability of adding and removing nodes dynamically, rolling upgrades of one node at a time ( different nodes can be at different version during the rolling upgrade), datastore availability when there is a disk failure and many other features. Nutanix Engineering is adding more features to improve resiliency and simplify workflows.

This allows any IT admin a breath of Relax and upgrades are becoming critical where there are SLAs to achieve which can range from 9999 to 99999 sometimes. Without any disruption you can upgrade or add / scale the Infrastructure which is very valuable to the any type of Industry working 24x7x365.

                                                       

The above diagram is about CAP theorem (Consistency, Availability and Partition Tolerance) and you can see the clear relationship between the various components and what Nutanix is offering is pretty much aligns with each of them. The diagram again is a courtesy by Nutanix blogpost.

Now finishing here with a pic of the very nice T-shirt received from Nutanix and as its a Wednesday today so it will be a perfect day to Web-Scale it. ;-)

Go and use the Hyper-Convergence with ease using the Web-Scale Infrastructure offered by Nutanix !!

Please share and care !!

Thanks for your time.

Some facts about vCenter Server Inventory Service

The Inventory Service uses some sort of XML database called xDB. Unfortunately, that database uses a flat file which ends in .log (which is stored on the filesystem of the VC) - during housekeeping, it is not uncommon for administrators to search for files that end in .log and to remove them.

The Inventory Service is used to search the VC inventory and it is used when checking the VC Service status. It was created to offload processing required to search inventory as the C# client hammers vpxd when searching.

By default, there are scripts that are installed on VC to backup and restore the IS database.

Under 5.0, the Inventory Service and vCenter Server both exist on the same machine. Under 5.1, vCenter Server and Inventory Service can exist on different machines.

After my digging, it appears a popular troubleshooting step is to reset the IS database. Our official documentation outlines how to do that:

http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc_50%2FGUID-EBB03FB7-F1AE-433C-A78D-A0E345EB0986.ht ml

The instructions to reset the IS database are also supplied in KB article below:

Logging in to the vCenter Server 5.0 Web Client fails with the error: unable to connect to vCenter Inventory Service http://kb.vmware.com/kb/2017750

Under 5.0, it appears you can reset the DB with no adverse affects as the information in the database will repopulate - although I'm not sure how long that takes or how that is done.

Under 5.1, it appears the IS database includes new inventory tagging information although I believe this feature is not well known or used very often so most customers probably do not have any tagging information in their IS database.

In 5.5 the behavior still needs to be verified. If anyone has seen this they can leave a comment and I will update the post with 5.5.

Thanks for reading and share please !!

Sharing is caring !

How to put Custom Disclaimer for VMware Horizon View 5.x portal

To put the custom disclaimer on the VMware Horizon View 5.x portal you need to actually use the portal download page and then follow the procedure below

■	Download the installer files for the types of View Client you want to use in your environment. The URL to the View Clients download page ishttps://www.vmware.com/go/viewclients.
■	Determine which HTTP server will host the installer files. The files can reside on a View Connection Server instance or on another HTTP server.

Procedure

1	On the HTTP server where the installer files will reside, create a folder for the installer files. For example, to place the files in a downloads folder on the View Connection Server host, in the default installation directory, use the following path: C:\Program Files\VMware\VMware View\Server\broker\webapps\downloads The links to the files would then use URLs with the format https://server-name/downloads/client-installer-file-name. For example, a server with the name view.mycompany.com would use the following URL for View Client for Windows:https://view.mycompany.com/downloads/VMware-viewclient.exe. In this example, the folder named downloads is located in the webapps root folder.
2	Copy the View Client installer files into the folder. If the folder resides on View Connection Server, you can replace any files in this folder without having to restart the VMware View Connection Server service.
3	On the View Connection Server machine, copy the portal-links.properties file and the portal.properties file located ininstall-path\Server\Extras\PortalExamples.
4	Create a portal folder the directory C:\ProgramData\VMware\VDM, and copy the portal-links.properties andportal.properties files into the portal folder.
5	Edit C:\ProgramData\VMware\VDM\portal\portal-links.properties file to point to the new location of the installer files. You can edit the lines in this file and add to them if you need to create more links. You can also delete lines. The following examples show properties for creating two links for View Client for Windows and two links for View Client for Linux: link.win=https://server-name/downloads/VMware-viewclient-x86_64-y.y.y-XXXX.exe#win link.win.1=https://server-name/downloads/VMware-viewclient-y.y.y-XXXX.exe#win link.linux=https://server-name/downloads/VMware-viewclient-x86_64-y.y.y-XXXX.rpm#linux link.linux.1=https://server-name/downloads/VMware-viewclient-y.y.y-XXXX.tar.gz#linux In this example, y.y.y-XXXX indicates the version and build number. The win text at the end of the line indicates that this link should appear in the browser if the client has a Windows operating system. Use win for Windows, linux for Linux, and mac for Mac OS X.
6	Edit C:\ProgramData\VMware\VDM\portal\portal.properties file to specify the text to display for the links. These lines appear in the section of the file called # keys based on key names in portal-links.properties. The following example shows the text that corresponds to the links specified for link.win and link.win.1: text.win=View Client for Windows 32 bit Client users text.win.1=View Client for Windows 64 bit Client users
7	Restart the VMware View Connection Server service.

When end users enter the URL for View Connection Server, they see links with the text you specified. The links point to the locations you specified.

All the above steps are taken from the official Documentation for easy reference.

Hope you will now think about changing the portal :-)

Enjoy !!

Share and Care !!