EMC VNXe 5100 and Checkpoint Folders on VMFS volumes

Hi,

Came across an interesting situation where the virtual machine stopped responding as the External Backup application tried to take the snapshot and the storage ran out of disk space

Infrastructure details

ESXi: 5.x

VM: Windows 2Kx OS

Storage: EMC VNXe 5100

4 HDDs are mounted on the VM which are coming from separate individual VMFS datastore.

All 4 VMDKs consumed the overall space of 4 TB in total.

The Datastore summary states the total size is 18.7 TB consumed out of 28.25 TB provisioned. This is the only VM using those 4 LUNs for each VMDK.

So the question is where is the additional space of 14TB lost apart from 4 TB used by the VM??

Here is the output of du -h for the VM and /vmfs/volumes

/vmfs/volumes/50cc5dd8-555143cbd-44d2-ac162d783818/SVRAP01 # du -h
332.6G  .
/vmfs/volumes/50cc5dd8-555143cbd-87d5-ac162d783818/SVRAP01 # du -h
3.7T    .

du -h
8.0K    ./.ckpt_group.vmware_21_sg_443.fs.13/lost+found
272.0K  ./.ckpt_group.vmware_21_sg_443.fs.13/.etc
3.7T    ./.ckpt_group.vmware_21_sg_443.fs.13/SVRAP01
8.0K    ./.ckpt_group.vmware_21_sg_443.fs.13/.vSphere-HA
3.7T    ./.ckpt_group.vmware_21_sg_443.fs.13
8.0K    ./.ckpt_group.vmware_21_sg_441.fs.13/lost+found
272.0K  ./.ckpt_group.vmware_21_sg_441.fs.13/.etc
3.7T    ./.ckpt_group.vmware_21_sg_441.fs.13/SVRAP01
8.0K    ./.ckpt_group.vmware_21_sg_441.fs.13/.vSphere-HA
3.7T    ./.ckpt_group.vmware_21_sg_441.fs.13
8.0K    ./.ckpt_root_rep_ckpt_51_832916_2/lost+found
272.0K  ./.ckpt_root_rep_ckpt_51_832916_2/.etc
3.7T    ./.ckpt_root_rep_ckpt_51_832916_2/SVRAP01
8.0K    ./.ckpt_root_rep_ckpt_51_832916_2/.vSphere-HA
3.7T    ./.ckpt_root_rep_ckpt_51_832916_2
8.0K    ./.ckpt_root_rep_ckpt_51_832916_1/lost+found
272.0K  ./.ckpt_root_rep_ckpt_51_832916_1/.etc
3.7T    ./.ckpt_root_rep_ckpt_51_832916_1/SVRAP01
8.0K    ./.ckpt_root_rep_ckpt_51_832916_1/.vSphere-HA
3.7T    ./.ckpt_root_rep_ckpt_51_832916_1
8.0K    ./lost+found
272.0K  ./.etc
3.7T    ./SVRAP01
8.0K    ./.vSphere-HA
18.7T   .


So as you can see the overall space used on the VM is approx 4 TB and at the datastore level is 18.7 TB.
After doing more research found that the these files are created during EMC SAN root replication to preserve a replication pair which had no prior replication relationship. So technically this includes all 4 LUNs used by the virtual machine.

Found few articles here (EMC community) and here (By Justin Paul) found the culprit as well for the Datastore space consumption and how to reclaim it.

As a work around we storage vMotioned smaller VMDK to another datastore and now the VM can be powered on and the users can work with it.

Plan recommended to contact EMC support for better guidance on how to reclaim that space properly without losing any data.

Hope this helps to find out the lost space on the VMFS datastore !

Share and care please !!

Virtual Disk Service entered into stopped State and Sophos AV

Hi,

Recently I was part of the discussion where the following infrastructure was involved.

ESXi 5.1.0

VM - Windows 2008 R2

User was  getting the error under Event Viewer inside the Guest Operating System

"Virtual disk service entered into a Stopped State"'

Now operation impact was none for the VM and multiple VMs are in the same situation.

No errors logged under the vmware.log of the virtual machine either. VMkernel log has no entries for such events so this led the focus more inside the Guest OS of the VM.

Upon investigating the issue further, found that user was running Sophos AV inside the virtual machine at the Guest Operating System Level. Unsure about the exact version of AV,  but once we stopped the service/application within the Guest Operating System then we stopped getting the error any more under the Event viewer.

Found a thread which was discussing about the version 7.6.18 (Updated 4/15/2010) which led towards the investigation at the Guest OS Level.

Hope this helps other users who are encountering the same issue.

Please share and care !!

Device or Resource Busy Error on vSphere client

I can hardly count the instances where anyone working with VMware vSphere environment has not encountered this error "Device or Resource Busy"

There are  some reasons why this error coming up.

Few instances mainly can be listed as follows

• When you browse a datastore, it does not have any files or folders.
• Under /vmfs/volumes/ there are no files or directories.
• While deleting a datastore, you see the error:
  
  Device or Resource Busy
   
• Deleting the partition table of this datastore does not resolve the issue.
• Deleting a vmdk fails for the VM which is not longer in use 

There are few articles you can find here which discusses all these conditions and how to resolve them.


http://kb.vmware.com/kb/1015791   and  http://kb.vmware.com/kb/1039362

If you have encountered other instances which are not documented then you can always provide the feedback to the article and it will be updated accordingly.

If you search http://kb.vmware.com/ you may find other articles which highlights other issues apart from the listed above.

Hope you will find this information useful.

Virtual Disk Format 5.0 on VMware ESXi

Let me ask you a question.

How much you know about the virtual disk format in detail??

Its in the format of vmdk, it resides on VMFS volume, Descriptor file and Raw Data files stored in two separate disks etc. etc.

Anything else you can think of now.............

Here I am ending your anxiety to know everything about the virtual disk format version 5.0 (which is the recent one).

Have a look at the document here.

If you are reading this through Twitter then DO NOT forget to RT the Tweet ;-)

Enjoy reading something interesting which you will not find in any published book either. Just Kidding.

Take care.