EMC VNXe 5100 and Checkpoint Folders on VMFS volumes

Hi,

Came across an interesting situation where the virtual machine stopped responding as the External Backup application tried to take the snapshot and the storage ran out of disk space

Infrastructure details

ESXi: 5.x

VM: Windows 2Kx OS

Storage: EMC VNXe 5100

4 HDDs are mounted on the VM which are coming from separate individual VMFS datastore.

All 4 VMDKs consumed the overall space of 4 TB in total.

The Datastore summary states the total size is 18.7 TB consumed out of 28.25 TB provisioned. This is the only VM using those 4 LUNs for each VMDK.

So the question is where is the additional space of 14TB lost apart from 4 TB used by the VM??

Here is the output of du -h for the VM and /vmfs/volumes

/vmfs/volumes/50cc5dd8-555143cbd-44d2-ac162d783818/SVRAP01 # du -h
332.6G  .
/vmfs/volumes/50cc5dd8-555143cbd-87d5-ac162d783818/SVRAP01 # du -h
3.7T    .

du -h
8.0K    ./.ckpt_group.vmware_21_sg_443.fs.13/lost+found
272.0K  ./.ckpt_group.vmware_21_sg_443.fs.13/.etc
3.7T    ./.ckpt_group.vmware_21_sg_443.fs.13/SVRAP01
8.0K    ./.ckpt_group.vmware_21_sg_443.fs.13/.vSphere-HA
3.7T    ./.ckpt_group.vmware_21_sg_443.fs.13
8.0K    ./.ckpt_group.vmware_21_sg_441.fs.13/lost+found
272.0K  ./.ckpt_group.vmware_21_sg_441.fs.13/.etc
3.7T    ./.ckpt_group.vmware_21_sg_441.fs.13/SVRAP01
8.0K    ./.ckpt_group.vmware_21_sg_441.fs.13/.vSphere-HA
3.7T    ./.ckpt_group.vmware_21_sg_441.fs.13
8.0K    ./.ckpt_root_rep_ckpt_51_832916_2/lost+found
272.0K  ./.ckpt_root_rep_ckpt_51_832916_2/.etc
3.7T    ./.ckpt_root_rep_ckpt_51_832916_2/SVRAP01
8.0K    ./.ckpt_root_rep_ckpt_51_832916_2/.vSphere-HA
3.7T    ./.ckpt_root_rep_ckpt_51_832916_2
8.0K    ./.ckpt_root_rep_ckpt_51_832916_1/lost+found
272.0K  ./.ckpt_root_rep_ckpt_51_832916_1/.etc
3.7T    ./.ckpt_root_rep_ckpt_51_832916_1/SVRAP01
8.0K    ./.ckpt_root_rep_ckpt_51_832916_1/.vSphere-HA
3.7T    ./.ckpt_root_rep_ckpt_51_832916_1
8.0K    ./lost+found
272.0K  ./.etc
3.7T    ./SVRAP01
8.0K    ./.vSphere-HA
18.7T   .


So as you can see the overall space used on the VM is approx 4 TB and at the datastore level is 18.7 TB.
After doing more research found that the these files are created during EMC SAN root replication to preserve a replication pair which had no prior replication relationship. So technically this includes all 4 LUNs used by the virtual machine.

Found few articles here (EMC community) and here (By Justin Paul) found the culprit as well for the Datastore space consumption and how to reclaim it.

As a work around we storage vMotioned smaller VMDK to another datastore and now the VM can be powered on and the users can work with it.

Plan recommended to contact EMC support for better guidance on how to reclaim that space properly without losing any data.

Hope this helps to find out the lost space on the VMFS datastore !

Share and care please !!

Is my VM running off the Snapshot Delta or Base disk?

I came across few instance recently where the the Virtual machine was either consolidating all the snapshots taken by backup software or adding/removing snapshots and at the same time Edit Settings of the virtual machine is not giving you any details as its grayed out.

User is finding so many delta disks on the respective datastore/s which is occupying the disk space on the Shared LUNs/local Datastores.

Now how to find if the VM is running off base disks or its running off snapshot files.

To recover the space consumed by the deltas which are not really having any information inside.

There are few ways you can find out if the VM indeed running on snapshot delta files or not.

1st method is to use the SSH/DCUI/DRAC/iLO/KVM/RSA whichever method you seems easy and you are comfortable with, using which access the console of ESXi.
Login with root account privileges and change to the virtual machine directory.

#cd /vmfs/volumes/datastore1/vmname

Once you are in the VM directory and then run the following command to see if the virtual machine snapshot file in use.

#less vmname.vmx | grep -i *.vmdk

The above command will list all the vmdk files in use for that virtual machine which includes all the snapshot delta file and base disks. The example will be scsi0:0 and the file name vmname-000001.vmdk or scsi1:0 and the file name vmname-000004.vmdk etc. etc. The number of scsi devices represents the number of VMDKs presented to the virtual machine.

Now there are many 3rd party applications available in the market which can do this job for you. You just need to run those tools and they will give you the report in various forms. You can use the PowerCLI script also to find out the same within the Datacenter or at the cluster level.


2nd method is if you can go under Edit Settings option then you can login through vSphere client and connect to the vCenter/ESXi host and then click on the Edit Setting of the Virtual Machine.

Click on HDD1 and go to the disk file option on top right corner and put the cursor and click "End" button on the key board which will take you to the end of the file name. If the datastore name and virtual name is shorter then you dont need to do that and you can see the file name right away and see if there is -000000.vmdk added to the file than you know that the VM is using the Snapshot file.


If the name does not have and if you see vmname.vmdk then you know its running off the base disk. For the subsequent disk you will see vmname_1.vmdk, vmname_2.vmdk and so on.


You need to repeat the step for each HDD which you are using on the virtual machine.


Hoping that the above will help you out finding unnecessary snapshot files for the VMs which are using critical space and you can recover that space by dealing with them accordingly.




Please share and care!!