EMC VNXe 5100 and Checkpoint Folders on VMFS volumes

Hi,

Came across an interesting situation where the virtual machine stopped responding as the External Backup application tried to take the snapshot and the storage ran out of disk space

Infrastructure details

ESXi: 5.x

VM: Windows 2Kx OS

Storage: EMC VNXe 5100

4 HDDs are mounted on the VM which are coming from separate individual VMFS datastore.

All 4 VMDKs consumed the overall space of 4 TB in total.

The Datastore summary states the total size is 18.7 TB consumed out of 28.25 TB provisioned. This is the only VM using those 4 LUNs for each VMDK.

So the question is where is the additional space of 14TB lost apart from 4 TB used by the VM??

Here is the output of du -h for the VM and /vmfs/volumes

/vmfs/volumes/50cc5dd8-555143cbd-44d2-ac162d783818/SVRAP01 # du -h
332.6G  .
/vmfs/volumes/50cc5dd8-555143cbd-87d5-ac162d783818/SVRAP01 # du -h
3.7T    .

du -h
8.0K    ./.ckpt_group.vmware_21_sg_443.fs.13/lost+found
272.0K  ./.ckpt_group.vmware_21_sg_443.fs.13/.etc
3.7T    ./.ckpt_group.vmware_21_sg_443.fs.13/SVRAP01
8.0K    ./.ckpt_group.vmware_21_sg_443.fs.13/.vSphere-HA
3.7T    ./.ckpt_group.vmware_21_sg_443.fs.13
8.0K    ./.ckpt_group.vmware_21_sg_441.fs.13/lost+found
272.0K  ./.ckpt_group.vmware_21_sg_441.fs.13/.etc
3.7T    ./.ckpt_group.vmware_21_sg_441.fs.13/SVRAP01
8.0K    ./.ckpt_group.vmware_21_sg_441.fs.13/.vSphere-HA
3.7T    ./.ckpt_group.vmware_21_sg_441.fs.13
8.0K    ./.ckpt_root_rep_ckpt_51_832916_2/lost+found
272.0K  ./.ckpt_root_rep_ckpt_51_832916_2/.etc
3.7T    ./.ckpt_root_rep_ckpt_51_832916_2/SVRAP01
8.0K    ./.ckpt_root_rep_ckpt_51_832916_2/.vSphere-HA
3.7T    ./.ckpt_root_rep_ckpt_51_832916_2
8.0K    ./.ckpt_root_rep_ckpt_51_832916_1/lost+found
272.0K  ./.ckpt_root_rep_ckpt_51_832916_1/.etc
3.7T    ./.ckpt_root_rep_ckpt_51_832916_1/SVRAP01
8.0K    ./.ckpt_root_rep_ckpt_51_832916_1/.vSphere-HA
3.7T    ./.ckpt_root_rep_ckpt_51_832916_1
8.0K    ./lost+found
272.0K  ./.etc
3.7T    ./SVRAP01
8.0K    ./.vSphere-HA
18.7T   .


So as you can see the overall space used on the VM is approx 4 TB and at the datastore level is 18.7 TB.
After doing more research found that the these files are created during EMC SAN root replication to preserve a replication pair which had no prior replication relationship. So technically this includes all 4 LUNs used by the virtual machine.

Found few articles here (EMC community) and here (By Justin Paul) found the culprit as well for the Datastore space consumption and how to reclaim it.

As a work around we storage vMotioned smaller VMDK to another datastore and now the VM can be powered on and the users can work with it.

Plan recommended to contact EMC support for better guidance on how to reclaim that space properly without losing any data.

Hope this helps to find out the lost space on the VMFS datastore !

Share and care please !!

VSAN Resources

Hello,

In my previous post on NSX Resources  I have mentioned about VSAN which was announced on 10th March 2014.


VSAN Download Bits as part of VMware vCenter Server 5.5 Update 1


Learning / WhitePapers :


VSAN Design and Sizing Guide

vSphere 5.5 VSAN Documentation

Virtual SAN Hardware Compatibility Guide

VMware Virtual SAN Datasheet

Getting Started with VSAN and Openstack

Proof of Concept for VSAN

VMware VSAN Resources

VSAN Release Notes

Ready Nodes and Ready Block Configurations

Labs :

VMware HOL on VSAN

Lab Manuals and PDF


Videos/Podcast :

VMware Communities Roundtable Podcast – 251 – VSAN

VSAN Youtube VMware Channel

VMware Walkthrough Videos


VSAN and APIs

VSAN APIs and Exploring APIs series By William Lam



VSAN using PowerCLI

Creating VSAN cluster using PowerCLI


VSAN Interoperability with Openstack

VSAN interoperability With Openstack


At VMworld 2013


VMworld 2013: What’s New in 5.5 – vSAN

VMworld 2013: What’s New in VSAN 1.0

VSAN posts by various Bloggers

VSAN VMware blogs
VSAN information by Duncan Epping
VSAN series by Cormac Hogan
Five Reasons Why VSAN Is Different 
Considering VSAN
VMware VSAN Launch Q&A
VMware’s new VSAN – What Matters
A few thoughts and opinions on VSAN and Hyperconvergence
VMware VSAN cements changes coming to the data center
vSphere 5.5 VSAN, Introducing Your New Storage Vendor VMware
Virtual SAN: Powerfully Simple and Simply Powerful
Virtual SAN VM Density Contest Results!
vSAN 1.0 Announcement
vSphere 5.5 Improvements Part 4 - Virtual SAN (VSAN)
In Which I Muse Over VMware’s Virtual SAN Architecture
What’s New in vCloud Suite 5.5: Virtual SAN (VSAN)
How to install VSAN in Nested Environments

Rebuilding your Virtual SAN Lab? Wipe the disks first!


Performance Benchmark:

Performance of VSAN in VMware View
VSAN Quick Monitoring reference Guide
VDI Benchmarking using View Planner - multipart series
Performance Series


Troubleshooting:

Additional Consideration for VSAN Cluster
KB 2064240 Enabling or capturing performance statistics using
Virtual SAN Observer for VMware Virtual SAN
KB 2058521 Network interfaces used for Virtual SAN are not ready
KB 2059091 Adding a host back to a Virtual SAN cluster after an ESXi host rebuild
KB 2058794 Virtual SAN clustering has been disabled on an ESXi host 
KB 2058529 irtual SAN networking experiences connectivity problems
KB 2058530 Virtual SAN network redundancy has been reduced
KB 2060786 Enabling Virtual SAN on vSphere 5.5 does not create disk groups
KB 2058424 Virtual SAN Requirements

KB 2073795 Horizon View 5.3.1 on VMware VSAN - Quickstart Guide
KB 2073790 Adding more than 16 hosts to a Virtual SAN cluster
KB 2071379 Virtual SAN node reached threshold of opened components
KB 2071753 Virtual SAN insufficient memory
KB 2074026 Storing ESXi coredump and scratch partitions in Virtual SAN

esxcli vsan command References

VSAN HCL Page

VSAN Ready Nodes

For now the supported Hardware list as follows so verify when its GA

• HP HBA H220i
• HP SMART Array p420i
• Dell PERC H200
• Dell PERC H310
• Dell PERC H710
• LSI 9207-8i
• LSI 9211-8i
• LSI 9240-8i
• LSI 9271-8i
• AHCI controllers (AHCI Driver only)

Certifications/Education :

(BETA) VMware VSAN Deploy and Manage


 Designing VSAN:

About VSAN capacity and VMDK placement
VMware Virtual SAN Scalability Limits - VSAN
STO5027 VMware Virtual SAN Technical Best Practices
Virtual SAN Hardware Guidance Part 1 – Solid State Drives
Virtual SAN Hardware Guidance Part II – Storage Controllers
Virtual SAN – Sizing Considerations
VSAN and Storage Controllers
Virtual SAN & Disk Groups
Three Hosts, two VSAN Datastores
Designing your hardware for Virtual SAN
Virtual SAN and Network IO Control
How VSAN handles a disk or host failure
Virtual SAN and Data Locality/Gravity
Isolation / Partition scenario with VSAN cluster, how is this handled?
VMware vSphere Virtual SAN design considerations…
(Virtual SAN Read IO – cache / buffer / spindles
Pretty pictures Friday, the VSAN edition…
vSphere Metro Storage Cluster using Virtual SAN, can I do it?
How about an All Flash Virtual SAN?
vSphere HA and VMs per Datastore limit!
4 is the minimum number of hosts for VSAN if you ask me

Exploring VSAN Ready Noes per Socket - by Chris Wahl

VSAN Home Labs:


Build vSAN vSphere Lab using PowerCLI
VMware’s VSAN in the Home Lab
VMware VSAN meets EZLAB
My VSAN home lab configuration
Testing vSphere Virtual SAN in your virtual lab with vSphere 5.5
Building Haswell VSAN Cluster by Derek Seaman
Homelab with vSphere 5.5 and VSAN by Erik Bussink
Prefect VSAN Lab by Ryan Birk
VSAN Build Part 1
Homelab Series Status Update By David Savercool
Homelab Thoughts of VSAN by Vladan Seget
My completely ridiculous VSAN Test by Gabrie


VSAN Observer Specific Posts:

Using RVC VSAN Observer Part 1 and 2 by Rawlinson Rivera
VSAN observer showing stauts
VSAN Observer


Other blogs and and media:


VMware Poised To Unleash vSAN Storage Tech, But Pricing Still Unclear
VMware to launch storage solution VSAN this quarter
SimpliVity CEO: We Love VMware And We’re Not Threatened By VSAN
VMware To Begin Selling VSAN Next Week, But Still Isn’t Sharing Pricing
Are Server SANs The Future Or Just Hype?
VMware VSAN creating buzz around hyperconverged infrastructure
New CTO offers guidance on VMware software-defined data center
VMware embiggens VSAN to petabyte scale
VSAN emerges at a WHOPPING 32 nodes and two MEEELION IOPS
VMware wanted to keep its VSAN pricing secret – now it’s all over the web
VMware Announces Broad Ecosystem Support for VMware Virtual SAN
VMware VSAN vs the Simplicity of Hyperconvergence


VSAN Pricing and Current Promotion during BETA period

Sales:

Sales

Reseller


Support :

Global VMware Product Support page


Social Groups / Community:

Linked Group on VSAN

@Captain_VSAN

@PunchingClouds

VMware Community for VSAN


I will try to keep this page updated as and when receives any feedback or RSS Feeds but you can always leaves the Feedback and I will try to update the post at the earliest.

Share and care as VSAN is indeed a big shake in the SDS !!

Thanks