VSAN-60-All-FlashSince the official announcement of VMware Virtual SAN All-Flash architecture, most of the conversations have been focused about the solutions incredible performance capabilities and attributes with regards to IOPS, predictable performance, sub-millisecond latencies. All of those attributes are great and part of the reason as to why Virtual SAN 6.0 as a storage platform and its use cases have been expanded to also focus on business critical applications and large enterprise environments.

I want to turn the spotlight onto one of the many supported use cases for Virtual SAN 6.0 and highlight one of the invaluable capabilities of the new platform with regards to Virtual Desktop Infrastructures (VDI).

Some of the functional requirements for large enterprise infrastructure designs for VDI include the characterization of boot, refresh, and provision times for standard operations and worst case scenarios.

I have seen a fair share of VDI designs and demonstrations of different platforms showcasing bootstorms, refresh and rebuilds times they all do a pretty good job. Now with that said I wanted to take the opportunity to showcase the powerful capabilities of the Virtual SAN 6.0 by demonstrating a bootstorm at the maximum supported capabilities of the platform. This bootstorm demonstration consists of 6401 desktops on a Virtual SAN 6.0 All-Flash 64 node cluster (BigDaddy).
The key and impressive items showcased as part of the demonstration are the following:

  • BigDaddy – 64 Node All-Flash Virtual SAN Cluster
  • Desktops – booting all 6401 desktops in the cluster at once (in batches of 1024 at a time)
  • Boot Time – 24 minutes booting all desktops plus allocation of IP address about 19 minutes for a total of about 40 minutes

This demonstration does not contain tampered or custom configurations of any of the Virtual SAN settings. This is what we generally call an Out-of-the-Box experience. Another important thing to point out here is my definition for completed boot time. What I mean by complete boot, is not just when the desktop is powered on, but when all the desktops have successfully acquired an IP address and are really up and running and ready to be use.

In the interest of time, the demonstration has been sped up from its original length of time to about 5 minutes. Feel free to pay attention to the timestamp as it is displayed in the command line interface to validate the accuracy of the booting time.

This demonstration successfully highlights the one of the many powerful capabilities of Virtual SAN 6.0.

- Enjoy

For future updates on Virtual SAN (VSAN), vSphere Virtual Volumes (VVOLs) and other Software-defined Storage technologies as well as vSphere + OpenStack be sure to follow me on Twitter: @PunchingClouds

VVOLs-LOGO2015In 2011 VMware introduced block based VAAI APIs as part of vSphere 4.1 release. This APIs helped improving performance of VMFS by providing offload of some of the heavy operations to the storage array. In subsequent release, VMware added VAAI APIs for NAS, thin provisioning, and T10 command support for Block VAAI APIs.

Now with Virtual Volumes (VVOLs) VMware is introducing a new virtual machine management and integration framework that exposes virtual disks as the primary unit of data management for storage arrays. This new framework enables array-based operations at the virtual disk level that can be precisely aligned to application boundaries with the capability of providing a policy-based management approach per virtual machine.

The question now is what happens to VAAI APIs (NAS and Block) and how will virtual volumes co-exist together?. With Virtual Volumes, aside from the data path, the ESX hosts also control of the connection path to the storage arrays. The Vendor Provider typically arranges the path to the storage arrays. In this case, virtual volumes can be considered as a richer extension of the VAAI NAS APIs. On July of las year I published an article in which I discussed the interoperability between VAAI and VVOLs during cloning operations in deferent scenarios “Virtual Volumes (VVols) vSphere APIs & Cloning Operation Scenarios” consider having another look. Now let’s go over a set of interaction scenarios between the VAAI APIs and Virtual Volumes.

VAAI vs VVOLsRev2

VAAI Block and VVOLs:

VAAI Block defines basic SCSI primitives, which allows vSphere (primarily VMFS) to offload pieces of its operations to the array. There is still a heavy dependency on VMFS playing the role of an orchestrator and sending individual VAAI Block command to the storage array.

With VVOLs, the storage array systems are aware of virtual machine’s disk and hence they can efficiently perform operations such as snapshots, clones, and zeroing operations on the virtual machines disks. But still VAAI Block and thin-provisioning primitives co-exists with VVOLs.

  • ATS – All config VVOLs objects that are stored in a Block VVOLs datastore are formatted with VMFS and hence require supporting ATS commands. This support is detected based on ATS support for a PE LUN to which VVOLs are bound.
  • XCOPY – With VVOLs, ESX will always try to use array based VVOLs copy mechanism defined using copyDiffsToVirtualVolume or CloneVirtualVolume primitives. If these are not supported, it will fall back to software copy. Since software copy involves copying data between  Protocol Endpoint (PE) LUNs and VMFS LUN, there is still potential to use XCOPY command during software data copy. When falling back to software copy, vSphere will use the XCOPY command when moving a virtual machine from a VMFS datastore to VVOLs datastore or between two VVOLs datastores. In the first release, vSphere will not try to use XCOPY if the virtual machine is moving from VVOLs datastore to VMFS datastore. vSphere will detect the support for XCOPY for individual VVOLs based on the support of VAAI XCOPY on PE LUN to which it is bound.
  • Block Zeroing – Main purpose of this primitive is to initialize thick disks provisioned on VMFS datastores. So this primitive is not used for VVOLs as in VVOL, VMFS on config VVOLs only contains small descriptor files. Main purpose of this primitive is to initialize thick disks provisioned on VMFS datastores. With VVols VM provisioning type is specified as part of profile information passed during VVOL creation. Config VVol which are formatted VMFS are “thin” by definition. Also size of config VVol is very small (default 4GB) and it contains small files such as disk descriptors, vm config files, stats and logs data. So Block Zeroing primitive is not used for VVOLs by vSphere.

VAAI NAS and VVOLs:

Unlike SCSI, NFSv3 is a frozen protocol, which means all features of VAAI NAS came via private RPCs issued by vendor plugins. VVOLs extends this model of communicating outside basic protocol. VVOLs defines the rich set of VASA APIs to allow offload of most of the vSphere operations. With vSphere 6.0, existing VAAI NAS will continue work but VVOL datastores will offer richer and faster experience than VAAI NAS. Also, VVOLs doesn’t need any vendor specific plugin installation. Another noteworthy point regarding NAS VAAI and storage vMotion is that NAS VAAI snapshots cannot be migrated, when an attempt is made to migrate a virtual machine with NAS VAAI “snapshots”, the snapshot hierarchy is collapsed and all snapshot history is lost.  This is not the case with VVOLs, and further we can translate snapshot hierarchies between NFS (Non-VAAI)/VMFS/VSAN/VVol (any source->target combination of the 4).

VAAI Thin-Provisioning and VVOLs:

  • Soft Threshold Warnings – similar to a VMFS datastore with TP support, Soft threshold warning for any VVOL virtual machine’s I/O, will be seen in vCenter. Also the corresponding container gets flagged appropriately. The container gets the yellow warning icon when soft threshold warning is issued. Essentially this could be potentially confusing for vSphere admin as this warning is virtual machine specific and warning message doesn’t provide the details on which virtual machine has the problem. This will be corrected in a future update.
  • Hard Threshold Warnings – Hard threshold warning behavior is similar to that on VMFS datastore.
    When VVOL virtual machine’s I/O gets a hard threshold warning, it will stun the corresponding virtual machine. Administrator can resume the virtual machine after provisioning more space or can completely stop the virtual machine.
  • UNMAP – Since there are no disks managed by VMFS, vSphere will not be actively using the UNMAP primitive. vSphere will not actively used UNMAP primitive. Although it will pass through UNMAP to backing VVOLs when guest issues it. Although it will issue UNMAP when guest issues it like XCOPY and ATS, vSphere will detect support for UNMAP for individual VVOLs based on the support of VAAI UNMAP on a Protocol Endpoint LUN to which it is bound. Another thing is that vSphere will not enforce any of the alignment criteria when UNMAP is issued by the guest. This behavior is very similar to the one found with an RDM LUN. With VVOLs UNMAP commands going from the guest directly to the storage array the same way we send all I/O, and the array will now finally see all the individual UNMAP commands guest operating systems issues. For example, a Windows Server 2012 will immediately become a source of UNMAP commands.  On the other hand for the Linux, the filesystem checks the SCSI version supported by the virtual device and won’t issue an UNMAP with the current level of SCSI support we present (SCSI-2). That is something that will be addressed in a future release.

Now let’s identify the supported operations and behavior for the different primitives.

Primitives Supported Operations and Behavior

Powered On Storage vMotion without snapshots

For a powered on VM without snapshots, the Storage vMotion driver coordinates the copy. The Storage vMotion driver will use the data mover to move sections of the current running point. The data mover will employ “host orchestrated hardware offloads” when it can…

Block VAAI & Block VVOLs:
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– XCOPY will be used to migrate (host orchestrated offload)

NAS VAAI
- No optimizations

NAS VVOL
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)

Powered On Storage vMotion with snapshots

For a powered on VM with snapshots, the migration of snapshots is done first, then the Storage vMotion driver will use the data mover to move the current running point.

Block VAAI
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– XCOPY will be used to migrate snapshots + current running point

Block VVOL
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– The cloneVirtualVolume and copyDiffsToVirtualVolume VASA APIs will be used to migrate all snapshots (Full hardware offload)
– XCOPY will be used to migrate the current running point (host orchestrated offload)

NAS VAAI
- NAS VAAI cannot migrate snapshots
– No further optimization

NAS VVOL
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– The cloneVirtualVolume and copyDiffsToVirtualVolume VASA APIs will be used to migrate all snapshots (Full hardware offload)

Powered off Storage vMotion without snapshots
For a powered off VM, the Storage vMotion driver is not in the picture. So, effectively a Storage vMotion of a powered off VM is a logical move (Clone + Delete Source).

Block VAAI
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– XCOPY will be used to migrate current running point

Block VVOL
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– The cloneVirtualVolume VASA API will be used to migrate the current running point (Full hardware offload)
– The copyDiffsToVirtualVolume VASA APIs will be used to migrate all snapshots (Full hardware offload)

NAS VAAI
- NAS VAAI clone offload will be employed to migrate the current running point

NAS VVOL (Same as block VVOL)
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– The cloneVirtualVolume VASA API will be used to migrate the current running point (Full hardware offload)
– The copyDiffsToVirtualVolume VASA APIs will be used to migrate all snapshots (Full hardware offload)

Powered off Storage vMotion with snapshots
- Same general idea as above, just with snapshots too…

Block VAAI
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– XCOPY will be used to migrate current running point + snapshots

Block VVOL
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– The cloneVirtualVolume VASA API will be used to migrate the current running point + snapshots (Full hardware offload)

NAS VAAI
- NAS VAAI cannot migrate snapshots
– NAS VAAI clone offload will be employed to migrate the current running point

NAS VVol (Same as block VVOL)
- Bitmap APIs will be used to determine only the relevant blocks to migrate (Space efficiency optimization)
– The cloneVirtualVolume VASA API will be used to migrate the current running point + snapshots (Full hardware offload)

– Enjoy

For future updates on Virtual SAN (VSAN), vSphere Virtual Volumes (VVOLs) and other Software-defined Storage technologies as well as vSphere + OpenStack be sure to follow me on Twitter: @PunchingClouds

VSAN-ALL-FLASH-LOGOOh well now that the cat is officially out of the bag, as they say!. Everyone in the world should now be aware of the fact that VMware Virtual SAN 6.0 supports an all-flash architecture. I think it’s time to discuss a couple of items with regards to a new architecture.

The Virtual SAN 6.0 All-Flash architecture uses flash-based devices for both caching and persistent storage.

In this architecture, the flash cache is used completely as write buffer. This all-flash architecture introduces a two-tier model of flash devices:

All-Flash-Arch

  • write-intensive, high endurance caching tier for the writes
  • read-intensive, cost-effective flash device tier for data persistence

The new device tiering model not only deliver incredible performance results, but it can also potentially introduce cost savings for the Virtual SAN 6.0 all-flash architecture depending on the design and hardware configuration of the solution.

Virtual SAN Configuration Requirements

In order to configure Virtual SAN 6.0 for the all-flash architecture, the flash devices need to be appropriately identified within the system. In Virtual SAN, flash devices are identified and categorized for the caching tier by default. In order to successfully enabled the all-flash architecture configuration we need manually to flag the flash devices that will be utilized for data persistence or capacity. This configuration is performed via one of the supported command line interface tools such as RVC or ESXCLI.

RVC handles the configuration of the devices at a cluster level. Below you’ll find an image, which illustrates the usable syntax for flagging the flash devices with RVC.

RVC

ESXCLI handles it at the per-host level. Below you’ll find an image, which illustrates the usable syntax for flagging the flash devices with esxcli.

RVC

Another command line utility that is worth knowing is the VSAN Disk Query (vdq). This utility allows users to identify when the flash devices are configured for used in the capacity tier as well as if they are eligible to be use by Virtual SAN.
Whenever vdq is used to query the flash devices on a host as illustrated below, the output will display a new property called “IsCapacityFlash”. This property specifies whether a flash device will be utilized for the capacity tier instead of the caching tier.

all-flash-vsan-6-vdq

For more in-depth information on the use of vdq, please take a look at a post by one of VMware’s elite engineers and VSAN Champion and my boy William Lam. #WreckingCrew

It’s important to highlight that flagging flash devices to be used for capacity cannot be performed from the option available in the vSphere Web Client UI. It has to be performed via the CLI. (wait for it….wait for it)

Once the flash devices, they will be displayed as magnetic devices (HDD) in the disk management section of the Virtual SAN management tab.

That’s about it, after the flash devices have been properly tagged, the rest of the Virtual SAN configuration procedure is as easy as it was in the previous version.

So in the spirit of making things easy and reduce any contention with getting into the CLI and manually flagging every disk. I’ve been able to design a tool along with my good pal and now a VSAN Champion Brian Graf that should take care of disk tagging process for just about everyone.

Here is a demo of how simple it is to configure a Virtual SAN 6.0 all-flash cluster with a teaser of the Virtual SAN All-Flash Configuration Utility. Oh yeah, I almost forgot to mention….. It’s a 64 node all-flash cluster (The BigDaddy).

- Enjoy

For future updates on Virtual SAN (VSAN), vSphere Virtual Volumes (VVOLs) and other Software-defined Storage technologies as well as vSphere + OpenStack be sure to follow me on Twitter: @PunchingClouds

VSAN-60It is with great pleasure and joy that I like to announce the official launch of VMware Virtual SAN 6.0, one of VMware’s most innovative software-defined storage products and the best hypervisor-converged storage platform for virtual machines. Virtual SAN 6.0 delivers a vast variety of enhancements, new features to the as well as performance and scalability improvements.

Virtual SAN 6.0 introduces support for an all-flash architecture specially designed to provide virtualized applications high performance with predictably low latencies. Now with support for both hybrid and all-flash architectures Virtual SAN 6.0 is ready to meet the performance demands of just about any virtualized application by delivering consistent performance with sub-millisecond latencies.

 

Hybrid Architecture

  • In the hybrid architecture, server-attached magnetic disks are pooled to create a distributed shared datastore that persists the data. In this type of architecture, you can get up to 40K IOPS per server host.

All-Flash Architecture

  • In All-Flash architecture, the flash-based caching tier is intelligently used as a write-buffer only while another set of flash devices forms the persistence tier to store data. Since this architecture utilizes only flash devices, it delivers extremely high IOPs of up to 90K per host, with predictable low latencies.

VSAN-Archs

Virtual SAN 6.0 delivers true enterprise-level scale and performance by doubling the scalability of Virtual SAN 5.5 by scaling up to 64 nodes per cluster for both hybrid and all-flash configurations. In addition, Virtual SAN 6.0 improves the number of virtual machines per host up to 200 for both supported architectures.

VSAN-Scale

The performance enhancements delivered in Virtual SAN 6.0 are partially due to the new Virtual SAN on-disk Filesystem (VSAN FS). The new version delivers a new VMDK delta file (vsanSparse) takes advantage of the new on-disk format writing and extended caching capabilities to deliver efficient performance. This results in the delivery of performance-based snapshots, and clone that are comparable to SAN snapshots.

Virtual SAN 6.0 now enables intelligent placement of virtual machine objects across server racks for enhanced application availability even in case of complete rack failures. Virtual SAN Fault Domains provide the ability to group multiple hosts within a cluster to define failure domains to ensure replicas of virtual machines data is spread across the defined failure domains (racks).

VSAN-FD

Along with all the new added features a significant amount of improvements have been added to enhance the management user experience:

  • Disk/Disk Group Evacuation – Introduce ability to evacuate data from individual disk/disk groups before removing a disk/disk group from the Virtual SAN.
  • Disk Serviceability features - easily map the location of magnetic disks and flash devices. Ability light disk LED on failures, Turn disk LED on/off from the vSphere Web Client.
  • Storage Consumption Models – adds functionality to visualize Virtual SAN 6.0 datastore resource utilization when a VM Storage Policy is created or edited.
  • UI Resynchronization Dashboard – the vSphere Web Client UI displays virtual machine objects resynchronization status and remaining bytes to sync.
  • Proactive Rebalance – provides the ability to manually trigger a rebalance operation in order to utilize newly added cluster storage capacity.
  • Health Services – deliver troubleshooting and health reports to vSphere Administrators about Virtual SAN 6.0 subsystems and their dependencies such as cluster, network, data, limits, physical disk .

VSAN-health

With all the major enhancements and features of this release, Virtual SAN is now enterprise-ready, and customers can use it for a broad range of use cases, including business-critical and tier-1 production applications.  Stay tune, there is a lot more to come from the world’s greatest software-defined storage platform. For more information visit the Virtual SAN product page.

– Enjoy

For future updates on Virtual SAN (VSAN), vSphere Virtual Volumes (VVOLs) and other Software-defined Storage technologies as well as vSphere + OpenStack be sure to follow me on Twitter: @PunchingClouds

VVOLs-LogoToday VMware announced the release of vSphere 6.0 and with this announcement comes the official release of vSphere Virtual Volumes. vSphere Virtual Volumes (VVOLs) is VMware’s new management & integration framework designed to deliver a more efficient operational model for external storage.

vSphere Virtual Volumes implements the core tenants of the VMware SDS vision to enable a fundamentally more efficient operational model for external storage in virtualized environments, centering on the application instead of the physical infrastructure.

vSphere Virtual Volumes allows application-specific requirements to drive storage provisioning decisions while leveraging the rich set of capabilities provided by existing storage arrays. The value outcome of vSphere Virtual Volumes is threefold:

  • vSphere Virtual Volumes simplifies storage operations by automating manual tasks and eliminating operational dependencies between the vSphere Administrator and the Storage Administrator that only add complexity. Provisioning is faster, and change management is simpler as the new operational model is built upon policy-driven automation.
  • vSphere Virtual Volumes simplifies the delivery of storage service levels to applications by providing administrators with finer control of storage resources and data services at the VM level that can be dynamically adjusted in real time.
  • vSphere Virtual Volumes improves resource utilization by enabling more flexible consumption of storage resources, when needed and with greater granularity. The precise use of storage resources eliminates overprovisioning.

vSphere Virtual Volumes virtualize SAN and NAS devices into logical pools of capacity, called Virtual Datastore. vSphere Virtual Volumes represents virtual disks natively on the underlying physical storage. Virtual disks become the primary unit of data management at the array level without the need of the VMFS filesystem.

vSphere Virtual Volumes VMware offers a new paradigm, one in which an individual virtual machine and its drives become the unit of storage management, rather than the traditional LUN.

vSphere Virtual Volumes is the implementation of the virtual data plane for external storage in the VMware SDS model. vSphere Virtual Volumes is composed by two key implementations:

Flexible consumption at the logical level

vSphere Virtual Volumes virtualizes SAN and NAS devices by abstracting physical hardware resources into logical pools of capacity (called Virtual Datastore) that can be more flexibly consumed and configured to span on or more storage arrays. The Virtual Datastore defines capacity boundaries, access logic, and exposes a set of data services accessible to the VMs provisioned in the pool. Virtual Datastores are purely logical constructs that can be configured on the fly, when needed, without disruption and don’t require to be formatted with a file system.

Finer control at the VM level

vSphere Virtual Volumes defines a new virtual disk container (Virtual Volume) that is independent of the underlying physical storage representation (LUN, file system, object, etc.). In other terms, with Virtual Volumes the virtual disk becomes the primary unit of data management at the array level. This turns the Virtual Datastore into a VM-centric pool of capacity. It becomes possible to execute storage operations with VM granularity and to provision native array-based data services to individual VMs. This allows admins to provide the right storage service levels to each individual VM.

To enable efficient storage operations at scale, even when managing thousands of VMs, Virtual Volumes uses vSphere Storage Policy-Based Management (SPBM). SPBM is the implementation of the policy-driven control plane in the VMware SDS model.

Efficient operations through automation

SPBM allows capturing storage service levels requirements (capacity, performance, availability, etc.) in the form of logical templates (policies) to which VMs are associated. SPBM automates VM placement by identifying available datastores that meet policy requirements and coupled with Virtual Volumes, it dynamically instantiates necessary data services. Through policy enforcement, SPBM also automates service level monitoring and compliance throughout the lifecycle of the VM.

VVOLs-Arch

Stay tuned, there is a lot more to come from the world’s greatest software-defined storage platform. For more information visit the vSphere Virtual Volumes product page.

– Enjoy

For future updates on Virtual SAN (VSAN), vSphere Virtual Volumes (VVOLs) and other Software-defined Storage technologies as well as vSphere + OpenStack be sure to follow me on Twitter: @PunchingClouds