vStorage API for Array Integration (VAAI) is an API framework developed by VMware that enables certain storage tasks to be offloaded from Esxi hosts to the storage array and thus lessens the processing workload on the Esxi host. VAAI was first introduced with vSphere 4.1.

It significantly improves the performance of storage -intensive operations such as cloning storage, block zeroing , and so on and thus reduces the overhead on the ESXi host. The main goal of VAAI is to help storage vendors to provide hardware assistance to speed up VMware I/O operations.

The APIs create a separation of duty between the hypervisor and its storage devices, enabling each to focus on what it does best: virtualization-related tasks for the hypervisor and storage-related tasks for the storage arrays.

What vSphere functions can be offloaded with VAAI?

1: Copy offload: Certain operations such as deploying a new VM from template or cloning a VM etc involves copying of virtual disk files. These operations can be offloaded to storage array with hardware-accelerated capability which results in these operation getting completed quickly rather than file-level copy operations at the host level. Also hardware offload can be used for Storage vMotion for moving disk around datastores quickly.  This can be very beneficial to an environment where VMs are provisioned on a frequent basis or when many VMs need to be created at one time.

2: Write Same/Block Zero:  Before any any data can be written to block of a virtual disk, it needs to be “zeroed” first. Lazy zeroed virtual disks do not zero each disk block until it is written to for the first time and causes a slight performance penalty when write operation happens. To avoid performance penalty that occurs on first write to a disk block, Eager-zeroed virtual disks can be used.

The formatting process (zeroing disk blocks ) can be both a time-consuming and resource-intensive process as Esxi hosts sends gigabytes of zeros to the array.  Array’s with VAAI capabilities can handle the process of zeroing all of the disk blocks much more efficiently. Instead of having the host wait for the operation to complete, the array simply signals that the operation has completed right away and handles the process on its own without involving the host.

3: Hardware-assisted locking: In my previous post  we discussed about VMFS locking mechanism and had a look on when ATS Only or ATS + SCSI Reservation locking mechanism is used.

When “SCSI reservations” locking technique is used, it locks an entire LUN using the RESERVE SCSI command when storage operations, such as incremental snapshot growth, metadata updation etc occur. This helps to avoid corruption but can delay storage tasks from completing as hosts have to wait for the LUN to be unlocked with the RELEASE SCSI command before they can write to it.

Atomic Test and Set (ATS) is a hardware-assisted locking method that offloads the locking mechanism to the storage array, which can lock at individual disk blocks instead of the entire LUN. This allows the rest of the LUN to continue to be accessed while the lock occurs, helping to avoid performance degradation.

VAAI Enhancements in vSphere 5.x

In vSphere 5, vStorage APIs for Array Integration were enhanced. The new functionality included thin provisioning of storage systems and expands support to network-attached storage (NAS) devices.

Thin provisioning of storage array allows to reclaim blocks of space when a virtual disks is deleted or migrated from one datastore to other, to mitigate the risk of a thinly provisioned storage array running out of space. VAAI also supports ‘Thin Provisioning Stun’ which temporarily pause virtual machines when space on datastore runs out and thus giving vSphere admins time to add storage or migrate the virtual machine to a different array.

Prerequisites for using VAAI

• ESXi 4.1 or later.
• vSphere Standard license for Esxi hosts. Prior to vSphere 6.0, VAAI was supported in only Enterprise and Enterprise plus version.
• Storage arrays that support VAAI storage-based hardware acceleration.

Limitations of VAAI: VAAI can not be used when:

• The source and destination VMFS volumes have different block sizes
• The source file type is RDM and the destination file type is non-RDM (regular file)
• The source VMDK type is eager zeroed thick and the destination VMDK type is thin
• The source or destination VMDK is any kind of sparse or hosted format.
• Cloning a virtual machine that has snapshots because this process involves consolidating the snapshots into the virtual disks of the target virtual machine.
• The VMFS datastore has multiple LUNs/extents spread across different arrays
• The logical address and/or transfer length in the requested operation are not aligned to the minimum alignment required by the storage device (all datastores created with the vSphere Client are aligned automatically)

How to verify if VAAI is supported by your datastore

To verify which datastores in your environment is VAAI enabled, run command: esxcli storage core device vaai status get. With this command you will get the a detailed status for all VAAI primitives.

You can verify the VAAI status from Web Client as well by navigating to Esxi host > Manage > Storage > Storage Devices. 

The status values are Unknown, Supported, and Not Supported. The initial value is Unknown. The status changes to Supported after the host successfully performs the offload basic operations. If the offload operation fails, the status changes to Not Supported.

In my lab none of my datastore have support for VAAI.

How to Enable/Disable VAAI

Login to vSphere Web Client and navigate to Host and Cluster view and select an Esxi host > Manage > Advanced System Settings and look for these options:

• DataMover.HardwareAcceleratedMove : Full Copy
• DataMover.HardwareAcceleratedInit: Zero Blocks
• MFS3.HardwareAcceleratedLocking : ATS

If the value is displayed as 1, it means its enabled and if the value is 0 it means its disabled If VAAI is enabled on Esxi host and storage devices do not support or provide only partial support for the host operations, the host reverts to its native methods to perform the unsupported operations.

Checking VAAI primitives from CLI

We can enable all VAAI operations on per device level as well using below command

Listing VAAI Plugins and Filters

VAAI Claimrules

Every block storage device that is managed by VAAI needs two storage claim rules. It needs one that specifies the hardware acceleration filter and it needs one that specifies the hardware acceleration plugin for the device.

list the filter claim rules:

list the plugin claim rules:

To configure hardware acceleration for a new array you need to add filter and plugin claimrules. You then need to load the rules into runtime.

Create Claim Rules

Load claim rules into the system:

We can prevent a particular storage array from using VAAI by using the VMW_VAAIP_MASK VAAI plugin rule.

Additional Reading

VAAI FAQ

ESXCLI commands for VAAI operations

Things you should know about VAAI and Licensing

vSphere NFS and VAAI monitoring

VAAI Compatibility and Displaying VAAI Statistics

A Complete List of VMware VAAI Primitives

I hope you find this post informational. Feel free to share this on social media if it is worth sharing. Be sociable 🙂