VMware LSI SAS vs PVSCSI vs NVMe Controller Performance

With all of the different variations of virtual controllers now available in VMware, I wanted to perform a simple test of disk performance benchmarks when using all three controllers on the same virtual machine to see if the controllers performed as expected and if there is actually performance improvements in using the new NVMe controller vs the PVSCSI or ParaVirtual SCSI controller with a virtual machine that is truly backed with an NVMe drive.  For my testing I have a virtual machine that resides on a Samsung EVO 960 NVMe drive and simply switched controllers on the virtual machine to perform each round of testing.  I found some very interesting results with the new NVMe controller in VMware.  Please indulge in my quick comparison of VMware LSI SAS vs PVSCSI vs NVMe Controller Performance.

VMware LSI SAS vs PVSCSI vs NVMe Controller Performance

To begin with, I want to detail a few things about the test environment, virtual machine, etc.

VMware:

• Version 6.5.0, Releasebuild-5310538, Update 0, Patch 19

Hardware:

• SuperMicro, model SYS-5028D-TN4T
• Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 1 proc, 8 cores
• 64 GB of memory
• Samsung EVO 960 (non pro)

Virtual Machine

• Windows 2012 R2 Datacenter
  • patched, no AV, VMware Tools installed and up to date
• 4 vCPUs, 8 gigs of RAM
• Hardware version 13, VMware 6.5

Disk Benchmark:

• ATTO Disk Benchmark v3.05
  • Using transfer size 512 B to 64 MB, total length 256 MB, Direct I/O, queue depth 4

VMware LSI SAS controller performance

The first controller performance tested with ATTO was the generic LSI SAS controller which is the default controller configured with Windows 2012 R2.  What kind of performance with ATTO running in the VM did I see with the LSI Logic SAS configured?  Below are the results from the ATTO Disk Benchmark.

VMware PVSCSI controller performance

The next test was to change out the LSI SAS controller for the ParaVirtual PVSCSI controller.  The virtual machine was powered down, controller changed, booted back up, and allowed to login and sit for a couple of minutes to make sure no processes would be competing for disk performance.  Below are the results from the run.

VMware NVMe controller performance

Now, for the new NVMe controller and performance.  Again, the virtual machine was powered down, disk controller was changed, booted back up, and allowed to settle down for a couple of minutes before the run.

Comparison between LSI SAS, PVSCSI, and NVMe

After running all three tests with the different controllers, I had some very interesting results when comparing between the three.  I have highlighted the “winner” for each category in green.  The PVSCSI seemed to shine with smaller file sizes.  However, the surprise here for me was the NVMe controller was behind the LSI SAS and PVSCSI in most categories in my single test comparison using the same virtual machine.

This wasn’t a very long or detailed test and I didn’t put each virtual controller type through any other battery of tests.  If someone has different results or sees an error in the above testing methodology, please correct me!  However, with all else being the same, these were the results I received by simply changing the virtual controller for the same VM.

Thoughts

The VMware LSI SAS vs PVSCSI vs NVMe Controller Performance test had some interesting results in my tiny lab environment.  I had the presumption going in that the PVSCSI would have superior performance to the LSI SAS and the NVMe would perform slightly better than the PVSCSI.  In reading through the VMware documentation the PVSCSI adapter provides 8% better throughput at 10% lower CPU cost.  PVSCSI adapters are capable of providing better throughput at a lower CPU cost than LSI Logic SAS adapters at extreme load conditions.  I will leave the results up to you to determine.  The above testing did not take into consideration CPU cost as this would be an interesting metric as well that I may take into consideration in a future test.  It will be interesting as well for VMware to update some of the documentation comparing the various virtual controllers from their perspective.  I will say that on VMs that I have backed with the NVMe controller in my home lab, the VMs “feel” like they perform better.  However, this isn’t a very scientific way of measuring performance as the data above is way more tangible.