Replace or Change VMware vSAN Witness Host with vSphere 6.7 Appliance

VMware vSan is getting more and more powerful with each and every release. With the introduction of vSphere 6.7 and vSAN 6.7, the HTML5 UI has received more functionality, Windows Failover Clustering is now supported, and all new redeisinged adaptive resync feature has been added, 4k Native Drive Support, and overall performance improvements have been brought to this release. So, it is certain a worthy upgrade from previous versions of VMware vSAN. In the home lab recently, I went through the “production” ugprade from running vSAN 6.6.1 to running VMware vSAN 6.7. This of course followed suit with upgrading my VCSA 6.5 U1g appliance up to VCSA 6.7 a and then upgrading my ESXi hosts accordingly as well. How did I go about “upgrading” the vSAN Witness node in my two-node vSAN stretched configuration? Let’s take a look at how to replace or change VMware vSAN witness host with vSphere 6.7 appliance.

Replace or Change VMware vSAN Witness Host with vSphere 6.7 Appliance

I had a special use case to replace my witness node with a different appliance as I was running the vSAN witness node on a physical Windows workstation inside of VMware Workstation with the networking plumbed into the vSAN network. The physical workstation in my lab was very long in the tooth and with vSphere 6.7 had a processor that is no longer supported for installation or upgrade. I decided to rearchitect the home lab and bring in another full blown ESXi host to house the vSAN appliance which opened up some more powerful capabilities than I had with the old workstation running VMware Workstation With VMware vSAN there is a provision to “replace” the vSAN Witness node in the stretched architecture.

As you know the two-node vSAN cluster is basically a special “stretched” configuration where you have two ESXi hosts that house the virtual machine data – the original copy and the replica that is created. With vSAN, a storage object that is created such as a VMDK needs more than half of its components to be accessible for the virtual machine to function. In a two-node configuration, if one host is down, you lose half of your components and so the virtual machine will be taken offline.

VMware has provided a vSAN Witness appliance that is essentially an OVA appliance download of a special ESXi installation that contains an embedded license key. This means the solution is free to use in the two-node installation – pretty cool!

So, now on to the process at hand – replacing the appliance.  For my use case, it was extremely simple and easy to replace the vSAN Witness appliance as the witness appliance simply contains special meta data objects that complete the picture of the storage components of a virtual machine, thus providing the “more than half” objects requirement.  The process involves the following steps:

• Deploy the new VMware vSAN 6.7 Witness Appliance
• Provision the networking of the appliance correctly to communicate with your vSAN network
• Run the Change Witness Host wizard
• Synchronize the vSAN objects with the new vSAN Witness Host

Proceeding with the below, I have already deployed the new VMware vSAN Witness Host as a nested VM running on a new ESXi 6.7 host.  I have added the vSAN Witness host into vCenter.  The next step that needs to happen is to change the host.

To change the vSAN Witness host, click on the cluster in vCenter, then navigate to vSAN >> Fault Domains and click Change in the stretched cluster configuration.

This launches the Select witness host wizard.  This wizard is a simple three step wizard that allows choosing the new witness host, selecting the disks that are to be used in disk groups and completing the operation.

If you select a witness host that is already part of another vSAN stretched cluster, you will see the error “The selected host is an external witness for another vSAN Stretched cluster“.

Next, through the wizard, select the new host, and disks you want to use from the new host and the process will complete fairly quickly to change out the witness host.

Synchronizing VMware vSAN Objects after Changing the Witness Host

The next really important step that should start to happen automatically is the synchronization of vSAN objects to the newly replaced vSAN Witness host.  You can monitor the progress of this operation as well or force it to take place if it isn’t already.  Navigate to Monitor >> vSAN >> Health and you will see the Data and vSAN object health in a failed state.  Note the reduced availability with no rebuild count.

You can choose to Repair Objects Immediately to force the synchronization of objects.  Note the number of reduced availability with no rebuild number is decreasing.

Still decreasing…

After a few minutes, the vSAN objects are fully synchronized after changing out the vSAN Witness host in my vSAN 6.7 cluster.

Takeaways

The two-node VMware vSAN stretched cluster is a great, cost-effective way to implement software defined storage into production environments and is well-suited for ROBO deployments where you don’t have a lot of infrastructure.  Additionally, it is a great way to see and play around with the benefits of VMware vSAN in a home lab environment, seeing enterprise functionality without the need for a physical SAN.  It is certainly the way forward for future storage architecture.  VMware has provided an extremely easy way to control and even replace the vSAN Witness host in these stretched cluster configurations.  In my case, this provided an extremely easy way to “upgrade” my vSAN 6.6.1 environment running on top of ESXi 6.5 and one I had the vSAN hosts themselves upgraded to 6.7, I simply “changed” the vSAN Witness Host out and allowed the vSAN data objects to synchronize.  This allows for really great flexibility in deployment and upgrade options.