Replacing VMware vSAN Two-Node Witness Host Checklist

One of the things I love about VMware vSAN is the ease of which you can do things operationally and the resiliency and robustness of the platform to have changes in the infrastructure and still keep running your workloads. The two-node vSAN cluster operates with a Witness appliance that runs to synchronize metadata for node majority purposes. Switching out the vSAN Witness appliance is actually a very non-eventful task for the most part. You can easily provision a new appliance, and let the repair objects timer kick in or force a synchronization of the objects to get back to a good state with a new vSAN Witness appliance. However, there are many considerations you need to make in an active vSAN two-node cluster than simply changing out the appliance. You need to remember the configuration and other settings don’t simply transfer over when you change out the appliance. Let’s take a look at Replacing VMware vSAN Two-Node vSAN witness host checklist to go over a few considerations.

Replacing VMware vSAN Witness Host Considerations

It is good to look at a Replacing VMware vSAN Two-Node Witness Host Checklist when you change out a vSAN Witness appliance. It is easy to forget when you change out your appliance, you are going back to an appliance that is not configured specifically for your environment and may also have skew with versions and other factors. Let’s look at the following:

• Check vSAN Witness Appliance network configuration (port groups, IP address, and jumbo frames
• Ensure the new ESXi host appliance is running the same version of ESXi as the physical hosts in your stretched cluster
• Ensure the on-disk format version is the same as the other physical hosts which may have been upgraded
• Synchronize VMware vSAN objects to the new vSAN Witness host

Check vSAN Witness Appliance Network Configuration

In VMware vSAN and most other HCI solutions with software-defined storage, the network is critically important. If it has issues, then the storage solution will have issues. The same is true with vSAN. When you swap out the vSAN appliance the network is going to be configured on the newly deployed appliance with all the defaults. There are three points of configuration that I want to emphasize here for checking when you deploy a new appliance as they are typically some of the most common network oversights when deploying a vSAN witness host.

• Make sure the virtual NICs of the deployed vSAN appliance which is a nested ESXi host are connected to the right portgroups on the physical ESXi management host that is housing the appliance. These portgroups need to be plumbed into the vSAN network of the two physical 2-node ESXi hosts.
• Assign an IP address to the VMkernel port of the VMware vSAN Witness host for vSAN communication
• Make sure if you are using jumbo frames on your physical hosts, that you make the adjustment to jumbo frames on the vSAN witness node, both in the virtual switch settings and at the VMkernel port level.

Below is the vSANWitness host deployed as a VM running in the management cluster. You want to make sure you are connecting the vSANWitness to the correct port groups on the physical ESXi host. Network adapter 1 is the management interface and Network adapter 2 is the vSAN interface.

If you are not running DHCP on your vSAN network which most likely you are not, you will want to assign an IP address to the VMkernel port of the witnessPg.

Configure jumbo frames on the witnessSwitch itself.

Configure jumbo frames on the vmk1 VMkernel port as well hosting vSAN.

Verify the vSAN ESXi Host Versions are Compatible

You always want to make sure you are running the same version of ESXi across your vSAN hosts, including the vSAN Witness host. If you have mismatched host, including on the witness appliance, you will see the following in your vSAN Health monitor: Hosts need software upgrade.

Take note of the version of vSphere you are running on your ESXi hosts and upgrade your Witness host to the same version as your physical vSAN hosts in the 2-node cluster.

Verify On-Disk Format Version and Upgrade if Needed

Generally, with each new release of VMware vSAN there has been an upgrade available for the on-disk format. The latest VMware vSphere 6.7 Update 3 has on-disk format version 10. As your physical vSAN hosts are upgraded, you have most likely upgraded your on-disk format version. When you deploy a new vSAN Witness host, you will need to make sure you upgrade the on-disk format for the Witness host disks.

Synchronize VMware vSAN Objects to the New vSAN Witness Host

Once you have the new vSAN Witness host in place and have the vSAN network lined out along with communication with the other vSAN hosts in the cluster, you should be ready for objects to synchronize. After you bring the new vSAN witness host online and perform the change witness host process, you should start to see objects start to synchronize. You can also force a synchronization of your objects to the new Witness host.

The change witness host process can be initiated under the <Your Cluster> > Configure > vSAN > Fault Domains menu.

You will see objects that need to be synchronized under the <Your Cluster> Monitor > vSAN > Resyncing Objects.

Wrapping Up

Hopefully this Replacing VMware vSAN Two-Node Witness Host Checklist has been helpful if you are needing to replace your vSAN Witness host appliance and not sure what to expect or what gotchas you might run into. Generally it is down to the list of things that we have covered here that can cause issues when switching over to a new witness host. If you pay attention to the details though, VMware has really made this process painless and resilient.