Create a VMware vSAN 6.7 U1 Nested ESXi Lab

For those who may not be running VMware vSAN in a home lab using physical hosts, or who are running VMware vSAN on physical hosts for “production” but who want a separate “learning” or “play around” environment for VMware vSAN, running a nested vSAN deployment in the home lab may be a great option. I currently run both in my home lab environment, both as the primary storage for my production home lab environment and also in a nested environment configuration. I wanted to step through a few considerations for running VMware vSAN in a nested configuration as well as step through how to create a VMware vSAN 6.7 U1 nested ESXi lab for learning and playing around with various vSAN functionality.

Why Nested VMware vSAN Lab?

Some might wonder, why would you create a nested VMware vSAN lab. Well, the answer is similar to why you create any nested virtualization lab. Virtualization labs that run as virtual machines that are nested inside a parent hypervisor allow for an extremely easy way to provision lab environments and affords all the benefits that come with virtual machines. This includes the ability to snapshot, backup, clone, and easily provision and tear down nested virtualization labs.

With today’s high performance NVMe drives that are extremely low cost, now more than ever before, having an extremely performant storage infrastructure layer on which to run a nested lab and also a vSAN lab at that, is easier than ever before.

As you know, vSAN nodes require at least (1) cache drive and (1) capacity drive to become part of the storage layer in a vSAN environment. This is easily accomplished with a nested ESXi virtual machine running (2) VMDKs to satisfy these storage contribution requirements.

Aside from these drive requirements, let’s look at a few other things to consider when you create a VMware vSAN 6.7 U1 Nested Lab.

Create a VMware vSAN 6.7 U1 Nested ESXi Lab

There are a few other considerations to make with VMware vSAN 6.7 U1 Nested ESXi Lab configurations.

• Cloning ESXi virtual machines for vSAN hosts
• Nested ESXi networking
• Physical Host storage considerations
• vCenter Server configurations

Cloning ESXi virtual machines for vSAN hosts

As mentioned above, one of the benefits of a nested lab is the ability to quickly provision hosts by cloning virtual machines. When thinking about cloning an ESXi virtual machine for use as a VMware vSAN host, there are a few things that need to be considered for building out an ESXi nested lab configuration. If you don’t have an automated deployment of ESXi machines in the home lab which means most likely you are deploying ESXi hosts via cloning an ESXi template VM, etc, you need to consider a couple of things that can cause issues with vSAN.

William Lam has a great write up here on how to properly clone an ESXi host for nested lab purposes. Quoting from William’s blog here:

The first issue is that you will get a duplicated MAC Address of the VMkernel interface(s) because the Nested ESXi configuration is exactly the same.

The second issue is having a duplicated ESXi System UUID, also known as a VMkernel UUID which should normally be unique and can sometimes be used for tracking purposes. You can see this System UUID by running the following ESXCLI command: esxcli system uuid get or by looking in esx.conf configuration file.

Be sure to take care of the ESXi VM via the cloning process as outlined by the steps that William provides to ensure you don’t have any duplication of UUID and VMkernel MAC addresses.

Nested ESXi networking

Often, in a nested lab, the ESXi networking aspect can get confusing or can cause issues with configuration of the flow of network traffic between the ESXi virtual machines due to the extra layer of abstraction or “inception” however you want to look at it. This can especially get tricky if you are nesting your ESXi virtual machines on different ESXi physical hosts. This requires that you generally give attention to the physical networking layer between the physical ESXi hosts. Be sure to consider the following:

• Nested ESXi host networking from both a management perspective as well as the ESXi vSAN VMkernel ports TCP/IP perspective.
• You can segment out your vSAN traffic by creating a separate vSwitch on the physical ESXi hosts that you assign to the virtual NICs of the ESXi host’s connections carrying vSAN traffic
• Give attention to jumbo frames configuration both in the vSwitch configuration as well as any physical devices configured to carry traffic between the ESXi physical hosts for the particular vSwitches involved/switch ports. This may have already been completed if you are running a physical vSAN configuration with physical ESXi hosts, however, it may need to be given attention from a vSwitch perspective depending on the vSwitch you have your nested ESXi hosts attached to
• If you are not using 10 gig connectivity between your nested hosts (especially if these are housed on the physical ESXi hosts) you will see a “warning message” displayed if you run the proactive Network Performance Test if the resulting bandwidth between the nested hosts vSAN network connections are less than 850 Mbps.

Physical Host Storage Considerations

Another item to consider if you are wanting to provision a nested vSAN environment in a physical ESXi lab you may already have running is the physical datastore storage. If you are running a vSAN datastore in your physical environment, you won’t be able to run nested ESXi hosts on an existing vSAN datastore.

There are proposed workarounds such as running the following command on your ESXi hosts:

• esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

In my testing, this still does not work and others seem to have confirmed that in testing this. The bottom line is this, you will need a traditional datastore of some type such as iSCSI, NFS, or a directly attached disk.

In my case, I have a 2-node vSAN cluster configured with the vSAN storage, then each host has a standalone traditional datastore for housing these nested ESXi virtual machines which works great.

vCenter Server Considerations

The great thing about running a VMware vSAN 6.7 U1 Nested ESXi lab is that you can join the lab to your “production” vCenter Server or you can run a second vCenter VCSA appliance to create a totally separate vSphere environment for testing, learning, etc. This requires more hardware resources as you are spinning up a second vCenter environment, however, this is my preferred way of doing things as it helps to keep your “test” environment separate from your “production” home lab environment. Additionally, if you are not worried too much about performance, you can also reduce the virtual hardware resources assigned to the test VCSA appliance.

Either way, if you create a new cluster in your existing VCSA vCenter environment, or the separate environment with a new VCSA deployment, you can effectively manage and interact with the new nested VMware vSAN ESXi lab.

Takeaways

I am a huge advocate of home labs. Yes, there are many options available in the cloud and other great free lab environments such as VMware’s HOL’s that allow getting your hands on a lot of the great technologies that you may want to work with or work with on a daily basis. However, there is just no substitute for working with the hardware, the networking, the provisioning of storage yourself. Building out a home lab is a great learning experience that can pay huge dividends in skillset and the ability to troubleshoot. Hopefully this high-level look at how to create a VMware vSAN 6.7 U1 Nested ESXi Lab will help those who may be considering building out a nested lab and why you might want to do so.