Nested ESXi Lab Build Networking and Hardware
Many who are learning VMware vSphere and those who may already work with VMware vSphere products each and every day benefit from having a lab environment. I have always been a proponent of lab environments. Even with many touting moving to the cloud and such, there will always be a need for people who actually know what goes on behind the scenes in the data center. Having a lab environment is a great way to build, break, troubleshoot, upgrade, and most of all, learn. To learn VMware vSphere, having a nested ESXi lab is a great way to learn the fundamentals of the VMware vSphere hypervisor, without breaking the bank on numerous physical lab hosts. Let’s take a look at nested ESXi lab build networking and hardware to see how you can go about successfully building your nested ESXi lab from scratch.
What is nested virtualization?
Nested virtualization is where you are basically running a hypervisor “on top of” another hypervisor. Think of the movie “Inception”. You can run the ESXi hypervisor as a VM on top of another physical ESXi hypervisor host. Now, you may be wondering why you would want to do this. The answer is simple – labs.
When you run a nested ESXi lab as VMs running on a physical ESXi hypervisor host, you have all of the benefits that you normally have with a VM running inside of vSphere. This includes being able to control and work with the ESXi host VMs as you would any other VM. This allows you to spin up multiple ESXi hosts that can build a nested ESXi cluster so you can start playing around with features such as HA, DRS, and vMotion as well as the other enterprise features that you want to work with.
With nested virtualization, you can learn and play around with many great technologies including vSAN, NSX, and stretched clustering.
Build a Nested ESXi Lab
How do you build a nested ESXi lab? There are many different ways that you can build a nested ESXi Lab. One of the first things that you need is hardware to run the nested lab.
This can be as simple as a laptop or workstation class machine running VMware Workstation. VMware Workstation Pro 16 contains great features for running a nested ESXi lab.
With VMware Workstation Pro 16, it supports vSphere 7 as well as containers, so lots of great features for learning new technologies and working with the latest technologies. You can read my post here about VMware Workstation Pro 16:
For my purposes, I have the need for a more permanent lab solution with dedicated hardware that I run 24×7. I use Supermicro servers for this purpose. You can check out my post here about using Supermicro servers for a VMware home lab for specifics about the models I use and what I have configured in my lab environment.
Check out my article below on how to setup your Supermicro server to automatically power on and power off to save money on your electric bill.
Deploying a Nested ESXi lab
Can you deploy your ESXi hosts one-by-one by simply deploying the ISO into a VM? Yes. This may be the way you want to deploy your ESXi hosts to begin with to see the inner workings of how the install works, initial setup, etc.
If you load your ESXi hosts manually, be sure to set the processor setting:
- Expose hardware assisted virtualization to guest OS
However, if you want to progress to an automated deployment, THE best way to deploy a nested ESXi lab is with William Lam’s nested ESXi lab script. William has updated his script to deploy ESXi 7.0.
Check out the nested deployment here:
What does a Nested ESXi lab look like?
In case you are having difficulty visualizing what is going on with a nested ESXi lab, let’s take a look at one of my nested ESXi labs that I have running on one of my home lab hosts.
If you notice below, I have two vAPPs running with (3) nested ESXi hosts running inside each vAPP. Each of the vAPPs with ESXi hosts represents a vSphere cluster that I have running. As you can see on the same physical ESXi host, I also have a vCenter Server running as well.
What does this look like from the nested ESXi lab side? You can see below, after you login to your nested ESXi lab vCenter Server, it looks like any other vSphere environment would look.
In fact, you can do more complex nested environments such as I have below. Below is a stretched vSAN cluster running “all-flash” vSAN with a Witness node provisioned. So, as you can see, nested environments allow you to do very complex lab environments, testing many different technologies.
How to Setup a Nested ESXi Lab Networking
For many, it is kind of a mind bender to visualize the nested ESXi lab networking and how this is configured. However, the easiest way to think of the lab is to think of the fact that your nested ESXi hosts are simply VMs running on your physical ESXi host.
The virtual networking of the ESXi hosts is simply running on top of the vSphere networking on the physical ESXi host. However, there are some special things that need to take place with ESXi networking. Since your nested ESXi lab has the capability of actually running virtual machines inside the nested environment, there are settings that must be enabled to make this work properly.
With nested virtualization, the nested ESXi host is hosting traffic for a number of other “nested VMs” that are ran on the nested instance of ESXi. When this is the case, multiple MAC addresses will appear in the source address 802.3 field. Each virtual workload hosted by the nested ESXi host would need to communicate using the virtual ESXi network adapter. These extra MAC addresses are rejected as forged transmits.
Chris Wahl has a great writeup on the specifics of why this is needed here:
The solution is to enable promiscuous mode and forged transmits on your virtual switches on the physical ESXi host, that you want to carry traffic for the nested ESXi host. There is another solution that has been covered in great detail by William Lam. That is the ability now of vSphere 6.7 ESXi with the vSphere Distributed Switch to learn the MAC addresses of nested ESXi traffic.
Check out William’s post here on that subject:
Either solution is required to enable nested virtualization network traffic for VMs that you have running in your nested ESXi environment to pass traffic to your real physical network.
Now, thinking about the physical host networking, you can house your nested ESXi lab on any VSS or VDS port group of your choosing. The vSwitch that carries the nested ESXi lab traffic can be untagged frames or VLAN tagged frames.
Below, the port group my nested ESXi hosts will be connected to is a VLAN-backed port group called Servers.
Now, the nested ESXi hosts have a vSphere Distributed Switch provisioned that is running on top of the VDS on the physical ESXi host. Note, you can’t tag frames in your nested ESXi host configuration. However, keep in mind the ESXi traffic from the nested server will be tagged with the VLAN of the physical ESXi host port group it is attached to.
So, with that being said you can split off your “virtual network adapters” connected to your nested ESXi hosts and connect those to different port groups on your physical ESXi hosts to connect nested VMs to different VLANs.
Nested ESXi Lab Licensing
I am and have always been a fan of the VMUG Advantage subscription. You simply won’t find a better value for a home lab than the VMUG subscription.
With a VMUG subscription, you get vSphere licensing for products like: vSphere, vSAN, NSX-V & T, VCF, SRM, Horizon, and others.
- Price $200! You won’t find a better deal than this anywhere. Period.
Read my articles on VMUG coverage, subscription, features, etc here:
- VMware Cloud Foundation vCF 4.0 added to VMUG Advantage
- VMware Cloud Foundation added to VMUG Advantage Eval Experience
- VMware TestDrive added to VMUG Advantage Subscriptions
- VMware Site Recovery Manager SRM added to VMUG Advantage
- VMUG Advantage adds vRealize Automation 7.3 Enterprise
- VMUG VMware vRNI bundled with NSX offering
Wrapping Up
Hopefully this Nested ESXi Lab Build Networking and Hardware discussion will help any who are wanting to delve into setting up a nested ESXi lab. The process is really easy.
You can get started with just a bit of hardware and even just VMware Workstation. Or if you have dedicated hardware you can allocate, you can load ESXi on physical hardware and then load your nested ESXi hosts on top of that.
There are great resources available such as a VMUG subscription, William Lam’s vSphere 7 lab scripts, and other community resources.
