Guest VM Running in Nested Nutanix CE on VMware vSphere Won’t Boot
I have been playing around quite a bit lately in the home lab with a nested installation of Nutanix Community Edition (CE) as it is a great way to get your feet wet with the Nutanix hypervisor for free and running in a nested configuration in a lab environment. I have already covered here how to get up and running quickly with Nutanix CE in a VMware vSphere 6.7 VM which is a fairly straightforward process. However, after getting the environment configured and everything working as expected, I found the guest VM running in Nested Nutanix CE on VMware vSphere won’t boot. What gives? Let’s take a look at the problem and the workaround that was able to get around this issue in the lab.
Nutanix CE Guest VM Not Booting in VMware vSphere
I noticed when I had created my first VM in the nested Nutanix CE installation and kicked off a boot of the VM, after waiting a few minutes, I didn’t see any network connections being made from the VM. A quick hop onto the console and sure enough there appeared to be an issue. The virtual machine was simply stuck on booting and simply displayed the BIOS screen with no other activity. Thinking this was a boot priority issue possibly, I spent some time trying to play around with the boot order and a few other tricks, however, nothing seemed to change this behavior. After quite a bit of time just tinkering myself, I decided to see if this was a known issue with running a nested installed of Nutanix CE or maybe in particular, running on top of VMware vSphere.
Below is a screen capture of the behavior seen. The boot of the Nutanix CE guest VM will simply stick on this screen forever.
After a bit of Googling, I stumbled onto this thread on one of the Nutanix Forums concerning issues booting nested VMs on Nutanix CE running on top of VMware vSphere. The good thing is there is a workaround if you want to run actual nested VMs inside of Nutanix CE and VMware vSphere.
Guest VM Running in Nested Nutanix CE on VMware vSphere Won’t Boot Workaround
Thankfully, the workaround to allow actually running VMs on top of the Nutanix CE AHV hypervisor only involves editing a couple of files, adding, modifying, and deleting a subset of lines from the configuration files.
First, edit the following file:
/home/install/phx_iso/phoenix/svm_template/kvm/default.xml
Create a backup of the default.xml file before making changes.
Edit the default.xml file and underneath the features section, add the line:
<pmu state=’off’ />
The next file that needs to be updated is located here:
/var/cache/libvirt/qemu/capabilities/<your number>.xml
Again, first, backup your existing file before editing.
Basically, in the qemu capabilities file, you are simply removing the existing pc-i440fx-rhel7.2.0 file and renaming the pc-i440fx-rhel7.3.0 to 7.2.0. My file looked like the following before making the changes.
In the end, it will look like this.
After making the changes to the two files needed, initiate a reboot of the Nutanix CVM and host. This is found under the Settings > Reboot menu in Prism.
After editing the two files in question, rebooting the Nutanix environment, now the guest VMs running inside of the nested Nutanix CE environment boot correctly! The great thing about the above solution is that it takes effect for all VMs that are provisioned in the Nutanix environment.
Wrapping Up
If you are running into the issue of a Guest VM Running in Nested Nutanix CE on VMware vSphere Won’t Boot, try the simple workaround listed in the post. For me, this worked like a charm, and again, kudos to the Nutanix CE forum members for tossing out this solution. I wanted to create this post to help give even more visibility to this workaround in case you have hit this snag in playing around with your Nutanix CE environment with actually running Guest VMs inside Nutanix on vSphere or another hypervisor. I have only personally tested this workaround with VMware vSphere 6.7 Update 2 so I cannot vouch for this solving this issue across all hypervisor environments. More to come!
If you missed the first part of how to deploy nested Nutanix CE in VMware vSphere, check out that post here:
