Roll Back VMware Meltdown and Spectre Microcode Patch
Update 1.24.2018 with William Lam PowerCLI code link
The last several days have certainly been a rollercoaster ride for VMware administrators and others who have been keeping a close eye on the latest news related to Meltdown and Spectre. There have been patches released and then there have been patches that have been pulled. This was the case with the VMware microcode patches that were associated with VMSA-2018-0004. VMware has officially pulled thoe patches associated with that particular security advisory, specifically ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG. The latest information on that front is found in the Intel Sightings in ESXi Bundled Microcode patches KB article found here. The patches were pulled around sightings made with Haswell/Broadwell processors. If you were proactive and applied the patches that have now been pulled and are running on Haswell or Broadwell processors, how do you Roll Back VMware Meltdown and Spectre Microcode Patch?
Roll Back VMware Meltdown and Spectre Microcode Patch
In the new security advisory linked above, VMware has provided a table of affected processors noted in the Intel sightings. The table found in the KB article contains the following CPUs.
The recommendation from VMware on how to Roll Back VMware Meltdown and Spectre Microcode Patch involves the following. I have highlighted a few important points to consider:
- On each affected ESXi host, add the following line in the /etc/vmware/config file:
- cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-“
- This hides the speculative-execution control mechanism for virtual machines which are power-cycled afterwards on the ESXi host.
- This line will need to be removed after applying a future fixed microcode from Intel in order to enable the full guest OS mitigations for CVE-2017-5715.
- When convenient, power-cycle virtual machines on the affected ESXi hosts; rebooting of the ESXi host is not required.
- Stateless vSphere ESXi Hosts using ESXi 5.5 or 6.0, this line must be re-applied every time the ESXi host reboots.
I simply enabled SSH and edited the file with WinSCP.
Again, according to the VMware recommendation you need to power-cycle virtual machines on the affected ESXi hosts.
***Update 1.24.2018*** – Automating Intel Sighting Remediation Using PowerCLI
William Lam has done a tremendously good job at providing very helpful automation scripts for various Meltdown/Spectre information gathering as well as remediation. He has produced a script that allows implementing the workaround for those who have already applied the VMware microcode patches on systems affected by the “Intel Sightings”. Additionally he has modified the script so that it does not need SSH enabled.
Take a look here: https://www.virtuallyghetto.com/2018/01/automating-intel-sighting-remediation-using-powercli-ssh-not-required.html
Another Possible Rollback Method
As many of you may already know in tinkering with home labs or in dealing with various production issues there is a roll back mechanism in place to roll ESXi back to a previous version. As noted in the KB found here, this is a fairly simple operation that involves hitting a keystroke when the ESXi host is booting. After having applied the microcode patch in my home lab but not the Supermicro BIOS patch as this has not yet dropped, I wanted to utilize this mechanism to revert my host(s) back to the U1 build I had loaded previously.
Note – This is not a supported means or at least not listed in the VMware KB article on the procedure to mitigate the microcode patch if already installed. This is simply a means to show how this procedure can roll a host back to a previous state. I was able to utilize this in the home lab successfully to easily get back to a previous build version. Important – back up your configuration data before making any changes. See the KB article here on Backing up and restoring ESXi configuration using the vSphere Command-Line Interface and vSphere PowerCLI (2042141).
Also key in being able to revert the host back to a previous version is the host has to have been updated using one of the following methods:
- VIB installation or removal
- Profile installation or removal
- ESXi host update using VMware Update Manager
- Updated from a ISO
Note below, you see the current build version is 6.5.0-1.38.7526125 which is the build containing the pulled microcode patch. You can hit ‘Y’ to proceed with the rollback, or ‘N’ to cancel.
After confirming the rollback operation, the server will quickly boot ESXi with the previous build version.
Beautiful – we are once again running on the Update 1 build.
After a quick update to Update 1 patch 2, we can see we have the build version prior to the microcode, etc patches.
Concluding Thoughts
With all of the patches being released at a rapid pace, there is a good chance it is a wise move to sit back and let the dust settle a bit, especially since Intel has pulled this first microcode release. There is no doubt there could be other findings related to patching guest operating systems as well as hypervisors. The good news is the process to Roll Back VMware Meltdown and Spectre Microcode Patch is fairly painless at least in the current form of the KB. Stay tuned for updates as the KB is updated or amended with new information.
