New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected

Yet another security vulnerability has been discovered that is based on the Spectre and Meltdown attacks found in Intel processors. The new exploit is called Foreshadow and exploits Intel’s SGX technology which is ironically designed to protect sensitive data such as credit card numbers, social security numbers, etc from alteration or unauthorized outside access. This is supposed to protect the data from malware or attackers in general. What is concerning as well is that virtual machines running on enterprise hypervisors can be vulnerable. Attackers can compromise one virtual machine and then read memory from another virtual machine hosted on the same server. Specific to VMware, let’s take a look at New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected and see what steps VMware has already take to mitigate the flaw.

New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected

Intel has disclosed a new type of CPU speculative-execution vulnerability that is collectively known as L1 Terminal Fault or “Foreshadow” that has vulnerable any past and current Intel processor from at least 2009-2018.  The vulnerability happens when affected Intel microprocessors speculate beyond the authorized data access.  This allows Intel microprocessors to expose a new side-channel for attack.  This most importantly affects virtual machines running in hypervisor environments such as VMware.  This allows a malicious virtual machine running on an affected CPU to speculate contents of the hypervisor or another VM’s privileged information that resides at the same time in teh same core’s L1 Data cache.

Below are the supported VMware Intel processors affected:

Current Intel processors share the physically-address L1 Data Cache across both logical processors of a Hyperthreaded enabled core, simultaneous scheduling of software threads on both logical processors can create the potential for security compromise and the stealing of information.

• Official CVE describing the security flaw:  https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3646

There are two attack vectors that are described here:

• Sequential-context – A malicious virtual machine can infer contents of L1 data of previous thread
• Concurrent-context  – A malicious virtual machine can infer contents of L1 data of concurrently executing thread

Below is the current information regarding the Foreshadow Intel flaw from a VMware perspective.

VMware has already outlined details of the flaw in the official KB article covering the security advisories here:

• https://kb.vmware.com/s/article/55636

The L1 Terminal Fault is covered in the following summaries:

• CVE-2018-3646 (L1 Terminal Fault – VMM)

• CVE-2018-3620 (L1 Terminal Fault – OS)

• CVE-2018-3615 (L1 Terminal Fault – SGX)

VMware Mitigate of Foreshadow L1 Terminal Fault Vulnerability

VMware outlines a two-fold approach to mitigating the new security vulnerability:

• vSphere updates and patches
• Enabling a new feature known as “ESXi Side-Channel-Aware Scheduler”

What does the ESXi Side-Channel-Aware Sheduler do?  Well in basic terms, it will only schedule the hypervisor and Vms on one logical processor of an Intel Hyperthreading-enabled core.  This means the ESXi scheduler will not make use of all the hyperthreading cores presented.  The feature has made mention that it will impose a non-trivial performance impact.  You can imagine that only scheduling of ESXi and VMs on a number of cores will certainly lower the performance of virtual machines as they won’t be able to take advantage of all the hyperthreaded cores available.

VMware has outline a (3) phase deployment of the mitigation process:

• Update Phase –
• Planning Phase
• Scheduler-Enablement Phase

The two attack vectors mentioned above are mitigated in the following ways:

• Sequential-context – mitigated by vSphere patches
• Concurrent-context  –  mitigated by ESXi Side-Channel-Aware Scheduler

Update Phase:

With the update phase, the sequential-context attack vector is mitigated by simply applying the relevant vSphere patches.  A note here is the mitigation releases for Foreshadow are dependent upon Intel microcode updates that have already been released as vSphere update patches which are documented here:  VMSA-2018-0020

The workflow is basic update flow, however, it needs to be mentioned:

• Update vCenter first
• Then ESXi updates

Updating ESXi before vCenter may result in very cryptic notification messages.

Planning Phase

With the planing phase, you start to plan on implementing the Concurrent-context attack vector mitigation.  With this phase, VMware has introduced an HTA Mitigation Tool that has as its purpose the complete mitigation of the L1 Terminal Fault (L1TF) vulenerability found in CVE-2018-3646.  The HTA Mitigation Tool requires enabling the ESXi Side-Channel-Aware Scheduler.  The scheduler is not enabled by default.

The HTA Mitgation Tool is available for download at the bottom of the page found here:

• Download HTA Mitigation Tool – Scroll to the bottom of the page and the .ZIP file is attached

With the assistance of the HTA Mitigation Tool, the planning phase helps to establish an understanding of the current environment and if sufficient CPU capacity is available to enable the scheduler without operational impact.

As already mentioned, disabling hyperthreaded cores from the scheduler can certainly lead to a performance impact, but also other areas of impact to be aware of before enabling including:

• Any virtual machine that has more vCPUs assigned than the number of physical cores on the ESXi host
• Any virtual machine that may have custom affinities assigned or NUMA settings
• VMs that are latency sensitive
• ESXi hosts that have 70% or higher CPU utilization
• ESXi hosts with custom CPU resource options
• HA Clusters where rolling upgrades would result in higher than 100% CPU usage

Scheduler-Enablement Phase:

In this phase, the actual mitigation of the concurrent-context vulnerability is implemented.  There are multiple ways to implement the ESXi Side-Channel-Aware Scheduler:

• Set the hyperthreadingMitigation advanced setting on a ESXi host
• vSphere Web Client or vSphere Client
  • Connect to the vCenter Server using either the vSphere Web or vSphere Client.
  • Select an ESXi host in the inventory.
  • Click the Manage (5.5/6.0) or Configure (6.5/6.7) tab.
  • Click the Settings sub-tab.
  • Under the System heading, click Advanced System Settings.
  • Click in the Filter box and search VMkernel.Boot.hyperthreadingMitigation
  • Select the setting by name and click the Edit pencil icon.
  • Change the configuration option to true (default: false).
  • Click OK.
  • Reboot the ESXi host for the configuration change to go into effect.
• ESXi Embedded Host Client
  • Connect to the ESXi host by opening a web browser to https://HOSTNAME.
  • Click the Manage tab
  • Click the Advanced settings sub-tab
  • Click in the Filter box and search VMkernel.Boot.hyperthreadingMitigation
  • Select the setting by name and click the Edit pencil icon
  • Change the configuration option to true (default: false)
  • Click Save.
  • Reboot the ESXi host for the configuration change to go into effect.
• ESXCLI command line utility
  • esxcli system settings kernel list -o hyperthreadingMitigation
  • esxcli system settings kernel set -s hyperthreadingMitigation -v TRUE

Takeaways

The New Intel Vulnerability Foreshadow L1 Terminal Fault VMware Affected is certainly a developing story.  VMware has reacted quickly and provided an interesting tool in the HTA Mitigation Tool.  There is one thing for sure, the Spectre and Meltdown fallout is continuing to add to the saga of worse performance.  With the mitigation for the concurrent-context, the utility enables scheduling of only one logical processor (Hyper-Thread) of a Hyper-Thread-enabled core.  This means as VMware stated, the performance impact will be non-trivial meaning it will certainly impact performance.  Keep an eye on this post as I will keep it updated if any new information comes to light.