Many VMware Homelabs are based on Intel NUCs. It is also very common that generations are mixed which can lead to compatibility issues when trying to vMotion VMs across different generations. This is typically where VMware EVC comes into play.
VMware EVC creates a baseline of CPU instructions for virtual machines running on ESXi hosts. When you add newer Hosts, EVC hides the new CPU instructions to the virtual machines. While this works great for Xeon CPUs used in professional servers, it has some limitations with consumer CPUs used in the Intel NUC ecosystem.
The problem has become worse with the latest 10th Gen Comet Lake/Frost Canyon NUC. Despite having a 10th generation CPU, it requires the EVC baseline to be configured to "Sandy Bridge", which is the 2nd generation of Intel Core-i CPUs:
- NUC10i7FNH/NUC10i7FNK (Intel Core i7-10710U - 6 Core, up to 4.7 GHz)
- NUC10i5FNH/NUC10i5FNK (Intel Core i5-10210U - 4 Core, up to 4.2 GHz)
- NUC10i3FNH/NUC10i3FNK (Intel Core i3-10110U - 2 Core, up to 4.1 GHz)
When you try to activate VMware EVC higher than Sandy Bridge, the following error message is displayed:
Compatibility
The host's CPU hardware does not support the cluster's current Enhanced vMotion Compatibility mode. The host CPU lacks features required by that mode.
When you try to add the Host to an EVC Enabled Cluster, the task fails:
Operation failed!
The host's CPU hardware does not support the cluster's current Enhanced vMotion Compatibility mode. The host CPU lacks features required by that mode.
CPUID faulting is not supported.
See KB 1003212 for more information.
Host is of type: vendor intel family 0x6 model 0xa6
A common EVC Mode that allows Gen5 - Gen8 NUCs to be used in the same Cluster is "Haswell". To add a 10th Gen NUC, you have to lower EVC to Sandy Bridge. The problem with the "Frost Canyon" is that it does not support the CPUID faulting feature, required since EVC "Ivy Bridge".
Generation 6-8 NUCs are very generous when it comes to vMotion compatibility. They have a similar feature set and you can migrate VMs without EVC. My first Idea was to retire the 5th Gen NUC and disable EVC. This could be a solution because the CPUID faulting feature can be disabled in the Virtual Machines configuration by adding the following line to the vmx configuration file:
cpuidFaulting.enable = "FALSE"
With "CPUID faulting" disabled, I can migrate VMs from Gen6-8 to Gen10, but unfortunately not the other way as the Gen10 has 4 new features:
- MDS_NO
- RSBA_NO
- IBRS_NO
- RDCL_NO
As a result, I tried to create my own EVC Mode to allow Gen5 - Gen10 NUCs to be in the same Cluster. Basically, you only need to remove Restricted Transactional Memory (RTM), Hardware Lock Elision (HLE) and CPUID faulting from the Broadwell EVC Mode to be compatible with all of my NUCs.
If you have the same problem, here is how you can do the same. Please keep in mind that the file might be overwritten when you patch your vCenter Appliance, so you have to copy it again. And of course, it's fully unsupported.
- Download EVC XML File (Right-Click -> Save as)
vSphere 6.7: evcModes67-NUC.xml
vSphere 7.0: evcModes70-NUC.xml - Make a backup copy of the old file:
# cd /usr/lib/vmware-vpx/ # cp evcModes.xml evcModes.xml.backup
- Copy the new XML to the vCenter Service Appliance at /usr/lib/vmware-vpx
- Replace the original XML file with the one downloaded for your vCenter version
vSphere 6.7 # cp evcModes67-NUC.xml evcModes.xml vSphere 7.0 # cp evcModes70-NUC.xml evcModes.xml
- Restart the vpxd Service
# service-control --restart vpxd
- To activate the EVC Mode you have to use PowerCLI. Verify that the EVC Mode is available:
PS> $Global:DefaultVIServer.ExtensionData.Capability.SupportedEVCMode |select key
- Activate the EVC Mode:
PS> Set-Cluster NUC -EVCMode "intel-nuc"
- Verify EVC Mode:
Get-Cluster NUC | Select Name,EVCMode Name EVCMode ---- ------- NUC intel-nuc
Reminder: VPXD Services fails to start when Clusters are configured with EVC mode intel-nuc and the mapping is missing (after patches). Copy the file again and restart vpxd.
I think the Gemini Lake/June Canyon NUCs require a similar hack. I had to go back to Westmere to permit vMotion to/from a DL380 Gen8.
Thanks for all the work you've put into this. I unfortunately have ran into an error when attempting to switch the cluster to the new EVC mode after following your steps. Wonder if you could take a look please:
"Could not obtain the result of task '/VIServer=xxxxxx.local:443/Task=Task-task-2002/'. Task name is 'ConfigureEvcMode_Task'. The following error occured: Unexpected error occured. Cannot convert type 'VirtualMachineFeatureRequirement' to 'Object'. Set-Cluster: 19/05/2020 16:38:08 Set-Cluster Update operation for cluster 'NUC' failed."
Any hints or thoughts as to why this could be?
I'm working with a NUC5i5MYHE & NUC10i7FNK.
I know it's been here for more than 1.5 years, but I just got this error and got it working, so posting here for the next person searching for this. No EVC mode was present in my cluster, so I created an empty cluster with the correct EVC mode, migrated one host to the cluster, moved manually the vCenter to this node, registered it again, moved the last node to the new cluster.
I have only Gen7, 8 & 10 NUCs. I created a new cluster and then assigned the custom EVC mode (Intel-nuc) in this article to it. When I try to add a Gen 10 NUC (since it has no VMs) to the cluster, I get "general runtime error occured" Can you please help?
I have 2 identical 7-th gen NUCs in the cluster with i5-7260U proceccors.
Yes, the highest EVC mode I can enable Haswell as you mentioned.
Just wondering why it is only Haswell and nothing newer like Skylake?
Probably because NUC-s are anyway not officially supported?
Consumer and Server CPUs are having different instruction sets. EVC modes are made for Xeon CPUs and therefore you might be unable to use them on consumer CPUs.
If you have identical CPUs you can run without EVC to get all available features.
Somehow the vMotion of the vCenter appliance did not work if the EVC was disabled.
That was the reason for enabling EVC. Not sure what I did wrong.
Thank you for this post! Based on this, I was able to create an updated version of the evcModes.xml file the latest vCenter (appliance) 7.0.3:
https://home.deds.nl/~tsa/evcModes703-NUC.xml
For the rest the steps are the same as mentioned in the original post.
Thanks Mr_Q !