I have been running pfSense (with dedicated quad port card using PCI-E passthrough) for some weeks with no issue as a Virtual Machine on TrueNAS which uses KVM (QEMU). I have been use the 'Custom' CPU option with no model selected which presents the following CPU in pfSense:
QEMU Virtual CPU version 2.5+
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: No
QAT Crypto: No
NB: QAT = Intel only.
The Problem
MI want to have hardware AES-NI support from the CPU (passed through by the real CPU) but the default QEMU CPU does not have the CPU flags. The other modes don't work either for some reason.
This is what happens when i try the different CPU modes in KVM/QEMU on TrueNAS:
- Custom Mode (Default/QEMU Virtual) CPU mode
- Does not support hardware AES-NI (QAT is Intel only) and does not have a lot of the other CPU flags a modern PC has.
- Exposed to various CPU attacks.
- pfSense runs fine with this CPU.
- A very compatible choice, but lacks performance.
- 'Host Model' CPU mode
- Allows pfSense to load, but the GUI and routing does not work.
- 'Host Passthrough' CPU mode
- Allows pfSense to load, but the GUI and routing does not work.
The issues here are probably caused by one or more of the following:
- CPU is too new (AMD Ryzen 9 7900 12-Core Processor with 128GB ECC on and x670 board)
- Being an AMD CPU
- The OS being FreeBSD (pfSense runs on this OS)
- FreeBSD driver issues.
The Question
Because my CPU is not compatible, for whatever reason, I will have to select one of the pre-made Custom CPUs (which will adds an emulator layer) but I need one with as many of the features as possible. I am not able to write ans apply my own CPU profile and I would also not want to make changes to TrueNAS manually which is definatley not recommended.
Which one should I choose to get the best out of my CPU?
The Solution
Be aware that as TrueNAS is developed, newer CPU models might become availabe to have greater parity with the QEMU repository.
Conclusions
- After a brief look of the CPUs supported by TrueNAS, it looked like the all of the newer CPUs, certainly the ones I could identify were server ones.
- The CPUs on offer are at least 3-4 years older than currently available CPUs.
- You should use a Custom CPU of the same brand i.e.
- An Intel Host CPU should use an Intel Guest CPU.
- An AMD Host CPU should use an AMD Guest CPU.
- You should choose a Custom CPU that is either the same generation or lower to make sure all the CPU features advertised by the flags can be fulfilled.
- I do not know what the different CPU modes are
- -Client
- -Server
- -noTSX
- -IBRS
- The Best CPU mode selection (in order)
- Host Passthrough = This passes the host CPU model features, model, stepping, exactly to the guest.
- Host Model = Automatically picks a CPU model that is similar the host CPU, and then adds extra features to approximate the host model as closely as possible.
- Custom (Named model) = These allow the guest VMs to have a degree of isolation from the host CPU, allowing greater flexibility in live migrating between hosts with differing hardware.
CPU Selection
Based on my research, my CPU selections are below:
- Intel
- Xeon Processor (Cascade Lake, 2019)
- Xeon Processor (Icelake, 2021/2022)
- Icelake-Client
- Icelake-Client-noTSX
- Icelake-Server
- Icelake-Server-noTSX
- AMD
- EPYC (1st Gen, 2017)
- EPYC-Rome (2nd Gen, 2018)
Notes
- If you are not sure if your OS supports a particular CPU, use the QEMU default. It is the most compatible but has security issues. Testing a CPU is always the best way to check compatibility but don't use it on a VM that has live data on it until you are sure.
- Use the same Brand of CPU as that of the Host Motherboard.
- You need to use 'CPU Mode = Custom' to use these CPUs.
Research
KVM / QEMU Information
- The way of KVM: guest's CPU flags | by CocCoc Techblog | Coccoc Engineering Blog | Medium
- How KVM virtualizes CPU architecture from host machine.
- The answer is simple: compatibility. By default, KVM sets the cpu mode to custom with generic model— to ensure that a persistent guest sees the same hardware no matter what host the guest is booted on
- By default KVM uses custom mode and set the CPU model to generic — which misses important flags: aes, pcid and rdrand. If live migration is a concern, use Host model, otherwise, Host passthrough should be used to maximize the features the host’s CPU supports.
- CPU Modes
- Host passthrough
- This passes the host CPU model features, model, stepping, exactly to the guest.
- Note that KVM may filter out some host CPU model features if they cannot be supported with virtualization.
- Live migration is unsafe when this mode is used as libvirt / QEMU cannot guarantee a stable CPU is exposed to the guest across hosts.
- This is the recommended CPU to use, provided live migration is not required.
- Custom (Named model)
- QEMU comes with a number of predefined named CPU models, that typically refer to specific generations of hardware released by Intel and AMD.
- These allow the guest VMs to have a degree of isolation from the host CPU, allowing greater flexibility in live migrating between hosts with differing hardware.
- Host model
- This uses the QEMU "Named model" feature, automatically picking a CPU model that is similar the host CPU, and then adding extra features to approximate the host model as closely as possible.
- This does not guarantee the CPU family, stepping, etc will precisely match the host CPU, as they would with "Host passthrough", but gives much of the benefit of passthrough, while making live migration safe.
- Host passthrough
- Qemu/KVM Virtual Machines | Proxmox
- Qemu (short form for Quick Emulator) is an open source hypervisor that emulates a physical computer.
- A short but concise overview of QEMU.
- QEMU User Documentation — QEMU documentation
- libvirt/src/cpu_map at master · libvirt/libvirt · GitHub - GitHub page with all of the QEMU CPU profiles, if you edit them you can see the CPU flags.
- Recommendations for KVM CPU model configuration on x86 hosts — QEMU documentation - Seems to be the same as the link below.
- QEMU / KVM CPU model configuration — QEMU documentation
- Host passthrough
- This passes the host CPU model features, model, stepping, exactly to the guest. Note that KVM may filter out some host CPU model features if they cannot be supported with virtualization. Live migration is unsafe when this mode is used as libvirt / QEMU cannot guarantee a stable CPU is exposed to the guest across hosts. This is the recommended CPU to use, provided live migration is not required.
- It is possible to optionally add or remove individual CPU features, to alter what is presented to the guest by default.
- Named model (Host Model)
- QEMU comes with a number of predefined named CPU models, that typically refer to specific generations of hardware released by Intel and AMD. These allow the guest VMs to have a degree of isolation from the host CPU, allowing greater flexibility in live migrating between hosts with differing hardware.
- It is possible to optionally add or remove individual CPU features, to alter what is presented to the guest by default.
- Host Model
- Libvirt supports a third way to configure CPU models known as “Host model”. This uses the QEMU “Named model” feature, automatically picking a CPU model that is similar the host CPU, and then adding extra features to approximate the host model as closely as possible. This does not guarantee the CPU family, stepping, etc will precisely match the host CPU, as they would with “Host passthrough”, but gives much of the benefit of passthrough, while making live migration safe.
- Default x86 CPU models
- The default QEMU CPU models are designed such that they can run on all hosts. If an application does not wish to do perform any host compatibility checks before launching guests, the default is guaranteed to work.
- The default CPU models will, however, leave the guest OS vulnerable to various CPU hardware flaws, so their use is strongly discouraged. Applications should follow the earlier guidance to setup a better CPU configuration, with host passthrough recommended if live migration is not needed.
- The following CPU models are preferred for use on Intel hosts. See for a list.
- Intel Xeon Processor (Cascade Lake, 2019), Intel Core Processor (Skylake, 2015).
- The following CPU models are preferred for use on AMD hosts. See for a list.
- AMD EPYC Processor (2017).
- Host passthrough
- QEMU User Documentation - Xilinx Wiki - Confluence - Seems quite in-depth.
- CPU Options (-Client/-Server/-noTSX/-IBRS)
- 1840008 – 5 more -noTSX x86 CPU models are newly introduced - This mentions -noTSX
- virtualization - KVM: Which CPU features make VMs run better? - Server Fault
kvm -cpu ?model x86 Opteron_G3 AMD Opteron 23xx (Gen 3 Class Opteron) x86 Opteron_G2 AMD Opteron 22xx (Gen 2 Class Opteron) x86 Opteron_G1 AMD Opteron 240 (Gen 1 Class Opteron) x86 Nehalem Intel Core i7 9xx (Nehalem Class Core i7) x86 Penryn Intel Core 2 Duo P9xxx (Penryn Class Core 2) x86 Conroe Intel Celeron_4x0 (Conroe/Merom Class Core 2) x86 [n270] Intel(R) Atom(TM) CPU N270 @ 1.60GHz x86 [athlon] QEMU Virtual CPU version 1.0 x86 [pentium3] x86 [pentium2] x86 [pentium] x86 [486] x86 [coreduo] Genuine Intel(R) CPU T2600 @ 2.16GHz x86 [kvm32] Common 32-bit KVM processor x86 [qemu32] QEMU Virtual CPU version 1.0 x86 [kvm64] Common KVM processor x86 [core2duo] Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz x86 [phenom] AMD Phenom(tm) 9550 Quad-Core Processor x86 [qemu64] QEMU Virtual CPU version 1.0
- How to add a new architecture to QEMU - Part 2 | Florian Göhler - In this article, I will explain how a new CPU can be added to QEMU.
- Qemu/KVM Virtual Machines - Proxmox VE - A article overviewing QEMU in Proxmox.
- QEMU/Options - Gentoo Wiki - This article describes some of the options useful for configuring QEMU virtual machines. For the most up to date options for the current QEMU install run man qemu at a terminal.
CPU Information
- Epyc - Wikipedia
- Epyc is a brand of multi-core x86-64 microprocessors designed and sold by AMD, based on the company's Zen microarchitecture. Introduced in June 2017, they are specifically targeted for the server and embedded system markets.
- Epyc processors share the same microarchitecture as their regular desktop-grade counterparts, but have enterprise-grade features such as higher core counts, more PCI Express lanes, support for larger amounts of RAM, and larger cache memory
- Xeon - Wikipedia
- List of Intel Xeon processors - Wikipedia
AES-NI / QAT
- pfSense AES-NI Hardware Crypto Acceleration in KVM | About - Learn more about pfSense AES-NI Hardware Crypto Acceleration in KVM. Our pfSense Support team is here to help you out.
- Hardware — Cryptographic Accelerator Support | pfSense Documentation
- Intel QuickAssist Technology (QAT) = Intel CPU only.
List of KVM/QEMU CPUs in TrueNAS-SCALE-22.12.3.3
pentium pentium2 pentium3 pentiumpro coreduo n270 core2duo qemu32 kvm32 cpu64-rhel5 cpu64-rhel6 qemu64 kvm64 Conroe Penryn Nehalem Nehalem-IBRS Westmere Westmere-IBRS SandyBridge SandyBridge-IBRS IvyBridge IvyBridge-IBRS Haswell-noTSX Haswell-noTSX-IBRS Haswell Haswell-IBRS Broadwell-noTSX Broadwell-noTSX-IBRS Broadwell Broadwell-IBRS Skylake-Client Skylake-Client-IBRS Skylake-Client-noTSX-IBRS Skylake-Server Skylake-Server-IBRS Skylake-Server-noTSX-IBRS Cascadelake-Server Cascadelake-Server-noTSX Icelake-Client Icelake-Client-noTSX Icelake-Server Icelake-Server-noTSX Cooperlake Snowridge athlon phenom Opteron_G1 Opteron_G2 Opteron_G3 Opteron_G4 Opteron_G5 EPYC EPYC-IBPB EPYC-Rome Dhyana POWER6 POWER7 POWER8 POWER9 POWERPC_e5500 POWERPC_e6500