Running multiple operating systems on the same computer is a very old trick which, as some have pointed out, goes back to the early days of IBM mainframe computer systems. When desktop PCs and Macs first came out, most operating systems and hardware went hand-in-hand, and there was little interest in trying to virtualise one operating system on another; that’s perhaps just as well, as processors and memory management weren’t able to assist. Since then virtualisation has developed and flourished, and Intel, ARM and others have added hardware support. This article explains some of the background, in the context of virtualisation on Apple silicon Macs.

Central to virtualisation is software known as a hypervisor, which makes it all possible. This provides a layer of abstraction somewhere between the guest operating system and hardware of the host, roughly equivalent to the kernel in an operating system which isn’t virtualised. The name is derived from it being the supervisor of supervisors.

Architecture

Almost fifty years ago, a distinction was drawn between what were then seen as the two types of virtualisation.

Type 1, also known as native or bare metal, runs a hypervisor directly on its hardware, and the guest operating systems run on that hypervisor. The hypervisor is then responsible for managing those virtual machines (VMs), giving them virtual CPUs (vCPUs), virtual memory, and other hardware resources. Examples of this type include VMware ESX Server and Microsoft Hyper-V.

Type 2, also known as hosted, runs a primary host operating system on the hardware, and hypervisors then run on top of, or in close conjunction with, that to deliver the same range of services to the guest operating systems. A host operating system can run more than one hypervisor at a time, and each hypervisor in turn can run more than one guest, but ultimately all converge on the host operating system and its kernel. These are now much more common on desktop computers, and include popular products such as VMware Fusion, Parallels Desktop and Oracle VirtualBox.

Handling problems

Another more recent distinction is based on how the hypervisor deals with guests trying to run instructions which interact with the system state, so are deemed sensitive. These could change the system state, or could be instructions which depend on the system state. Sensitive, unprivileged instructions can’t be virtualised, and a hypervisor needs some method of handling them if a guest operating system is to be run virtualised.

The older approach, required before hardware support for virtualisation became available, enables unmodified operating systems to be run as guests. This requires the translation of sensitive instruction sequences before they can be executed, imposing performance overhead. Known as full or software virtualisation, it was the basis for early versions of VMware, for instance.

With the introduction of hardware support, running unmodified guest operating systems became simpler, as problem instruction sequences can be executed directly. This avoids any translation overhead, and is known as hardware virtualisation, HVM or HV. Hypervisors adopting this can use Intel’s VT-x feature set, typically with Extended Page Tables (EPT) and Unrestricted Mode, or ARM’s AArch64 virtualisation, sometimes referred to as its Virtualization Extensions. It’s no coincidence that those are the hardware requirements for the respective hypervisors supported by macOS.

ARM Virtualization Extensions feature an additional ‘exception level’, EL2 hypervisor, which offers stage 2 translation, EL1/0 instruction and register access trapping, and virtual exception generation. Stage 2 translation allows a hypervisor to control which memory-mapped resources a VM can access, and where those appear in the VM’s address space, augmenting the stage 1 translation controlled by operating systems. Trapping allows a hypervisor to trap operations, such as those configuring low level controls, and emulate them where necessary.

System hardware uses interrupts to signal software, such as a GPU sending an interrupt to indicate it has completed rendering a frame. ARM Virtualization Extensions include support for virtual interrupts in EL2, and the forwarding of others to a specific VM or vCPU.

Device support

Another important consideration with Type 2 hypervisors is support for hardware in the form of standard devices. Apple has separated these from its Hypervisor framework, introducing them in its higher-level Virtualization framework in macOS 12. This provides an abstraction layer over I/O devices conforming to the Virtio standard (for most devices), and requires both the guest and host operating systems to support that standard. When a guest passes requests to a front-end Virtio device para-driver, they are handled by a Virtio back-end driver, in turn interacting with the device. These are illustrated below.

Virtio support in the Virtualization framework limits the range of guest operating systems to Linux and macOS 12 and later. Although third-party Virtio support for Windows is available on Linux hosts, it’s not currently supported on Apple silicon with macOS hosts.

Core types

Currently, as far as I know, all other hypervisors have been designed for SMP processors with their identical cores, and Apple’s appears to be the first to run at scale on SoCs with heterogeneous core types. Although macOS has an elaborate scheme for allocating threads to different core types in Apple silicon chips, based on Quality of Service (QoS) settings, the macOS hypervisor currently treats its vCPUs as having identical cores, and threads run on them are allocated to physical CPUs as if they all have the same high QoS. Guest macOS thus ignores QoS and the normal core allocation scheme.

Apple documents vCPU life cycle as a single thread created using pthread_create and running from hv_vcpu_create through hv_vcpu_run to hv_vcpu_destroy.

I hope this has provided better insight into macOS Hypervisor and Virtualization features on Apple silicon Macs, showing where they’re currently at in the context of hypervisors and virtualisation more generally. What no one outside Apple knows is where this is all heading, and its impact on macOS itself. It’s certainly a feature worth watching in the evolution of Apple silicon and macOS.

Reference

Edouard Bugnion, Jason Nieh, Dan Tsafrir (2017) Hardware and Software Support for Virtualization, Morgan & Claypool. ISBN 978 1 62705 693 9.

Previous articles:
1 How well does it work?
2 How does it work?
3 Configuration, VM and boot
4 Core allocation in VMs
also see Why are Disk Images so slow at writing?