Booting an M1 Mac from hardware to kexts: 3 XNU, the kernel

In the previous articles in this series, I have explained the first three of four stages in the booting of M1 Macs, at the end of which iBoot has verified the kernel, loaded it, and handed the Mac over to it. The kernel now has to use hashes from LocalPolicy to complete the validations to satisfy the selected level of Secure Boot, get the rest of the SoC up and running, including the other 7 or 9 CPU cores, and load the rest of macOS ready for the user to log in. Discovering what happens here is far easier, as with the kernel comes the Unified Log to give us a detailed account and information about any errors as they occur.

Even so, there are substantial gaps in the information. In the log, the marker of the start of the log record is the distinctive message
=== system boot:
followed by the UUID for that boot. That’s likely to occur about 8 seconds after the M1 Mac began to start up, and there’s an immediate silence of more than 5 seconds before the log entries start in earnest, with the announcement of Darwin Kernel and iBoot versions.
08.289794 === system boot: [UUID]
13.742101 kprintf initialized
13.742296 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:41 PST 2021; root:xnu-8019.61.5~1/RELEASE_ARM64_T6000
13.818102 iBoot version: iBoot-7429.61.2

The kernel prepares for tasks ahead by loading tools such as its validator for Image4 files, then security policy, including that for Apple Mobile File Integrity (AMFI), a key tool for checking executable code, Sandbox and Quarantine policy. Then comes the Secure Enclave Key Store.

At the same time as these, most of the hardware in the rest of the SoC is starting up and initialising, then reporting its state of readiness in entries by the kernel in the Unified Log. Among the most important of the hardware to be started up at this stage are the other CPU cores. Until about 14 seconds after the start of the boot processes, everything is run on a single core. Now the cores are started up in sequence using cpu_start(). On the M1 Pro/Max, these are listed in three clusters:

  • cluster 0 contains the two E cores, type 1, numbers 0 and 1;
  • cluster 1 contains the first four P cores, type 2, numbers 2-5;
  • cluster 2 contains the remaining four P cores, type 2, numbers 6-9.

The original M1 chip has just two clusters:

  • cluster 0 contains the four E cores, type 1, numbers 0-3;
  • cluster 1 contains the four P cores, type 2, numbers 4-7.

The next waypoint in security is enabling Gatekeeper, by which time the kernel is ready to access disk storage once the APFS file system has been loaded. This is reported in the log, giving the version number:
14.779543 apfs apfs_module_start:2568: load: com.apple.filesystems.apfs, v1933.61.1, apfs-1933.61.1, 2021/11/30

Before moving on, here’s my diagram showing some of the key points during this stage of the boot process.

SecureBootM1v2xnu

and here’s the link to the tear-out PDF version: SecureBootM1v2

Validating the SSV

As with other components discussed here, Apple uses two different terms, referring to the SSV as being the Signed System Volume (Platform Security Guide), and a sealed system volume (APFS Reference). Furthermore, when the hashes which seal such a volume are broken, that’s referred to as a broken seal rather than a broken signature. In any case, as I explained in the previous article, in this context what’s termed a signature is in fact the hash of the root seal or hash.

Sealing is performed by computing the hash for a leaf in the file-system B-tree, then computing hashes all the way up that tree to the root node, where the hash is known as the Seal. In APFS, the hashed B-trees of a sealed volume store both file system data and a hash of that data.

Using a tree of hashes brings important benefits, which are used when the SSV is mounted. The System is around 15 GB in total size; even if it consisted of a single file, computing one hash for the whole of that data would take a long time. An M1 Pro using Apple’s own library to compute a SHA-256 hash on a single 15 GB file takes around 9 seconds, an unacceptable delay in booting. When using a tree of hashes, once the root hash has been validated, further hash validation can progress down through the file system as it is accessed, so avoiding any noticeable delay when mounting the sealed volume.

Validating the actual root hash against a saved value, either directly or using their hashes/signatures, is an ingenious way of ensuring that the root hash never changes from the value computed when macOS was last updated or installed. Following any change to the boot snapshot forming the SSV, the hash of the root hash (its ‘signature’) is compared against that set by Apple. If they match, the snapshot is accepted as an identical copy of that version of macOS, and its root hash and signature are stored securely. By validating the actual root hash against either of those (obtained by iBoot), the kernel demonstrates that it still matches that set by Apple, without having to receive any signature from Apple.

The next task is to locate the boot volume, ready to mount its snapshot
14.827815 AppleFileSystemDriver AppleFileSystemDriver: publishing boot-uuid-media=disk3s1 (Macintosh HD)
14.828281 Got boot device = IOService:/AppleARMPE/arm-io/AppleT600xIO/ans@8F400000/AppleASCWrapV4/iop-ans-nub/RTBuddyV2/RTBuddyService/AppleANS3NVMeController/NS_01@1/IOBlockStorageDriver/APPLE SSD AP2048R Media/IOGUIDPartitionScheme/Container@2/AppleAPFSContainerScheme/AppleAPFSMedia/AppleAPFSContainer/Macintosh HD@1

That is followed by the longstanding and traditional announcement in the log of the BSD root, which I think goes right back to the first release of Mac OS X:
14.828295 BSD root: disk3s1
14.828300 , major 1, minor 13

The kernel next roots from the boot snapshot
14.898802 apfs apfs_vfsop_mount:2118: disk3 Promoter has been locked
14.899252 apfs apfs_vfsop_mount:2188: disk3s1 Rooting from snapshot with xid 183952.
14.899261 apfs apfs_log_mount_unmount:1828: disk3s1 mounting volume Macintosh HD, requested by: kernel_task (pid 0); parent: kernel_task (pid 0)
14.899386 apfs handle_snapshot_mount:885: disk3s1 mounting snapshot w/snap_xid 183952 and sblock oid 0x23263e

The first task with that snapshot is to validate its seal
14.901008 apfs is_root_hash_authentication_required_osx:369: disk3s1 Release kext with internal build: 0, ARV disabled: 0, booting xid: 0
14.901013 apfs is_root_hash_authentication_required:478: disk3s1 root volume, root hash authentication is required
14.901018 apfs authenticate_root_hash:546: disk3s1 successfully validated on-disk root hash

Note that successful validation of the root hash in the snapshot took a mere 0.000005 seconds – that’s 5 µs – hardly time to validate the whole 15 GB, just the root hash or Seal.

Other volumes which are validated and mounted after that include Recovery, VM, Preboot and Update, all from the boot container, together with some from the Apple_APFS_ISC container on the internal SSD.

Userland kernel management

Like the SSV, iBoot took the first steps towards validating kernel extension collections, but left it to macOS to perform their full validation, using kernelmanagerd. This is well-documented in the Unified log, as follows.
13.642280 kernelmanagerd Starting userland kernel management subsystem (KernelManagement_executables-262.60.4)
14.121399 kernelmanagerd Disabling kext auditing: We are on Apple Silicon
14.137320 kernelmanagerd Initializing with settings:
14.139417 kernelmanagerd finding bundles in repositories:
/Library/Extensions
/Library/Apple/System/Library/Extensions
/AppleInternal/Library/Extensions
/System/AppleInternal/Library/AuxiliaryExtensions
/System/AppleInternal/Diagnostics/AuxiliaryExtensions
/System/Library/AuxiliaryExtensions
/System/Library/DriverExtensions
/Library/DriverExtensions

After that, kernelmanagerd validates each of the Boot Kernel Collection with a log entry such as
14.164688 kernelmanagerd validating extension at /System/Library/DriverExtensions/com.apple.DriverKit-AppleUSBFTDI.dext

Once that’s ready to load as a collection, that’s reported
14.207370 kernelmanagerd Preparing collection load into kernel
and each kernel extension is acknowledged as it is loaded
14.316533 kernelmanagerd Received kext load notification: com.apple.kpi.bsd

When kernelmanagerd has finished, the kernel shuts it down, and no more kernel extensions can be loaded
29.313999 Kext loading now disabled.
29.314006 Kext unloading now disabled.
29.314009 Kext autounloading now disabled.
29.314011 Kernel requests now disabled.
29.314448 kernelmanagerd Kernel requested shutdown. Goodbye!

which completes much of the work of Secure Boot on an M1 Mac.