One of the mysteries of the Unified log since its introduction almost ten years ago in macOS Sierra has been how it keeps such extensive records of what happens between the start of kernel boot and the user logging in. This must have been sufficiently challenging before Catalina, but since then the separate Data volume, where log files are stored, has been locked away by FileVault until the user’s password enables it to be accessed.

Log entries are initially stored in memory, and from there most are written to tracev3 files in their respective directories in /var/db/diagnostics. logd is responsible for that, and for maintaining the contents of that directory. Ironically, logd maintains its own old-style log in text files inside that directory, but its terse entries there reveal little about how it does its job. What they do reveal is that logd has an assistant, logd_helper, with its own old-style text logs that are a bit more informative.

In Apple silicon Macs, logd_helper apparently connects every few minutes to five coprocessors, SMC (System Management Controller), AOP (Always-On Processor), DCPEXT0, DCPEXT1 and DCPEXT2 (Display Co-Processors). There appears to be nothing equivalent in Intel Macs. It also conducts ‘harvests’ shortly after it has started up, so solving the mystery of where all the log entries are saved prior to unlocking the Data volume.

Soon after the start of kernel boot, once the Preboot volume is generally accessible and logd is running, logd starts writing log entries to temporary storage in the Preboot volume. You can see that in the path /System/Volumes/Preboot/[UUID]/PreLoginData, where there’s a complete diagnostics directory, and one to store longer log entries in the warren of directories in uuidtext. Those are identical in layout to permanent log storage in /var/db.

Shortly after user login, with the Data volume unlocked at last, logd_helper is started, and it merges log files from Preboot/[UUID]/PreLoginData/diagnostics and those files in Preboot/[UUID]/PreLoginData/uuidtext into permanent log storage in the Data volume in /var/db/diagnostics and /var/db/uuidtext, a process it refers to as harvesting. logd_helper can also harvest from entries stored in memory.

Once this merger has been completed, log directories in Preboot/[UUID]/PreLoginData/diagnostics are left empty, as are logdata.statistics files there, making the log record of kernel boot complete, right up to unlocking of the Data volume.

That explains how tens of thousands of log entries can still be recorded faithfully in a Data volume that can’t be unlocked for some time yet.

Once normal logging to /var/db/diagnostics is running, logd maintains the tracev3 files containing log entries there. Its goals appear to be:

• in the Persist folder, file size is normally 10.4-10.5 MB, although it may be smaller when truncated by shutdown;
• Persist files are removed with age to maintain a typical total size for that folder of just under 530 MB in just over 50 files, bringing the size of the whole diagnostics folder to between 1-2 GB;
• in Special and Signpost, log file size is normally 2.0-2.1 MB when closed, but entries are weeded progressively until each file is empty and can be deleted;
• timesync files are less than 1 KB;
• HighVolume is seldom if ever used.

The overall effect on historical log entries is that those in Persist files are rate-sensitive and removed sooner when log entries are written more frequently. However, selected entries in Special files may last considerably longer, but become less frequent with age. A few of those may be retained for hours or days longer than the oldest in Persist files. I have no insight into the rules that logd follows when deciding when to weed entries from Special files.

Extended entries stored in the warren of folders in /var/db/uuidtext are purged periodically on request from CacheDelete, as with other purgeable storage, at least once a day. That should ensure that the contents are only retained while they’re still referred to by entries in the log files.

As far as I’m aware, the user gets no say in the size limits imposed on log storage, and there’s no option to increase them to allow logs to be retained for longer. However, as both /var/db/diagnostics and /var/db/uuidtext folders should be backed up by Time Machine and most third-party utilities, you can always analyse those backups when you need to check older log entries.