How some log entries vanish sooner than others

I’ve started looking at how M1 Macs run iOS/iPadOS apps, a subject I’ll be returning to in the coming days and weeks, for which I’ve been immersing myself in the unified log. One morning I captured a log excerpt of some interesting behaviour, and went back later that day to re-examine that earlier section of log. Only all the entries I was interested in had vanished. Why?

To understand what happened, you need to appreciate how different the unified log is from conventional logs.

Prior to the introduction of the unified log in macOS Sierra, logs worked much as in other Unix systems. Entries were appended at the end of the text file forming the current log. Periodically, log files were rolled, archiving the current file in favour of a new active log, and removing the oldest archived log file(s). Once an entry had been written, it could be retrieved from that log file until that file was eventually removed, as nothing tampered with the log, you hoped.

When you browse any instant in the unified log in Big Sur, the entries you see are a composite drawn from three or four different log files, in folders within /var/db/diagnostics/: binary compressed .tracev3 files in HighVolume, Persist, Signpost and Special. Log rolling and maintenance is performed by logd, which periodically deletes the oldest files to keep folder sizes within limits. logd also weeds log entries in those files stored in the Special folder so that they gradually shrink with age until there are no log entries left in them, when they’re deleted.

Many of the entries that are most significant in the log are those with the category Info, which is used extensively by the more interesting sub-systems such as LaunchServices for its most informative entries. Unfortunately, standard log policy is that all Info log entries are saved in the log files in the Special folder, and are therefore weeded during the hours after they have been written. Their loss seems to start within a few hours, and – on my Macs running Big Sur 11.2.3 at least – this is complete by the time that entries are about 7 hours old.

Look back through your logs using Ulbow or Consolation, and you’re very unlikely to see any Info entries older than about 7 hours. That explains why important log entries which you see a few minutes after they were entered into the log may have vanished a few hours later. This is particularly significant for anyone trying to perform forensic analysis on logs: be aware that they quickly cease being complete records shortly after they have been written.

The user does have some control over logd‘s maintenance activities, by setting preferences in /Library/Preferences/Logging, but none seems able to determine when its weeding starts, nor whether Info entries are saved into Persist rather than Special logs. For the moment at least, there seems no avoiding the vanishing of important log entries, it’s just the way the unified log works.