This week I’ve been chasing problems which I stumbled across when using Ulbow to browse my log. That revealed that my Mac had two stuck Time Machine snapshots, which were triggering lots of log entries each time that there was an automatic backup which tried unsuccessfully to ‘thin’ or delete them. As the snapshots were conveniently datestamped in their names, I wanted to look back in the log to see what might have gone wrong when they were first created. Although that was only a few days before, I was shocked to discover that the unified log had already rolled those entries out. The last time that I looked at how far back the log ran, it was up to 20 days. So how come mine only extended back 4-5 days? This article tries to explain.

The background service which performs all log maintenance is logd, which has a sparklingly informative man page which even includes ASCII graphics. This explains how logd reads pages of memory, which are shared with all processes using the log, stores them in a memory buffer, and writes most out to storage on the Data volume, to form the log’s compressed binary tracev3 files.

logd also maintains those files. Once the current log file reaches around 10.5 MB, logd closes that and opens a new one. It also maintains those log files within the folders inside /var/db/diagnostics, deleting them when they get old. Regular log entries are normally written to the tracev3 files in the Persist folder; Signposts are stored in separate tracev3 files in the folder of that name; log files in the Special folder are progressively weeded over time, until they’re finally empty and deleted.

There doesn’t appear to be any way of changing when the log files in the Persist folder are removed: although there are methods of configuring what gets logged, logd seems to make the decision as to when to remove its oldest log file on the basis of the size of the Persist folder. Typically, it maintains about 52 tracev3 log files there, for a total folder size of just over 500 MB.

Back in the days of Sierra, when the unified log was first introduced, most healthy Macs kept their log files for up to 20 days before they got purged by logd. As the quantity of log entries has steadily increased since, the period covered by the log has reduced. On this Mac, when running Mojave, it was around 10 days, but upgrading to Catalina has brought that down to only 4-5 days, which is comparable to the logs before they were unified in Sierra. The less healthy your system is, the more frequently entries are likely to be made in the log, and the shorter lifespan they will have before deletion.

This is important for forensic work, and when trying to diagnose problems which occurred in the past. The good news is that, unless they’re excluded, the default behaviour in Time Machine is to back up the whole log folder from /var/db/diagnostics. However, macOS doesn’t provide a tool for reading backups of logs: Console and the log show command (and my log browsers Ulbow and Consolation) can only read the active system log, or logarchive bundles. If you want to access logs from a backup, you’ll have to turn the files and folders into a logarchive, a format which Apple doesn’t appear to have documented.

In the final article describing my journey through logs and snapshots, I’ll explain how to turn a log backup into a logarchive.

Meanwhile, the answer to my original question is that macOS keeps around 52 tracev3 log files in /var/db/diagnostics/Persist, so the active log extends back as long as it has taken to write those, which could be anything from a couple of days to a month, depending on how rapidly entries are being made. Wouldn’t it be useful to be able to set how large that folder could become, perhaps?