It has been well over a year since Apple told us much about the new unified log which it introduced in Sierra, and continues in High Sierra. At the time, at WWDC in June 2016, it was all rather novel, and didn’t arrive until the release of macOS Sierra the following September (2016).

Inevitably, the focus at WWDC was on explaining to developers how to write to the new log, and precious little was revealed about the log itself. This article is a small step forward in understanding the unified log, which is used in iOS, watchOS, and tvOS too.

Traditional logs, found on most conventional Unix systems, are a bunch of text files stored in several different folders. They’re easy to access, entries are relatively sparse, and they are separated into different voices: printing system errors and other log entries will normally go into one or more print logs, for example. More significant entries may be written into a system or console log, but there is no centralised log in which everything is recorded.

Apple’s grand design eliminates almost all other logs (macOS Server and some services still maintain their own, though), and fills a single unified log with many hundreds or thousands of log entries every minute. As Apple’s team stated in their 2016 presentation “we want as much logging on all the time as possible”. And we’ve got it.

Log folders and files

Log contents are stored as tracev3 files – an undocumented compressed binary format – in /var/db/diagnostics/Persist/ and /var/db/diagnostics/Special/. Other folders in /var/db/diagnostics/ and /var/db/uuidtext/ contain ancillary information for those logs.

/var/db/diagnostics/ consists of:

• The folder Persist, which contains the main tracev3 files for the log as saved to disk.
• The folder Special, which contains additional tracev3 files holding supplementary log content.
• The folder timesync, which contains timesync files numbered in order, extending back to the Sierra upgrade over a year ago. These contain binary data about time synchronisation between log events and the high-resolution Mach system clock.
• A series of plain text files named logdata.statistics.0.txt and others in numeric sequence – the log logs.
• Other data files such as shutdown.log containing predominantly plain text details of significant log events.
• version.plist.

/var/db/uuidtext/ contains a series of folders, named 00 to FF, each containing files named by UUID for specific log events, which extend beyond normal log entries. These include crash events and other items which overflow single line entries in the log. There is another folder named dsc which contains similar files.

Log management

Logs are managed by a background daemon logd, which is undocumented but responsible for compressing and writing log entries, and for maintaining the log files. Periodically it rotates log files, and weeds the entries in those deemed Special until they are eventually reduced to nothing and removed. Another daemon involved is diagnosticd, which can feed a live log stream such as that viewed in Apple’s Console utility, or in the log stream command. Both logd and diagnosticd run as root.

It’s not immediately clear why the log separates entries into Persist and Special log files, as they both contain entries of the same types. However, Special log files are weeded over time to ‘compress’ them, which is one of logd‘s functions. Older Special log files thus get progressively smaller as they age, until they contain almost nothing, and are automatically deleted.

Persist files are kept intact until they reach an age of about twenty days, or more, when they are simply deleted during log file rotation.

Persist log files are named using hexadecimal numbers in sequence, starting from 0000000000000001.tracev3 when that Mac is first started up in Sierra or later, and switching to a new log file when there is around 10.5 MB of data in the current file, although rotation occurs at irregular intervals, so the time periods and sizes vary.

Special log files are named using similar hexadecimal numbers, but appear to reuse recent numbers as well. They vary more widely in period and size, and their coverage may not be continuous.

When you use Console, Consolation or log to view live log entries (streamed live via diagnosticd, or historic entries in the current system log), those shown merge entries from the Persist and Special log files. However, the log command and Consolation also allow you to browse individual log files, in which case only those from the selected log file are shown. To get a full view of log entries for a given period, it would thus be necessary to view both the Persist and Special log files covering that period, when browsing them individually, such as in a logarchive.

Log logs

Of the other files written into the log folders, by far the most interesting and accessible are those named logdata.statistics.0.txt and similar. These are logs of the log files, written by logd as it performs its log file maintenance. They are numbered using the convention for log files, with the newest having the lowest number (0), and the oldest having the highest number.

Entries detail the following types of event:

• Memory Rollover – these appear to mark the writing of a log file to disk from entries stored in memory.
• File Rotate – rotation of Persist or Special log files to store future log entries in a new file.
• UUID and Persist purge records – removal of log or supplementary files, listed by file name.
• Special Periodic Compaction – weeding of Special log files to reduce their size, and eventual deletion.
• Persist Periodic Compaction – deletion of oldest Persist log files.

Each of these is accompanied by a datestamp for the event, and the total number of log entries affected by that action. Those involving files on disk also report the file name.

Memory Rollover and File Rotate events are accompanied by a statistical summary of entries in that log file, listed by freqency of the process writing each entry. For example:
6796713, 19.2, /System/Library/PrivateFrameworks/CalendarAgent.framework/Executables/CalendarAgent
4779552, 13.5, /usr/libexec/trustd
3026552, 8.6, /System/Library/Frameworks/WebKit.framework/Versions/A/XPCServices/com.apple.WebKit.Networking.xpc/Contents/MacOS/com.apple.WebKit.Networking
2289064, 6.5, /usr/sbin/wirelessproxd
2115299, 6.0, /usr/libexec/sandboxd
1938784, 5.5, /usr/libexec/opendirectoryd
1888672, 5.3, /usr/libexec/coreduetd
971480, 2.8, /usr/sbin/securityd
903072, 2.6, /System/Library/CoreServices/powerd.bundle/powerd
845221, 2.4, /System/Library/PrivateFrameworks/ApplePushService.framework/apsd
776128, 2.2, /usr/libexec/DuetHeuristic-BM
745384, 2.1, /System/Library/PrivateFrameworks/CloudKitDaemon.framework/Support/cloudd
744848, 2.1, /kernel
668672, 1.9, /Applications/Tweetbot.app/Contents/MacOS/Tweetbot
667664, 1.9, /Applications/Safari.app/Contents/MacOS/Safari
584424, 1.7, /usr/libexec/UserEventAgent
582590, 1.6, /System/Library/PrivateFrameworks/IDS.framework/identityservicesd.app/Contents/MacOS/identityservicesd
523984, 1.5, /System/Library/PrivateFrameworks/UserActivity.framework/Agents/useractivityd
426992, 1.2, /System/Library/PrivateFrameworks/GeoServices.framework/Versions/A/XPCServices/com.apple.geod.xpc/Contents/MacOS/com.apple.geod
396328, 1.1, /usr/libexec/sharingd
The first figure given is the total number of log entries, and the second is the percentage of the total written by that process.

Interestingly, logd itself doesn’t appear to make any entries in the unified log, apart from very rare errors and faults.

Access

In making the tracev3 log file format closed and proprietary, Apple has ensured the entire log system is closed, and dependent on Console and the log command. Where log files need to be moved for access by another system, such as when browsing iOS logs, or performing forensic analysis, Apple advocates use of the logarchive format, which it describes as “portable”. However, Apple hasn’t documented that format either, although it has now been reversed to enable other apps to create valid logarchives and to access individual log files within a logarchive.

While the unified log is potentially a very powerful tool for advanced users, system administrators, developers, and many others, it is now limited in its value by the tools available to access it. As a result, most of those users who used to be familiar with El Capitan’s Console and logs are now unable to get anything useful from Sierra’s unified log, and have abandoned trying to access it. It’s all very well capturing the information, but if you don’t provide good tools to facilitate access and analysis, that information is largely wasted.

Probably the most common use for the unified log now is to generate voluminous sysdiagnose output to send to Apple.