Explainer: Logs

Like a ship’s log, a computer’s log is a record of events, normally a time-ordered sequence of messages written so those events can be analysed to determine the cause of bugs or crashes, investigate security breaches, assess performance, and for many other purposes.

Prior to macOS Sierra, Mac OS X followed Unix conventions, in that its logs consisted of various text files stored in obvious places, such as the Logs folder in each of the standard Library folders. Key entries were scattered across several different files, including system.log and console.log, which were regularly rotated as part of daily housekeeping. Many individual apps and processes also wrote their own log files, which didn’t have to be in standard locations. Each log entry consisted of writing a message in plain text, which was then tagged with a time and other details.

Although the vestiges of that old system still remain, the vast majority of entries written using standard system calls now go into the Unified Log, which is stored in compressed binary form and far more detailed. Apple’s main aims in introducing this were to consolidate (almost) all log entries into a single location, and to make the process of writing a log entry both lightweight and efficient.

Currently, each log entry consists of up to 27 different fields, including its date and timestamp, message type and category, subsystem, multiple ID numbers for thread and process, path and UUID for process and sender, and specialised fields for activities and Signposts, the latter being used for performance monitoring. Those are stored in binary format and compressed, in proprietary-format tracev3 files. Although their format has been reversed, in practice the only reliable means of access are the Console app and the log command.

Despite considerable additional information about each event recorded in the Unified log, it’s far more efficient than the traditional mechanisms, and can write tens of thousands of entries every second without imposing any detectable overhead. Unfortunately, widespread adoption by subsystems within macOS mean that the Unified log is dominated if not overwhelmed by incessant system chatter, making access a real challenge.

Many processes in macOS now routinely write details of most of their actions and activities to the Unified log. Both Console and the log command support the use of search predicates to filter the entries found. Unfortunately, writing effective predicates isn’t trivial, and requires detailed knowledge of the processes that you’re interested in.

Another stumbling block for users is Console. Formerly an accessible browser of logs of the recent past, the new version to accompany the switch to the Universal log only gives easy access to the current log message stream, not entries already saved. This effectively renders it useless for popular tasks such as checking recent errors or crashes, or simple everyday monitoring such as checking whether recent Time Machine backups completed successfully. It is possible to use Console to browse past log entries, but to do so you must first export the active log to a logarchive, and browse that with Console. Even using the app to watch the streaming log is demanding, as without an effective predicate filter entries scroll past too quickly.

In keeping with its privacy protection policies, the Unified log automatically censors content in many messages with <private>. In many places, this renders its entries meaningless. Although it’s possible to turn this privacy protection off, Apple has deliberately made that difficult, and it only applies prospectively. Log entries written when normal privacy protection is in force don’t save the censored content, which can therefore never be recovered from the log. This is a common problem when trying to examine recent crashes or other events.

Since its introduction, access to the Unified log is only available to admin users, making it hard to analyse problems when running as a regular user.

That said, the Unified log is a mine of information, giving a detailed moment-by-moment commentary on everything that macOS is up to. It reveals innermost secrets, such as how macOS schedules and despatches background tasks, the long and elaborate exchanges which make iCloud work, and how Time Machine backs up to APFS.

I offer four major tools here which ease access to the Unified log. For Time Machine, T2M2 analyses its entries in the log to measure its health and performance, and report any errors. For iCloud, Cirrus includes a dedicated browser to show log entries for the subsystems involved, and can reveal problems in them. Mints includes specialist log browsing for seven separate features: iCloud (similar to that in Cirrus), TCC (privacy protection), Time Machine, the App Store, DAS (background) scheduling, any boot during the previous 24 hours, and Spotlight search. Ulbow is my general log browser, which includes innovative frequency charting and other tools which are unique to it. If you prefer a dialog-based browser, there’s also my more traditional Consolation 3. Each is free, of course, and supports all versions of macOS from Sierra to Monterey. Help files for Ulbow and Consolation provide extensive information about the Unified log, and how to get the best out of it. I also have dozens if not hundreds of articles in this blog explaining various aspects of the Unified log.