The new unified log in Sierra and High Sierra contains a great deal of valuable information, which can cast light on software and hardware problems, and may be of use in forensic examination too – if you can access it.

Currently, the log command, Console, and Consolation support two sources of log data:

• live logs, stored in /var/db/diagnostics and /var/db/uuidtext on the current startup volume;
• logarchives, which are created from live logs by the log collect command, and can be opened and examined on any Mac.

Logarchives are the standard means of accessing logs from iOS, watchOS, and tvOS devices too.

This is a problem if all you have are the former live log files on a drive, for example from a dead Mac, or recovered from a Mac or iOS device which is being assessed. Apple does not document the tracev3 format used to store logs, nor the logarchive format, nor does it provide any utility which is capable of converting former live log files into readable logarchives.

I previously detailed the structure of files in a live log, which can be seen in their hierarchies in the following two screenshots.

According to the BlackBag Training Team, to convert live logs into a logarchive, all you need do is “export the contents of /private/var/db/diagnostics and /private/var/db/uuidtext to a folder on your desktop”, then rename that folder as a .logarchive bundle. You can then open them using Apple’s bundled Console.app. This does indeed work, but Consolation, which calls the log command to access data from that logarchive, cannot browse them, returning an error code 64 (which inevitably doesn’t seem to mean anything helpful).

This is because the logarchive which you think that you have created is not in fact a logarchive in the current (version 3) format. Those bundles have quite a different format, summarised in the following Finder view.

At the top level of the logarchive bundle are the numbered folders 00…FF and dsc from the original /var/db/uuidtext folder, and the folders named Extra, Persist, Special, and timesync. Extra contains the logdata.statistics.0.txt, etc., files, shutdown.log, and version.plist.

Two additional files which are of importance are at the top level of the bundle: logdata.LiveData.tracev3, which seems to contain a snapshot of the live data at the time, and Info.plist, which contains important and undocumented settings for that logarchive.

As assembling a properly-structured logarchive version 3 bundle can get tedious, I have written a tool named MakeLogarchive, which you point at a /var/db folder containing diagnostics and uuidtext folders. It will then copy the contents and generate a correctly-formatted logarchive V3, with the exception of the crucial Info.plist settings.

You can obtain MakeLogarchive from here: MakeLogarchive

Although its two buttons have different titles, currently they do exactly the same thing. Click on either of them, select the /var/db or equivalent folder containing the diagnostics and uuidtext folders, then select where to save the resulting logarchive.

The snag for the moment is that, without its Info.plist, that logarchive cannot be accessed using Consolation. You can access it with Console, though.

When you use MakeLogarchive, you will also discover the biggest problem in trying to work with this 500 MB logarchive bundle: it typically contains over 25 million log entries. One potential solution is to use the log command to export them in a format suitable for importing into a (heavyweight) database, but the only such format supported is a minimalist implementation of JSON, which often proves tricky to import, and I dread to think what that would be like for so many entries.

Apple’s Console seems happy to try to handle all these log entries, and let you browse them, but has no features to support any meaningful analysis, and no means of exporting entries to other apps such as databases or spreadsheets, as far as I can see.

Consolation, to my delight, seems relatively happy to support its rich range of browsing and other features, although you shouldn’t try to load the whole logarchive!

My next goals are to reverse engineer the logarchive V3 format sufficient to give Consolation good access to these logarchives, and to provide it with a means of being more selective in the tracev3 files which it accesses within a logarchive. For the time being, I’m afraid that you’re stuck with the log command, or Console, but at least you can now turn raw log files into a logarchive without having to do so manually in Terminal.