If you’ve got problems with your Mac and use Apple Support, one thing they’re likely to do is run
sysdiagnose on it: it even has its own keystroke combination, of Command-Option-Control-Shift-. (period or full stop). The same goes for iOS, watchOS and tvOS devices.
Did you realise that, contained within the large bundle of diagnostic files generated by
sysdiagnose is a history of your Mac’s (or other device’s) activity for up to the last 3 months, or more? In fact, there’s not just one copy of that information, but two!
This article is about that hidden log, how you can access it, and the valuable information it contains. I will illustrate this with a brief analysis performed on my own Mac, for the period from 28 June to 11 October.
Macs running Sierra and High Sierra, and iOS/watchOS/tvOS devices running recent versions of their operating systems, use a new unified log system which is quite unlike that on older Macs, running El Capitan or earlier. This new log collects huge amounts of information – sometimes more than 10,000 entries in a minute – from almost every process running, from the kernel up to each app.
These log files have to be maintained, and the system daemon responsible for that is
logd itself writes almost nothing into the unified log, but records its own activities in a private log, tucked away in the hidden folder /var/db/diagnostics. When
logd rolls the log, to open a new file to put log entries in, it records details of the twenty processes which have made the most log entries in the last log file (which it is just closing).
As the log is rolled every 10-12 hours, if your Mac is reasonably active, and
logd keeps its records for over three months,
logd‘s logs are a mine of information about your Mac, and its activities. I write its activities, because all entries, for all users, go into the same logs, and get analysed just the same.
So, after a few minutes spent retrieving and analysing the files kept on my Mac, I can tell you that on 26 July I worked hard in Xcode, and on 13 September I was also busy building apps, signing their code, and writing their documentation. I have no diary or records of those, but my Mac does.
Accessing the data
If you’ve got a Mac running Sierra or High Sierra, which has been in reasonably extensive use over the last few weeks or months, it will take you just a few minutes to browse your Mac’s activity records. Download MakeLogarchive from Downloads above, and look at its documentation. You first need to make a logarchive of that Mac’s live logs, then analyse
logd‘s logs inside that logarchive.
Open MakeLogarchive, and click on the Copy for Consolation button. In the Open File dialog, navigate to the (normally hidden folder) /var/db, and click on the Open button. MakeLogarchive shows all hidden folders, which makes this simple. You’ll next see the File Save dialog, prompting you to save the new logarchive. Save it in an accessible folder, such as ~/Documents, with a name which includes the .logarchive extension given. This will take a while, and the resulting ‘file’ (it’s actually a folder bundle) may well be large, around 500 MB in size.
Once that is complete, click on the CSV checkbox so that it is ticked, then on the Analyse button. In the File Open dialog, select the logarchive file which you just created, then click on the Open button. The text box below should then fill with a CSV-formatted summary of all the activity on your Mac going back several weeks, or for over three months. Click on the Save button and save that output, then import it into your favourite spreadsheet.
The figures kept in the log give a breakdown of the log load of the twenty processes which have used the log most during the period in which that log file was collected. Your spreadsheet gives the date and time that the log file was closed and analysed (the end of the collection period), the total amount of log entries in that whole log file, and the amount of log entries (and as a percentage of total) for each of those twenty processes.
Here’s a simple display of the data from just one of those log files, which reveals that in the log file closed at 17:13 on 29 June 2017, the process
com.apple.WebKit.Networking was responsible for almost half (42.3%) of all the log content collected over that period. The figure isn’t the number of log entries, but the amount of uncompressed log content, as far as I can tell.
Third on the list is my favourite Twitter client, Tweetbot, which remains open when I am using my Mac. Although Safari is well down in the list, with only 1% of the log load, that is because much of its work is done by WebKit components, including the top two in the list here.
This is a simple chart of the total size reported for each log file closed from 28 June to 11 August, the period covered in this analysis. That is a lot longer than is covered by log files, as they get cleaned up periodically to save space. But their figures remain in
Over this period,
logd analysed 154 log files, missing a few along the way, with an average total size of 97 million, which I think is their uncompressed size in bytes, so 97 MB each. (The standard deviation is 28 million, confirming the wide scatter in their size.)
In almost all those logs, the largest share of total log load was borne by
com.apple.WebKit.Networking, and this shows how its percentage share varied over the whole period. For several log file collection periods, it was the majority shareholder, sometimes by a long way. This reflects the amount of work that I do online, and that fairly constant access from Safari and other apps which use WebKit. Periods of low log load probably reflect those when I was away from my Mac, or asleep.
Add in now the log load of the kernel, and you can see how that follows a very different pattern. The kernel is usually fairly quiet in the unified log, and its messages normally of importance. It doesn’t indulge in idle chatter! You’ll notice a cluster of high loads from the kernel from 25 September, which coincides of course with the release of High Sierra, and my several attempts to install it on this iMac.
I have now added more processes which are contained in
logd‘s logs. Because this chart is busy, you can click on it to enlarge it into a single page, and can zoom into that too.
When I ran these analyses, I saw some surprising processes listed: I had expected to see Apple’s Xcode, as it is such a hefty app, but here I show two other processes which I had never expected,
codesign and Nisus Writer Pro.
codesign is the command tool used to sign the apps which I have been developing, and I use Nisus Writer Pro to write documentation for them.
You can see bursts of log load from those three processes which are closely co-ordinated, and represent the periods when I have been developing apps in Xcode, signing them using
codesign (called from Xcode, not directly by me), then updating their documentation. There was a peak of activity on 3 October, when I pushed out many updates to fix my code-signing problems. Those put little load on Xcode, but involved lots of signing and documentation updating.
I have explained more technical details in my article here yesterday, which explains my concept of log load, and the factors which affect it, among other things.
Although the information contained in the
logd logs is very detailed, and very extensive, it doesn’t discriminate between users, and doesn’t show what anyone was doing with those processes – you can’t tell whether someone was accessing particular websites or files, for example. It does, though, give great insight into the activity of different processes on that Mac over a surprisingly long period. On this iMac, my unified log files go back for around 20 days, which I thought was amazingly long. This information goes back about five times that, which is extraordinary.
One immediate use for this information is when trying to work out which specific log files to examine, when diagnosing a problem (or bug). Before this, you had nothing to go on other than the symptoms reported. Now you can look back and work out exactly which log files are likely to contain most entries from the process(es) you are interested in. If they were still in the live log when you made the logarchive, you can then browse them using Consolation 3 (from Downloads above, version 3 only) (as far as I am aware, Apple’s Console is unable to browse specific log files, even in High Sierra.)
This is a very simple example of what you can do with
logd‘s log, when accessed and analysed in this way. I have no doubt that Apple finds them invaluable when it analyses the output from
sysdiagnose. You too can now access that previously hidden information. Please use it wisely.
I will also be developing better tools over the course of the next few days and weeks.