Combining my free command tool blowhole with Consolation allows any app which can run a command to write to Sierra’s new log, and those log entries can then be retrieved and analysed. I have already made suggestions as to how this could be used to debug and test documents created using Storyspace or Tinderbox. You could also use this with AppleScript, and more.

Normally, we use log entries to diagnose problems, by inspecting entries which provide clues as to what is going wrong. With the new log system, blowhole (or any other method of writing to the log), and Consolation (or the log show command in Terminal), we can now do much more – if we can analyse the log entries, that is.

This article is a first look at that last problem. Let’s say that we have instrumented an AppleScript, or a Storyspace hypertext book, collected dozens or even hundreds of log records made via blowhole, and exported them into a text file. How can we usefully analyse them?

Log extract formats

The log show command, thus Consolation which relies on it, offers three different formats for log extracts.

Syslog format is most popular when using the logs for diagnosis, and looks like this:
2017-02-20 16:23:34.595557+0000 localhost backupd[28986]: (TimeMachine) [com.apple.TimeMachine.TMLogInfo] Backup completed successfully.

It doesn’t use fixed field/column widths, nor separators such as commas or tabs. Each line consists of a fixed format timestamp, two spaces, localhost, one space, process name[processID]:, one space, optional (sender) and a space, [subsystem.category], one space, eventMessage. That is hardly ideal for trying to import into another app.

Default format is drawn most directly from the internal format of the log database, and looks like this:
2017-02-20 16:23:34.595557+0000 0x1c605ff Info 0x0 28986 backupd: (TimeMachine) [com.apple.TimeMachine.TMLogInfo] Backup completed successfully.

Again, it offers us no help with fixed field widths or separators. Each line consists of a fixed format timestamp, hex threadID, messageType, activity (0x0 usually), processID, process name, optional (sender), [subsystem.category], eventMessage. That is little better than syslog format for importing into an app.

JSON format is probably the least likely to be used for anything else, as each log entry is written as a JSON object, such as:
{
"category" : "TMLogInfo",
"processImageUUID" : "4A459310-B20B-333F-B101-C4496F99DA66",
"processUniqueID" : 228279,
"threadID" : 29754879,
"timestamp" : "2017-02-20 16:23:34.595557+0000",
"traceID" : 1721366688328581380,
"messageType" : "Info",
"senderProgramCounter" : 400527485,
"processID" : 28986,
"machTimestamp" : 1233550103688069,
"timezoneName" : "Europe\/London",
"subsystem" : "com.apple.TimeMachine",
"eventMessage" : "Backup completed successfully.",
"senderImageUUID" : "151DDC25-1634-35FD-8BA2-C51EA520C4BB",
"processImagePath" : "\/System\/Library\/CoreServices\/backupd.bundle\/Contents\/Resources\/backupd",
"senderImagePath" : "\/System\/Library\/PrivateFrameworks\/TimeMachine.framework\/Versions\/A\/TimeMachine"
}

JSON is ideal if you are going to process the data yourself, particularly using JavaScript, and has specific support in two databases, MongoDB and Apache CouchDB. If you want to follow those up, MongoDB is available from here, and the friendly Robomongo front end to it is here. Apache CouchDB is available from here.

Importing to common apps

I have tried importing from each of those three formats into a range of apps which you might consider for manipulating or analysing data from log extracts. The results are dismal.

Apple’s Numbers spreadsheet lacks the import features of Microsoft Excel, and simply imported the Syslog and Default formats into a single column, with no options to do anything different.

Microsoft Excel (15.31) does at least give you some options to split the data into columns, but quickly shows its limitations, particularly in the face of variable-width fields without a delimiter. Syslog-format files are split into date (as date), time (as text, as there is no time option), the useless localhost entry, then everything else is dumped into the last column.

The Default format fares slightly better, as it can also turn threadID, messageType, activity, and processID into separate columns before dumping the rest into a single column.

Like many databases, FileMaker Pro 15 will import XML data, but does not yet stoop to JSON. My efforts to convert from JSON to XML have been frustrated by a wide choice of tools which are generally useless. The two which came closest are JSON Wizard from here, for $9.99, and JSON Editor from the App Store, which is packed with other options, many of which have not been updated for years.

JSON Wizard has a reputation as being the best editor, and I can well believe it. However when you want to export to XML, it does nothing, so presumably that is an advertised feature which has yet to be implemented. It will, though, export to a Property List (.plist) file, which is identical to that which can be exported from JSON Editor. This looked promising, as Property Lists are valid XML, but FileMaker Pro declined to import them into a database.

I’m sure that there is a way, but for the moment I have just grown tired of apps like spreadsheets which only import ancient data formats and ignore modern ones, and browsing long lists of apps which I just know are going to turn out to be utter disappointments.

I will be back once I have a solution which does not rely on duct tape or missing features.