The internet is nothing without search engines like Google. Imagine what it would be like if you could only monitor new postings in real time, trying to sift through the vast torrent of data. Or if you could only dump the last couple of hours of all fresh postings, then try to rummage through them to find what you want.

Sierra’s new log system has been a bit like that. The two tools which Apple has provided work in those two modes. The app Console can be used to filter real-time log entries, or to browse complete dumps in logarchive format generated using the command tool log. Working with those new logs is like trying to use the internet without search engines such as Google.

If there is one lesson which we learned about information during the twentieth century, it was that collecting and storing data is useless without powerful tools to retrieve and analyse it.

My latest beta-release of Consolation may seem to be another slight twist in a tool for nerds. Using the Console app has been a good criterion for identifying the nerds among Mac users. After all, if you choose to use a Mac, you’re casting your vote against command lines, logs, and all the tech underpinnings which you’d come to live with in Unix or Linux.

Sometimes, though, writing software is like painting (an idea which Paul Graham has developed in his book Hackers and Painters). One minute you’re toiling away trying to get some important detail right, then you step back, and the whole thing just pops. The painting has developed a look of its own, or the app has enabled some truly novel uses. That is what just happened to Consolation.

Let me run through some scenarios to get you thinking of what you can do, armed with Consolation and a spreadsheet or other app to analyse data.

Some years ago, I wrote a system to be used in some trials looking at human fatigue when performing computer-based tasks. The software ran a radar simulator, which at random or programmed times placed a new object on the display, which the operator had to interrogate and respond appropriately to. We wanted to measure the time that it took for each operator to take action on each event, and whether their action was correct.

I wrote an internal log system which the experimenters could then pass into their statistics apps to analyse response times, etc. Adding that back-end formed a major part of the development, and like the rest of the system it had to run flawlessly. If after several days of trials the log became corrupted, or couldn’t be analysed, that would have been disaster.

Now, with Sierra and Consolation, I could leave macOS to take care of all the logging, and I would save around a third of the total development time.

At the moment, I’m spending quite a lot of time authoring hypertext documents using Storyspace. I’m very keen to know how readers use hypertext documents which I have written. Most readers are only too happy to tell you what they liked and didn’t like, but those are subjective impressions. When I have put a lot of time and effort into a particular item or feature, it would be very valuable to know how often that item was visited (that is currently straightforward to implement), and for how long.

For a commercial product, careful and objective reader-testing is very important, but also very difficult without a lot of additional effort, which is then stripped out when you create the final release version.

If you write any sort of code, from simple scripts or ‘macros’ to whole apps, one of the fundamental requirements for development is to be able to write simple messages out to some sort of debugger or monitor. But many scripting systems give no write access to your Mac’s logs. This means that you have to rely on your debugging environment, and hope that bugs will not damage that environment to the point where it ceases to help. Tools for performance analysis, to work out where your code runs inefficiently, are even trickier, as most disrupt the runtime environment so much that their results are hard to interpret.

For each of these, and many other purposes, you need two tools: one to make log entries, the other to convert structured log data into a form which can be imported into analysis apps such as spreadsheets and databases.

For the moment, very few apps or creative environments give direct access to Sierra’s new logs. AppleScript doesn’t, and it looks increasingly unlikely that it will ever do so. It does, though, allow you to run command tools, as do a surprising number of apps which incorporate scripting features. If your language or app can run a command tool, then you can use Blowhole (see Downloads in the menu above) to make a log entry for you.

This is not as quick and efficient as making a log entry directly from a language such as Swift. When called in a compiled Swift for loop, an app can make a fresh log entry every 2 microseconds – that’s an entry rate of half a million log entries per second, although clearly that rate could not be sustained for long. Called from a series of lines in AppleScript, a script can make a fresh log entry every 50 milliseconds via the blowhole command, for an entry rate of twenty log entries per second.

The latest beta-release of Consolation 2 can now export data from Sierra’s log to CSV format, which can then be imported into very many apps, including spreadsheets (Microsoft Excel, Numbers), databases (FileMaker Pro and most others), statistical analysis apps (R and more), and general maths environments (Matlab, Mathematica, and more).

One of the most exciting moments over the past week has been to convert the last two minutes of log entries and open them in Excel. Seeing that information laid out in structured format was a revelation of its potential. We now have the tools – all we need is the imagination.