We get few glimpses inside Apple. I think I’m more familiar with the lift buttons in its new campus than I am with how macOS engineering works. Perhaps that is just as well, because last week has given two disturbing insights into the nature of the beast.

The first was, of course, the very mundane bug discovered by Matheus Mariano, in which High Sierra’s shiny new Disk Utility stored the plaintext password in place of a password hint. It’s mundane because it is the sort of bug which is very common when coding.

Daniel Martin (@dmartincy) tweeted his dissassembly of the code from StorageKit, confirming our suspicions that the bug arose because the engineer writing the code had simply used the wrong variable. In the dropdown sheet for password entry, verification, and provision of the password hint, the code stored the password in place of the hint in the dictionary containing values obtained from the sheet. Martin ascribed this to a copy/paste error, which is possible, but I think that it is more likely to have arisen through Xcode’s autocomplete feature.

When you type in text in Xcode, it automatically suggests autocompletions, which can be very helpful and a great aid to productivity. Let’s imagine for the moment that the password string variable was named aNewPasswordStr, and the hint string was named aNewPasswordHintStr. The engineer starts typing the characters aNewP, and Xcode pops up a long list of possibilities, with those two close to the top. Instead of selecting aNewPasswordHintStr for the string to be saved as the hint, aNewPasswordStr is at the top of the list, the engineer presses Return, and the bug is set.

This is, in coding terms, a mundane error in some pretty mundane code.

What is truly remarkable is that such a mundane error remained in that code not just through code review and in-house testing, but through beta-testing, production of the Golden Master, and into the first release of what Apple claims is the world’s “most advanced operating system”. Neither is this in an insignificant section of code in an unimportant component of macOS. It is that which transforms a mundane bug into a major gaffe, and shows how failures must have occurred at many levels.

There are many questions which Apple should be asking itself in its investigation of this remarkable failure in its quality management of a flagship product. One significant factor which undoubtedly made it easier for this bug to slip out into the final release, is that this code only exists in macOS, and not iOS. Not that iOS 11 seems to have been any better managed in terms of its quality: in less than a month from release, it has already had two urgent updates to bring it to 11.0.2.

The really bizarre twist with High Sierra, though, is Apple’s handling of this and other major flaws with its urgent ‘Supplemental Update’. That update didn’t increment High Sierra’s minor version to 10.13.0.1, perhaps, but left the version unchanged at 10.13. Not only that, those apps which are replaced by this update, including Disk Utility.app, retain the same version and build numbers as in the original release of High Sierra, although they have clearly changed.

It’s as if Apple is pretending that the original release of High Sierra never contained those dreadful bugs in th first place.

It will also lead to all sorts of confusion, as outwardly the two versions of Disk Utility appear identical, but behave very differently. The bug was in any case in a hidden macOS component, StorageKit, although it was perpetrated by Disk Utility – one of the problems of using such a complex software architecture.

The other insights into Apple’s problems came when I was reversing its logarchive bundle format, for my tool MakeLogarchive.

Before macOS Sierra brought its new unified log, a great many Mac users browsed their logs using Console. Many just used logs to check that Time Machine backups were running without errors, and they were good for a whole load of other valuable tasks such as checking for extension problems, viewing who had logged into a Mac, and working out why an app or waking from sleep caused problems.

Sierra changed a lot of things about logging and logs. For a start, logs are no longer text files, but are now locked into a proprietary compressed format .tracev3, which Apple refuses to document. Apple controls access to logs through two utilities: the Console app, which is next to useless and has barely been updated in High Sierra, a whole year after its first release, and the log command in Terminal, which provides a basic level of access and needs to be combined with other tools to do anything useful.

The log can be exported in another format, .logarchive, which Apple again refuses to document, and is just a bundle which maintains log content in tracev3 files anyway.

The strange thing about logarchive bundles is that their format differs from the ‘live’ log, which is stored in /var/db/diagnostics and /var/db/uuidtext, but to no useful purpose. Make a logarchive, and (with one exception) it contains exactly the same files that are in a ‘live’ log, just shuffled around to make the format more opaque. The one exception is a property list, much of which either duplicates information already in the log files, or appears redundant.

When reversing the format, I have tried to think why Apple chose that format, rather than simply copy the two live log folders as they are into a bundle. The only reason that seems to hold good is deliberate obfuscation. As – until recently – the only tools which can access logarchive bundles were Console and the log command, Apple again retains absolute control.

Everyone who has tried to access the new unified log has discovered that, because it contains so much information, it is significantly harder to use than traditional logs. Yet Apple has provided us with tools which are – and remain a year later – manifestly inadequate or difficult to use, and has prevented any alternative means of access. Apple doesn’t want us to access the new log, does it?

If you’re still unsure of my conclusion, use MakeLogarchive (in Downloads) to convert your current live log into a logarchive, and try browsing its 25 million or more log entries using Console.

It’s no small wonder that Apple doesn’t want macOS to be documented effectively.

(Thanks to Miles Wolbe for the information about version number changes, etc., in Disk Utility, which he made in comments to a previous article here.)