In teaching a lesson about the role of cold in many diving accidents, I used to mention the ‘accident chain’. I used it to show how a series of relatively minor sins of omission and commission could and did so easily lead to catastrophe. Now, I could use the encryption passphrase vulnerability in High Sierra instead.

Thanks to diligent log browsing by Sarah Edwards, we now know that High Sierra has been leaking APFS encryption passphrases since its first release. Although these leaks have finally stopped with the 10.13.4 update, Apple hasn’t yet dismantled the chain which led to this vulnerability in the first instance. As far as I can tell, it goes way back to much earlier versions of Mac OS X, and remains ready and waiting in this latest release of High Sierra.

It is based on bad log discipline, a long-standing problem in Unix and related operating systems which Apple tried to kill with its new unified log, introduced in Sierra. There’s no set log architecture across different species of Unix, and Apple doesn’t ever appear to have explained the purpose and scope of the install.log found in /var/log.

Among its long-standing contributors are the services which prepare disks and volumes, including storagekitd and diskmanagementd. This makes sense up to a point, but it seems inappropriate that every disk which is erased and formatted by macOS should have a blow-by-blow account written to install.log.

When Apple introduced its new unified log in macOS Sierra, almost every other system log went silent (other than in residual entries by legacy products), apart from install.log. Why that was spared has never been explained by Apple, which suggests that it wasn’t part of its plan. This has now proved helpful in many situations, as sysadmins and others can still examine installation and update problems without having to do battle with thousands of other entries in the unified log.

An anomaly remained in some parts of macOS, including diskmanagementd, which had previously apparently been writing to two logs simultaneously, presumably with private frameworks, using both the syslog call to write to the system log, and a custom call to write to install.log. With the arrival of the new unified log, those components write to the new log (via diversion of syslog calls) and to the old install.log. Their behaviour was not unified in the slightest, but no one rectified that.

One potentially good reason for continuing to write to a traditional log as well as the new unified log is the weakness of Apple’s tools – then and now – for accessing the unified log. Console still lacks any ability to browse history in the unified log, except when the live log is converted into a logarchive, and even then it is hard to use. install.log remains far more convenient to examine, and the engineers working on those parts of macOS which have been writing to it have probably been very grateful that they were not forced to work with the unified log alone.

High Sierra was rushed to release months ahead of maturity. A succession of glaring bugs and vulnerabilities in its first release demonstrate that Apple’s engineers were given neither the time nor the quality management to deliver a product which was fit for release. One of its weakest points was Disk Utility and its supporting tools, including diskmanagementd.

From release, new core features in Disk Utility, executed for it by diskmanagementd, supporting APFS were hacked in hastily. Handling very sensitive tasks such as the creation of an APFS encrypted volume were bodged by calling underlying tools, including newfs_hfs and newfs_apfs, rather than using secure methods which should have been mandatory in release software.

Those who wrote these hacks appear to have been unaware that their calls to external tools resulted in detailed logging of the command parameters, including encryption passphrases. If they were aware and had deliberately left those details in the log, they failed to mark that code for debugging only. No one reviewed that code properly, at least not until 10.13.2 was in development, and even then, any such review was cursory and failed to identify obvious bugs.

What did change in 10.13.2 was that someone noticed that creating a new encrypted APFS volume should be performed using a more secure method, rather than just calling newfs_apfs direct with an undocumented option. They did not check thoroughly enough, though, and left the original insecure hack in place when encrypting previously unencrypted APFS volumes. Neither did they notice that other calls using newfs_hfs and newfs_apfs were still being written with full parameters to the unified log, nor that they were also being written to install.log.

Just a few days before the scheduled release of 10.13.4, Sarah Edwards discovered the original bug which had existed before 10.13.2. Then she discovered that 10.13.2 didn’t fix a second bug, in which passphrases were disclosed when an APFS volume was converted from unencrypted to encrypted.

Apple’s engineers once again turned around a fix in very quick time, and it was included in a slightly delayed update to 10.13.4. Unfortunately they appear to have been unaware of what Sarah Edwards was to discover next, only a day after that update was released: that the passphrase disclosures prior to 10.13.4 had not just been in the new unified log, but they had been duplicated into install.log too.

Although Apple had fixed what it thought to be the bug, it had not considered that plaintext records of passphrases used to encrypt APFS volumes would remain in install.log and all its backups for far longer than matching entries in the unified log (typically removed after around twenty days). install.log isn’t replaced with each major macOS upgrade, but normally only when the startup disk is erased. Most who upgraded from Sierra to High Sierra will still have all their old entries from Sierra in their current install.log, where they may remain through 10.14 later this year.

At present, it looks as if encryption of the startup volume, either performed as part of its erasure or by conversion from an HFS+ volume, does not result in the leak of its passphrase into install.log (although I have yet to see that confirmed). However, if you look back through the install.log of a Mac running High Sierra which has encrypted other APFS volumes, you are highly likely to see their passwords there, for anyone to read.

Returning to the accident chain behind this, I can see the following links:

1. Bad log discipline allowed diskmanagementd and others to write non-install messages to install.log.
2. Those log entries were not confined to the new unified log when it was introduced in Sierra, but continued in install.log too.
3. Inadequate tools to provide access to the unified log made those entries in install.log convenient if not valuable in debugging.
4. High Sierra was rushed to market, and APFS support in Disk Utility and diskmanagementd were kludged using insecure calls.
5. Engineers appeared to be unaware of the parameters being written to logs by diskmanagementd, as they were not marked as temporary workarounds or for debugging; they were therefore left in production code.
6. When the first bug (affecting creation of new encrypted APFS volumes) was fixed, no action was taken to warn users that plaintext passphrases were recorded in both the unified log and install.log, probably because no one recognised that was occurring.
7. When the second bug (affecting conversion of APFS volumes from unencrypted to encrypted) was fixed, that was neither recorded in the security release notes for 10.13.4, nor were users warned that plaintext passphrases had been recorded in both the unified log and install.log.

Although a little more long-winded than the diving accidents which I used to illustrate accident chains, in many ways this is even clearer. As is the fact that, so far, Apple has proceeded piecemeal, and only tackled items 6 and 7. diskmanagementd and other processes still write copiously about erasing all disks – even USB Flash drives – to both install.log and the unified log. Apple still hasn’t provided a tool with the capabilities of El Capitan’s Console for use with the unified log, even after 18 months.

Disk Utility and diskmanagementd are still being dug out of the parlous state that they were in when released in 10.13, although in 10.13.4 they appear much improved at last. diskmanagementd still writes full command parameters to install.log and the unified log when erasing and formatting disks, information which by Apple’s privacy rules would normally be deemed private, and censored from the unified log.

With so much of the accident chain remaining in place, it can only be a matter of time before it is found to link to another accident.