Last Week on My Mac: Hollywood’s lessons

Almost a week ago, Variety, a publication not generally known for its coverage of Mac technical issues, reported that a “mysterious data corruption issue” had struck the Mac Pro (Late 2013) computers used in many Los Angeles film and TV editing studios. Further careful investigation by Mr. Macintosh, Avid, Google and others revealed that what had happened was an unusual combination of circumstances.

For your Mac to be vulnerable to this severely debilitating problem, it had to:

  1. have SIP disabled (or be running OS X 10.10.x or earlier, which don’t have SIP), and
  2. have Google’s Keystone updater software, used to autoupdate Google Chrome, try to apply a new update, which has since been pulled.

Google’s rogue Keystone update erroneously attempted to remove the /var symbolic link on the startup volume. As that’s normally protected by SIP, if your Mac was running with SIP enabled, it couldn’t do that, and no harm was done. However, many of the affected Macs were running with SIP disabled, apparently to enable support for third-party video cards. As the Keystone updater also cast aside the last protection on that symbolic link by running as root, it was able to delete that vital link. Without the /var symbolic link, the affected Mac is unable to boot normally.

In case you know anyone who’s still affected by this, a complete set of solutions is available here.

As is so often the case, there are lessons for all of us from this unusual problem.

System Integrity Protection is intended to do exactly what its name declares: protect the integrity of macOS. If for any overriding reason you need to run a Mac with SIP turned off, or not fully on (there are degrees which are controlled in Recovery Mode), then that Mac must be treated as very special. Exposing a Mac with SIP disabled to the slings and arrows of the Internet isn’t something that you should ever choose to do. If the only way that you can use a particular video card or other hardware/software is by turning SIP off, then you shouldn’t think of risking that Mac in general use.

By far the best strategy for macOS security involves, at its centre, keeping SIP enabled fully and at all times. Above all else, this prevents inadvertent modifications being made to your system, regardless of how well protected you might feel that Mac is from malware.

Running old and unsupported versions of macOS which lack protections like SIP is also not something to do without a great deal of care. If it is necessary to be able to use essential hardware or software, then again that Mac must be treated as very special. Once Apple stops providing security updates for a version of macOS, it accumulates a list of known vulnerabilities and is ripe for exploitation by malware. Here, without even the protection afforded by SIP, Yosemite and earlier shouldn’t be exposed to the risks of the Internet, regardless of the browser that’s being used.

We tend to assume that, just because an old version of macOS was fairly safe when Apple stopped providing security updates for it, then that’s the way it remains. That might be true if it contained no vulnerabilities, but not only is that a false assumption, but its vulnerabilities become even better known.

This is also a good illustration of how automatic updating can be dangerous. In a studio full of Macs, it’s a wise plan to update one test system first, and verify that isn’t adversely affected as a result. Had that been observed here, the bug should have become manifest on that test system alone, and others not updated until the bug was fixed.

That Google should have released an auto-update, which it knew would be rapidly deployed across millions of Macs, that contained a script or tool which attempted to delete a key symbolic link within macOS, should also be cause for grave concern. To perform this, the script or tool would have to be running as root, and this bug is prime evidence that Google’s update quality assurance was woefully inadequate.

Several wise people have laid part of the blame on SIP, which is a bit like blaming a sprinkler system for causing a fire. This claim is based on the fact that, when tested on a Mac with SIP enabled, the bug wouldn’t have become apparent. Unfortunately, that reasoning is flawed in two respects.

First, no script or tool in an autoupdater should ever attempt to delete such a key symbolic link at the root level of the boot volume. Any code which attempted that should never have cleared review, let alone got into internal testing.

Assuming the terrifying thought that such code was so obfuscated that the bug wasn’t glaringly obvious, when tested with SIP enabled that command or tool should have returned an error during pre-release testing to indicate its failure to delete the symbolic link, because of the intervention of SIP. That error should have been investigated and the bug discovered well before release. An updater which doesn’t handle, record and report its own errors is a succession of disasters waiting to happen. For these reasons, I have now removed Google Chrome and its updater from my Mac.

The plain fact of the matter is that, in this case, SIP did exactly what it’s designed to do: it prevented third-party software from damaging the system. Who could possibly condemn one of macOS’s primary security mechanisms for doing its job as intended?

Furthermore, as has been pointed out, Catalina will do this better, even when users do turn SIP off. Because its system volume will still be mounted read-only, a rogue installer would not only have to run as root, but also mount the system volume for writing. Roll on Catalina.