Last week a great many Mac users were struck by one or both of two serious problems. This article explains what happened, and what you can do about them.

MRT 1.68 update

Some time before 2200 UTC on Monday 19 October, Apple pushed routine updates to its security tools, XProtect and MRT. These normally happen every two weeks or so, and are silent in that Apple doesn’t announce them, or even admit that they’ve happened. Macs which run with Software Update set to Install system data files and security updates then automatically download and install those updates. That doesn’t happen immediately, but for Macs in daily use, that means that most will have been updated by 2359 UTC on 20 October.

Some, but by no means all, Macs then suffered a problem as a result of the new version of MRT. Unlike XProtect, whose data files are used on demand, MRT is normally run in two situations: shortly after it has been updated, and soon after a Mac starts up. Affected Macs, which can be running any version of macOS from El Capitan to Catalina, suffer severe problems running the new version of MRT. The Mac becomes sluggish, and looking in Activity Monitor normally shows two processes, MRT and trustd, taking large amounts of CPU. MRT keeps crashing and being automatically restarted, and this affects overall performance. Although affected Macs aren’t completely unusable, they’re severely affected by this.

Many Macs which were updated to MRT 1.68 haven’t been affected at all. It’s not clear why this is, and there’s no obvious pattern to these problems. For the moment, if your Mac successfully updated to the new version of MRT without suffering these problems, there’s no reason to suspect that it will develop them later.

This appears to be the result of a bug in MRT 1.68. Apple pulled that update early on 24 October, but unfortunately that doesn’t force affected Macs to be downgraded. Because of the way that these updates are pushed, your Mac won’t see the older version, and even if it did, simply trying to install it won’t force it to be downgraded from 1.68 to 1.67.

There are many different potential solutions for Macs which are affected by this bug. The best will be when Apple pushes MRT 1.69, which should replace the buggy version at last. Until then, you’ll need to follow a method which will remove 1.68 and either replace it with 1.67 from a backup, or allow Apple’s update servers to push 1.67 at it. There are several different and reliable ways of doing that, and if you’re uncertain please contact Apple Support so that they can talk you through resolution.

HP printer software, Amazon Music

Reports about problems with HP printers didn’t start appearing until 23 October, well after those users affected by MRT 1.68 had run into problems. Symptoms were completely different: when trying to print or otherwise access some HP printers, various error messages appeared and access failed. The most worrying of the alerts came from Gatekeeper, which warned that the printer software was malicious and “will damage your computer” and refused to run it.

Not all users reported such characteristic errors, though. In some, there were what looked like regular errors with opaque numbers, and for Catalina users there were crash reports when macOS force-crashed HP software because the code signature is invalid. Problems were most frequent in Mojave and Catalina, and with older HP printers.

These behaviours aren’t anything to do with MRT, whose purpose is to remove malware, or XProtect, which checks for signatures of known malware, but come from Gatekeeper, when it’s checking the signature on software being launched. Errors aren’t consistent because they depend on where the signature is failing: if you try to open an app with a fatal signature problem, Gatekeeper will display that alert or (more brutally in Catalina) crash the app; if an app tries to run a component with a fatal signature problem, that generates an internal error which the app may not communicate explicitly, and just report an error in that component.

Checking the signatures on some of the affected software revealed that the certificate used to sign them had been revoked, presumably shortly before problems appeared on 23 October. So who might have revoked those certificates?

Security certificates provided to developers for signing macOS and other Apple platform apps come from Apple, as the Certificate Authority (CA), a very demanding responsibility which resides with a separate team within Apple, its PKI, operating under stringent rules.

Under those, revocations are performed by the CA most commonly at the request of the certificate owner, in this case HP. The most popular reason for revoking a certificate is when it has been superseded, although many are revoked because of their compromise. As with all CAs, Apple PKI issues a Certificate Revocation List (CRL), which details all the certificates it has revoked. In the last year or so, that amounts to a huge number, in the tens of thousands, and on 20 October 2020 alone there were hundreds of revocations. One reason why this revocation seems to have affected Mojave and Catalina systems first is that they now appear to use a different and quicker system for being notified of revocations from those used by earlier versions of macOS.

Before blamestorming about the sudden blocking of this HP printer software, it’s essential to discover who instigated the certificate revocation. According to an anonymous spokesperson quoted by The Register, that was HP, not Apple. So someone in HP decided to revoke their certificate used to sign that printer software, instructed Apple PKI to make that revocation, which then propagated to Macs. When those Macs next performed a certificate check on the affected software, as part of its Gatekeeper checks, HP’s software was then correctly prevented from opening.

At some time during the night of 24-25 October, Apple PKI withdrew the revocation of HP’s certificate, presumably at HP’s request in response to the many complaints from users. HP’s software should therefore now work normally again.

What to do?

In the case of the MRT update, Apple has already removed MRT 1.68, but that happened too late to help many affected users, and reverting to the previous version isn’t simple – because of the way that Apple distributes these ‘silent’ updates, any bug like this is hard to diagnose and tougher still to fix. I’m a great believer in Apple’s security software. Although it doesn’t provide ‘perfect’ protection (what does?) it’s important for users to keep it up to date. However, in the light of this problem Apple needs to review how these updates are distributed so that users are more aware when updates occur, and have a straightforward method of reverting to an older version when problems arise with an update.

Rather than letting Software Update install updates automatically, it may be preferable to manually download their installers using
softwareupdate -da --include-config-data
and that’s an option which I’m considering for my own utilities SilentKnight and LockRattler. Users can then wait a day or two after each update to verify that problems are unlikely, and retain copies of installers for older versions should they need them.

Some users have suggested that the only way to avoid problems such as that with HP’s printer software is to do away with code signing altogether. Although superficially attractive, I can hear malware developers shouting ‘yes please!’ at the thought that there would then be almost nothing to replace signatures as a baseline in app security.

Instead what we should be asking is how HP came to revoke a certificate still being relied on by many users. I have certificates issued by Apple with which I sign my apps here. Could you see me ‘unintentionally’ instructing Apple to revoke them? That’s a very deliberate act which no corporation could ever leave to an intern to handle. They should have a complete audit trail which establishes exactly where their error was made, with the safeguards which were bypassed to enable such a serious mistake. Such errors aren’t addressed by abandoning a whole layer in macOS security protection.

What should surprise and anger us more than anything else is not that these failures occurred, but that neither Apple nor HP have seen fit to explain what has happened and advise us what to do to work around the problems they have created. Instead, as usual, they leak a little unattributed comment to a reporter and then pretend that nothing happened at all. That isn’t being open or honest with their customers, and only brings discredit to both companies.

Postscript (27 October 2020):

HP has now published a support article explaining what affected users should do to remedy this problem. I suspect this only works with its software for relatively recent printer models. Thanks to @macinteractive for drawing my attention to this.