For the first six years or so of Mac OS X, its system provided little if anything to detect, remove or combat malicious software. It seems that the first step taken was the voluntary introduction of code signatures in around 2007, a feature promoted by an Apple engineer known only as “Perry the Cynic”. These were part of the first Gatekeeper sub-system, which developed slowly until its formal introduction in 2012.

At the same time, Apple’s security engineers were busy developing the App Sandbox, also introduced in 2007, but which didn’t really come of age until it was made a requirement for App Store apps in June 2012, although some older apps have enjoyed grandfathered exemptions ever since.

Gatekeeper brought its mechanism for distinguishing apps which had been downloaded from untrusted Internet sources, by the attachment of an extended attribute putting them in quarantine. Quarantined apps are then checked on their first launch by three distinct mechanisms:

• code signature checks, to ascertain signing identity and app integrity, by sub-systems such as AMFI (Apple Mobile File Integrity), known generically as Gatekeeper;
• blocking software such as vulnerable versions of Java, and scanning for characteristics of known malware, by XProtect;
• detection and removal of known malicious code by the Malware Removal Tool, MRT.

You can read a description of their presence and actions as of 2015 in this article. To address certain forms of malware behaviour, additional measures have been adopted, such as app translocation, which in some circumstances launches a quarantined app from a special location.

Signature checks and more

Checks on code signatures fall into two phases: first the validity of stored cdhashes for different parts of an app, and second the validity of the certificate used to sign the app, to ensure that it hasn’t been revoked. From Mojave in 2018, Apple has added another set of checks with the introduction of notarization.

Until 2018-19, it appears that macOS stored information about certificate revocations locally, in the ‘Gatekeeper’ database at /private/var/db/gkopaque.bundle, which Apple updated every couple of weeks. Those Macs which have kept pace with the latest release of macOS stopped accessing that database in September 2019, with the release of macOS 10.15 Catalina. Apple hasn’t released an update to it since 26 August 2019, and anyone with a fresh installation of Big Sur will have a truly ancient version installed. As I pointed out here, that ‘Gatekeeper’ database is now effectively disused.

As this checking system developed, well before High Sierra and probably before El Capitan too, Gatekeeper started to perform online OCSP queries to check the validity of code signing certificates, initially only for quarantined apps undergoing their first run. However, around July 2019 (macOS 10.14.6), these checks were extended to apps which had already cleared quarantine. By the release of Catalina in October 2019, certificates were being checked on loading all executable code even when no quarantine flag was set. As far as I can tell, those include systematic checks of both signature and cdhashes, which overlap with the added requirement of notarization.

Although Apple had long maintained that users would remain able to run completely unsigned code in macOS, that too changed in November 2020, with the release of the first Apple Silicon Macs. All code running natively on ARM processors is required to be signed, although at present this is to provide cdhashes for transmission to Apple, and possible remote checks against malware values. No requirements are currently imposed on signing certificates used beyond those of existing Gatekeeper controls on the first launching of quarantined apps, which now include checks for notarization too.

In November 2020, Apple’s use of online OCSP checks came under fire, driving it to take immediate steps to protect privacy, and to state that certificate revocation checks will change in the following year to feature:

• “a new encrypted protocol”;
• “strong protections against” [OCSP] “server failure”;
• “a new preference for users to opt out of these security protections”, which presumably means both hash lookup and certificate revocation checks.

As far as I’m aware, none of those three changes has yet been implemented, although there are only four months left before that year elapses.

XProtect

XProtect is an on-demand service buried within macOS security services, with a more visible set of data files which are updated periodically by Apple. The XProtect.bundle containing data files is located in /Library/Apple/System/Library/CoreServices/, among the unprotected Data volume additions to /System/Library/CoreServices/ on the System volume in Catalina; in Mojave and earlier, that’s simply /System/Library/CoreServices/

Its purpose is to scan files and check them against lists of blocked software, and against Yara signatures of known malware. It has developed over the same period as Gatekeeper, although its emphasis has changed considerably over that period. Originally it was used to ensure that old versions of Java, Adobe Flash Player and similar products weren’t used, checking them against lists in XProtect.meta.plist. For example, at the end of January 2013, Apple’s security group put Java 1.6.0_37 on XProtect’s blocked list, and Macs which still had that or older versions installed were unable to run Java. This function now appears largely disused, and those version numbers haven’t been updated for at least a couple of years.

XProtect also uses a set of Yara definitions to detect known malware. These definitions contain rules which are applied when scanning files; if a rule is satisfied, XProtect declares that item malicious, prevents it from being opened, and alerts the user. Until 2018, Apple identified malware signatures using generally accepted names; since then it has obfuscated malware identities with the use of code names for the malware it can detect.

Before 2019, XProtect only checked quarantined files. Apple then announced that the executable code of every app and command tool would be checked whenever they are run, regardless of the quarantine flag, and this has applied from Catalina onwards. From Big Sur, XProtect also runs an initial scan during late startup, at least on Intel Macs. Apple hasn’t revealed which files it scans then.

In recent years, with the proliferation of malware, Apple has usually pushed updates to XProtect’s Yara definitions every two weeks, in its efforts to keep pace with changes made by malware developers to evade the latest signatures.

MRT

While XProtect depends on Yara definitions and block lists to detect vulnerable software and malware, the Malware Removal Tool (MRT) contains a library of routines which try to remove malicious and unwanted software, based on its own opaque rules. Although MRT appears to be an app, it consists of a command tool which is run in two modes: as an agent, or a daemon. The dummy app is located in /Library/Apple/System/Library/CoreServices/, among the unprotected Data volume additions to /System/Library/CoreServices/ on the System volume in Catalina; in Mojave and earlier, that’s simply /System/Library/CoreServices/. It’s written in Swift, and is normally run by two launchd property lists.

MRT runs predictably on two occasions: in the late phase of startup, after the user has logged in, and immediately after it has been updated. It may also run briefly during Gatekeeper checks, but appears silent then. Unfortunately, since 2018, Apple has obfuscated the names of malware which MRT can remove. Prior to that, it was possible to search the string content of its executable and discover the names of malware which it claimed to be able to ‘remediate’.

MRT has undoubtedly evolved since its introduction in the early days of Gatekeeper, but its much lower profile compared to that of XProtect makes it harder to investigate. It has though emerged twice into prominence.

In early July 2019, MRT was called into play to remove part of an app which wasn’t malware, but left users with a serious vulnerability: Zoom. One of the biggest problems posed by that old version of the Zoom client was that it installed, in a hidden folder, a web server which was left behind, still active, when you uninstalled its app. This web server was capable of reinstalling the Zoom client, and was found to have its own vulnerability as well. However Zoom responded to the other issues in its client software, it was vital that all copies of this web server were removed, particularly on Macs whose users may have forgotten that they had ever installed Zoom’s client. This wasn’t something that Zoom was able to handle alone: it needed Apple, just as Apple needed to remove Zoom’s web server before it could be exploited.

The solution lay in gently repurposing MRT to detect and destroy Zoom’s web server in its hidden folder, much in the way that it does for malware. The delivery vehicle had therefore to be an urgent ‘silent’ security update containing the new version of MRT, which Apple pushed out on 10 July 2019.

MRT’s second moment of fame was less impressive, when version 1.68, released on 19 October 2020, caused serious problems on many Macs. Apple silently addressed that in version 1.69.3 eleven days later.

Notarization and compulsion

In the grand scheme of Mac security, notarization is very recent, controversial and probably largely misunderstood. Expectations were that Apple’s malware detection system would ensure that no malware could ever be notarized, which is unrealistic given the experience of such detection schemes. Instead, notarization presents a significant challenge to malware developers to eliminate all but the most determined, who are then limited in the strategies they can use to evade automatic detection during notarization checks.

More importantly, it provides Apple with a copy of all notarized software, giving the cat something of a head start on the mouse for once. In the event that a notarized app turns out to be malicious, certificate revocation can be immediate, and Apple’s security engineers already have a sample of the app complete with its cdhashes, which can be rapidly incorporated into XProtect’s Yara definitions and rules for MRT.

To some the greater concern is that notarization, which currently remains voluntary in the sense that Apple doesn’t enforce it on developers (except in specific cases such as extensions), and users can choose to ignore it, becomes mandatory. Yet it has taken thirteen years for “Perry the Cynic’s” code signing to become mandatory on Apple Silicon Macs.

The following three diagrams summarise how these sub-systems have changed most recently.