How Monterey checks apps and executable code

It has been a good while since I last looked at how macOS checks apps and executable code before running it. Here I’m not referring to the first run, with a quarantine flag set, which used to be the only time Gatekeeper checked apps, but every time after that. What I attempt here is a coherent account of how macOS checks executable code before it’s loaded and run in macOS 12.

These processes don’t just occur when you open an app, but also when any other executable code is loaded, for example when using a command tool, or when running code needs to load an executable from another source such as a dylib. I’m not going to consider the loading of code contained within the protected System volume, nor that signed by Apple, and that run in sandboxed apps supplied from the App Store may also differ in parts.

Code signing

There’s an important difference between loading natively executable code on Intel and Apple Silicon Macs: the former allows code to be completely unsigned, but the latter refuses to load ARM code which isn’t signed at all. However, that requirement is enforced not to establish any chain of trust from its signing certificate (using an ad hoc certificate is sufficient), but so that the cdhashes created during the signing process are available.

Running Intel code on Apple Silicon Macs is different. Because Rosetta 2 translates that into ARM native code, it also has to sign its translations. As far as I can tell, its ad hoc signature supplies the required cdhashes, and provides users with a convenient workaround for running unsigned Universal binaries on Apple Silicon Macs.

Has this code been run before?

The first question to be answered is whether this copy of macOS has run this code before. To determine that, if the code is signed its hash is looked up in the cdhashes created during the signing process. If there are no cdhashes (because the code is unsigned), then macOS has to compute its hash on the fly. That takes time, particularly if this is a large app, so Apple Silicon Macs attain superior performance by refusing to compute the hash of native code, so block its loading. The code signing requirement on Apple Silicon Macs therefore eliminates delays which many have noticed on Intel Macs in the past. Although a minor nuisance for some users, this should hasten app launch and execution.

Armed with the hash for the code to be run, this is looked up in the local security database, which establishes whether that code has been run before. If it has, checks move on to examine the code signature.

If this is the first time that code with that hash has been run, the hash is added to the local security database, and that hash is sent to Apple via iCloud. On first run, this enables notarization to be verified, but the hash is still checked each time the code is run to verify its integrity.

Is it signed using a developer certificate?

Code which hasn’t been signed using a developer certificate now can’t be checked any further, except by routine checks in XProtect to see if it’s known malware and should be blocked. Otherwise, it is now loaded and run.

Where a developer certificate has been used to sign that code, the second question to be answered is whether that certificate is still valid. The first step in this process is to check in the local OCSP cache whether this certificate has been checked recently. If it has, and hadn’t been revoked at that time, the code is loaded and run.

Certificates which aren’t in the local cache, because they were last checked some time ago (12 hours has been used in the recent past), are checked with Apple’s OCSP service. Previously, at least in Catalina and Big Sur, that was performed using plain HTTP to Apple’s servers at ocsp.apple.com. At some time in the last year or so, Apple has changed that process for Big Sur and Monterey, and this check is now performed using TLS to ocsp2.apple.com.

Because it may not be possible to get timely information about revocation status of the certificate, Big Sur and Monterey now perform this online check asynchronously, allowing the code to be loaded and run while macOS continues to try to get an answer. A rapid series of 40-50 attempts to connect to the server may be made before macOS accepts the check can’t be completed. Because this occurs asynchronously, it shouldn’t result in any delay to the code being run, although you do wonder what would happen if it turned out that the certificate had been revoked because it had been used to sign malware.

This is all summarised in the diagram below, which should be correct as of Monterey 12.2.1.

SignatureCheck12

How can you block transfer of information?

Checking whether executable code can be run is a fundamental part of your Mac’s security. Apple can and does revoke certificates because they’re being abused by malicious software. If you were to prevent OCSP checks on signatures, you could end up running known malware whose certificate has already been revoked. Interfering with these checks thus has serious consequences, and shouldn’t be considered unless you have a really compelling reason, and have assessed the security risk of the consequences.

For those who don’t or can’t risk the OCSP exchange and transmission of new hashes, there are solutions which should mitigate against that. For instance, provided that an app has already been run and its cdhashes entered into the local security database, no repeated copies of those hashes should be sent to iCloud. Blocking outgoing connections to ocsp.apple.com and ocsp2.apple.com is readily performed using a software firewall such as Little Snitch or LuLu.

Muting the OCSP query is just as simple. It can’t occur when the code isn’t signed, so stripping an app’s signature will ensure that no query can be sent. However, unsigned code can’t be run on Apple Silicon Macs, and on both architectures its integrity can’t be checked either. For specific apps whose use might be sensitive, one approach is to strip any developer signature and replace it with an ad hoc signature. That isn’t technically difficult at all using codesign, and used only on specific apps shouldn’t increase security risk significantly.

By far the best solution would be for Apple to provide a preference for users to opt out of these security protections, something it promised to do over a year ago, but still hasn’t implemented.

Key external sources:

Apple’s updated account of ‘Gatekeeper’ checks, including details of intended changes,
Jeff Johnson on OCSP checks,
Jeff Johnson on hash reporting,
Jacopo Jannone has an excellent account of the OCSP checks.

I am very grateful to Jeff Johnson @lapcatsoftware for repeatedly providing valuable information for well over a year.