The basic idea behind quarantine in macOS is simple: anything that comes from a dubious source, particularly when downloaded from the Internet, should be marked as such, so that macOS security tools can check it for known malware before it’s first used.

Apple introduced the quarantine ‘flag’ in Mac OS X 10.5 back in 2007, but it wasn’t until 10.7 in 2011-12 that Gatekeeper appeared and started to perform checks on apps with quarantine flags. The arrangement was, and is still, essentially voluntary. While most items downloaded from the Internet via browsers get a quarantine flag attached when they’re saved on your Mac, it’s still possible for some tools like curl to download items which don’t take part in this scheme, and don’t get flagged.

To ensure that these quarantine flags don’t cause problems with the integrity of downloaded files, they’re attached as an extended attribute of type com.apple.quarantine, within which are details of the flag itself, the time at which the flag was attached, the app or agent which attached the flag (normally the same as the app which downloaded the file), and a UUID reference to its entry in the QuarantineEvents database.

When an app or other executable code is run from the Finder or GUI, macOS checks its quarantine flag before loading it. If the flag hasn’t been changed to indicate that app has already been checked, it’s handed over to Gatekeeper for full ‘first run’ security checks. If they’re successful, the flag is changed to show that it has passed those checks, and it’s loaded and run, but the flag remains to indicate that checking was successful.

Two common situations which catch users out with this system are notarization and AirDrop.

When Apple made notarization obligatory for third-party developers, it incorporated checks for that in Gatekeeper. Although the user can override these by opening a quarantined app using the Finder’s Open command, this is a two-step process in which there’s a warning that the quarantined app hasn’t been notarized. This is to ensure that users are fully aware of the risk in running unnotarized apps which have the quarantine flag set.

That can become more inconvenient if you copy apps between Macs using AirDrop, which is deemed insecure and results in copied apps being put into quarantine. If you regularly use AirDrop as a means of copying unnotarized apps, this makes a convenient transport mechanism tedious.

As quarantine flags aren’t protected in any way, one solution to these and other problems is simply to strip the com.apple.quarantine extended attribute, which defeats the whole purpose of quarantine and Gatekeeper, and invites malware.

Where quarantine becomes more irksome and less purposeful is when it’s applied to documents, which has been happening since its introduction in 2007. There are two big differences, though, when compared with app quarantine: checking of documents to determine if they have malicious content is an even tougher nut to crack than checking apps, and recent versions of macOS have taken to routinely attaching quarantine flags to documents which have never left that Mac.

To see this new quarantine system at work, you’ll need an app like my free xattred which can display extended attributes (xattr) easily. Take a screenshot and inspect its xattrs. You’ll see that one of them is of type com.apple.macl – I’ll return to that shortly. Open the screenshot in Preview, and save it. When you inspect its xattrs now, you’ll see that this image, which has never left your Mac, now has a quarantine flag as well as that strange macl xattr. Now try cutting or deleting the macl xattr, and you’ll see that you can’t, as it’s protected by SIP.

Sandboxed apps, which includes many of those bundled with macOS and all delivered by the App Store, attach quarantine flags to files which Apple considers to be executable unless the app has the com.apple.security.files.user-selected.executable entitlement. Apple has previously explained:
“By default, when writing executable files in sandboxed apps, the files are quarantined. Gatekeeper prevents quarantined executable files and other similar files (shell scripts, web archives, and so on) from opening or executing unless the user explicitly launches them from Finder.
If those executables are tools that are intended to run from the command line, such as shell scripts, this presents a problem.”

While that’s bad enough when you’re trying to edit a shell script in a text editor, quarantine flags are now the rule rather than the exception for documents. They have become so commonplace that it might be more efficient to have a xattr which indicates that a file isn’t in quarantine.

The recent addition of the com.apple.macl xattr has complicated this further. Although Apple doesn’t appear to have documented this, even in its Platform Security Guide, which also fails to explain the use of quarantine flags on non-executable documents, this is thought to relate to the TCC privacy protection system.

When quarantine flags became more widespread on documents and the macl xattr was introduced, they often resulted in frustrating limitations in macOS. For example, Preview became unable to save some documents, and it wasn’t unusual to find other apps locked out of their primary purpose. In Monterey, this is far less frequent, but can still occur.

Whatever Apple is doing attaching all these quarantine flags and macl xattrs to documents, the value of scanning documents in Gatekeeper is in great doubt. While some malware can be detected reliably in documents using signatures and hashes, perhaps using XProtect, the ease with which such measures can be defeated when changing a single byte in that file makes this ultimately futile. Maybe one day Apple will get round to explaining why all our documents are being put into quarantine even when they never leave that Mac.

You can read more technical details in this article.