Last week, I looked at quarantine flags on documents, and wondered what their true purpose was. I think I’ve found it: they can stop you from opening documents, even those which are known to be free of malware and have never been near the Internet. Yes, in macOS the purpose of document quarantine is to get in your way and ruin your day.

A little experiment

Find a document, any document, so long as it has a quarantine flag attached to it. This could be because it was downloaded from the Internet, or has merely been opened by a sandboxed app such as Preview. Then make a copy of it: don’t do this with the original, as by the end of this all you’ll want to with that copy is trash it, as it will drive you crazy.

Select that copy and use the Finder’s Get Info command to change the app which opens it to something other than the default for that document type. It doesn’t matter which app it is, so long as it’s not the default. Once that’s selected, and without applying this change to other documents of that type, close the Get Info dialog, and double-click on the document icon to open it using that new app.

You should see a dialog similar to that above, which refuses to let you open that document apparently because it is from an unidentified developer (not source), and your security preferences won’t allow it. None of this makes any sense, does it?

Select that document again, open the Get Info dialog, and set the app to open it back to the default, the first item in the popup menu. Close Get Info, and double-click on the document. It should open fine now. But if you ever again try to change that document’s individual setting for the app to open, you’ll find that macOS once again refuses to open it, with the same strange dialog which makes no sense at all.

Why does this happen?

To understand why this occurs, you need to know about two different ways of opening documents, and two different extended attributes (xattrs) which can become attached to them. Set your document’s app back to the non-default which caused that refusal, and this time open the non-default app and use its Open command in the File menu to open the document. Although the Finder refuses to play, you can still open that document from within the app, without generating any warnings or errors.

This is because, when you double-click a document, drag it to an app icon, etc., macOS uses LaunchServices to handle that interaction and opening. From within the app, though, LaunchServices aren’t (so) involved, and the app itself does the opening.

The more frequently-encountered xattr involved is the ‘quarantine flag’, com.apple.quarantine. This comes in two varieties for documents: a genuine form, which is essentially the same as that used for apps, and a spurious form which is attached whenever the document is opened by an app which runs in the sandbox. In this case, they both behave exactly the same.

The other xattr is less well-known, and is of type com.apple.LaunchServices.OpenWith. This is where information about a non-default app is stored, with its ID and location.

The OpenWith xattr is only attached to documents for which the user has specified a different app with which to open that document, as you did in the Get Info dialog above. For the great majority of your documents, there’s no OpenWith xattr, but once you have assigned a document a non-default app, this xattr seems to stick as tenaciously as a quarantine flag. Change that app back to the default, and the xattr isn’t removed but just changed to point to the default app instead.

That in itself could prove a problem if you were to change the default app for that document type, of course, which could leave some documents still being opened by the previous default app. I haven’t checked whether that actually happens, though. What the Finder should surely do is remove the OpenWith xattr altogether when the app is set back to the user’s default.

What’s most important here is that both xattrs are opaque to the user, who is usually blissfully unaware of their presence and content.

When you open a document inside an app, without calling on LaunchServices to handle it for you, the quarantine flag and OpenWith xattr seem to be effectively ignored: they don’t change any behaviours which are visible to the user.

Opening a document using LaunchServices, by double-clicking or dragging and dropping it, is very different. macOS then checks both the quarantine flag and the OpenWith xattr. If the latter points to the same app as the default for that document type, then regardless of the quarantine flag, the document is opened as expected; similarly, if there’s no quarantine flag, none is enforced.

But if a document has both quarantine flag and OpenWith xattr, and the app specified in the latter isn’t the default, macOS refuses that request when it’s made through LaunchServices. This results in the dialog that you see, which doesn’t make any sense because it’s actually intended for apps which can’t pass their first run tests, not for documents at all.

If the request is refused in this way, you may find the Finder’s app association in the Get Info dialog vanishes, leaving the popup blank, which is unhelpful.

How to fix it

With a growing number of documents having a quarantine flag attached, this can only become more frequent, causing more and more users increasing problems when trying to open them from the Finder, thus via LaunchServices.

The only effective ways to work around or fix this problem involve removing one of the three elements required: LaunchServices, the quarantine flag, or the OpenWith xattr. As the latter two are opaque to the user without entering the command line or using a tool such as my free xattred (from Downloads above), this isn’t simple.

The quickest workaround is to open the document from inside an app rather than relying on LaunchServices, but it’s more than just inconvenient to be unable to use the Finder and related services, and questions the very existence of macOS. A more lasting solution is to open Get Info and set the default app to open that document, but that isn’t what you want.

Removing either of the xattrs is highly effective, although that requires the command line or a third-party tool such as xattred, or my free Sandstrip (also from Downloads) which removes only spurious quarantine flags written by sandboxed apps. The snag then is that:

• removing the OpenWith xattr changes the app which opens the document, and is not what you want;
• removing the quarantine flag only works until a sandboxed app writes another flag to the document, and then you’re back to the square one.

Ultimate cause

macOS is clearly behaving in this way as a defence against malware, which might install an innocent-looking document but set its OpenWith xattr to ensure that it’s processed (installed or run in some way) using a third-party tool instead. However, there are several serious flaws in the way that this is currently implemented, in particular the differences in operation between app and document quarantine.

In normal circumstances, documents are left in permanent quarantine, because their flags are never unset or removed. This ensures that even many years after they were last opened by a sandboxed app, in this respect they still behave as if they had just been downloaded from the Internet.

One solution would be to change the behaviour of document quarantine, so that once a document has been opened successfully, its flag was changed to indicate that, and it was no longer treated as being in quarantine. Treating almost every document on a Mac as if it’s poised to suddenly turn malicious is a gross misjudgement. This wouldn’t be so much of a problem if sandboxed apps weren’t happily writing quarantine flags to every document that they open, even when they don’t even save that document. Such promiscuous behaviour will ultimately make opening documents from the Finder, etc., so dysfunctional as to be unreliable.

Furthermore, determining document behaviours like this through opaque metadata prevents the user from making judgements of their own on which documents to trust. It essentially deems every document untrusted for ever, which is most bizarre in comparison with the treatment of apps, which once they have passed their first run checks are so trusted that they can even have broken signatures and macOS doesn’t bat an eyelid at running them.

Many thanks to Martin, who kindly gave me the clues to find this little nightmare. He also reported the bug to Apple as ID 50305685, but I suspect this will turn out to be well-known there already.