I recently drew attention to the fact that, without a quarantine flag set on a download, it’s all too easy for malware to gain entry to a Mac, particularly if it’s running Mojave or earlier. This article looks in more detail at how setting the quarantine flag is controlled by apps and macOS, and explains how Apple mitigates this issue.

Although apps and other software can set and remove quarantine flags using explicit code, this is most usually left to a setting in the Info.plist property list which every app is required to contain. The entry there which controls flag behaviour is named LSFileQuarantineEnabled, and you can inspect this in each app to check what should happen when that app creates a new file, for example when downloading something from the Internet. When this is set to true, every new file created by that app should have the quarantine flag set; when false, they won’t unless macOS overrides that behaviour. If an unsandboxed app’s Info.plist doesn’t set LSFileQuarantineEnabled explicitly, then the default is not to set the quarantine flag.

You can of course edit an app’s Info.plist, but in doing so will break its signature. You may be able to get away with this for the time being, particularly on older versions of macOS, but it generally isn’t a wise choice.

macOS also provides a set of overrides to what appears in the Info.plist of many apps. These are listed in the Additions item in /System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Exceptions.plist.

The Exceptions.plist property list contains five dictionaries:

• Additions, which assigns a lot of app categories, sets Java version requirements, and determines default settings for quarantine on documents created by apps.
• AppNapOverrides, which sets App Nap behaviours.
• HighResolutionOverrides, which overrides High Res options for apps.
• LaunchOverrides, which can disable specific version ranges of apps from being launched; these prevent many older apps from being run.
• MergeDocumentTypes, which merges some document types such as doc and docx for specific apps.
• Overrides, which can override other settings.

For example, the entry in the Additions dictionary for the popular BitTorrent client Transmission reads:
<key>org.m0k.transmission</key>
<dict>
<key>LSApplicationCategoryType</key>
<string>public-category.internet</string>
<key>LSFileQuarantineEnabled</key>
<true/>
</dict>

Referring to the app by its ID of org.m0k.transmission, that first assigns the app to an app category of public-category.internet, and then sets the app to set the quarantine flag on all documents that it creates, including everything that it downloads.

Among the existing overrides in Catalina, for example, are org.pythonmac.unspecified.BitTorrent and org.xlife.Xtorrent, which ensures that Transmission, Xtorrent and PythonMac BitTorrent clients should write quarantine flags to all their downloaded files. Although this Exceptions property list doesn’t cover every client, it should ensure that most do protect their downloads with quarantine flags.

There are two snags to this otherwise protective system: first, the file containing these overrides is protected, in Catalina being on the System volume, so the user is effectively prevented from changing it. Unlike app preferences, which can be managed by the user at the command line, there’s no way for the user to add their own overrides. If you download items using an app which doesn’t itself require the quarantine flag to be set, and Apple doesn’t provide an override for it to do so, there doesn’t appear to be any good way to add that yourself.

It’s also unclear whether this system works with command tools, which are single file executables. They can have their Info.plist embedded in the executable, but this is rare unless they need to be notarized. For many users, it might be helpful, for example, if the standard tool curl were to set quarantine flags, as it’s often used to bypass quarantine and thus presents a significant vulnerability.

Finally, for some users at least, an app setting the quarantine flag isn’t of much use, as that user routinely strips the flag from downloads. If you do that, you’re steering into as much as danger as you would using an app which never sets them in the first place.

I’m enormously grateful to @rosyna who pointed me in the right direction, again.