Quarantine flags and Gatekeeper are at the heart of Ventura’s defences against malicious software. But what we refer to as a ‘flag’ is nothing of the kind: it’s actually an extended attribute, something that relatively few file systems support. And it was very nearly lost from macOS.

When Apple’s engineers were designing the Mac’s original file system, they did something unusual. At that time, in the early 1980s, and now, the great majority of file systems use a single fork: one file, one block of data for that file’s contents.

The Mac was going to be different. Each file would thus consist of two forks, one a regular data fork as in normal file systems, the other a structured database of resources, the resource fork. This enabled an app to keep different localisations of its menus and other text contents in resources. Resource forks were allowed for any file, so an app could store a document’s window settings in the document’s resource fork, and many did.

Resources were used to store a lot of standard structured data, such as the specifications for and contents of alerts and dialogs, menus, collections of text strings, keyboard definitions and layouts, icons, windows, fonts, and chunks of code to be used by apps. You could extend the types of resource supported by means of a template, itself stored as a resource, so developers could define new resource types appropriate to their own apps.

Apple developed a resource editor which quickly became one of the best-known apps on the Mac: ResEdit, last seen in version 2.1.3 way back in 1994. ResEdit was the Mac power user’s quintessential tool: if you didn’t like a particular dialog in an app, ResEdit could be used to change it; if you wanted to create your own custom keyboard layout, it was the first choice for that too. If you wanted to mess someone’s Mac up, you could go into their system files and change things around when they were out at lunch. Not that we ever did that, of course.

Each resource type had a four-character name, such as ALRT for an alert definition, DLOG for dialogs, KCHR for keyboard layouts, and so on.

You could have many different KCHR resources, each numbered, so that changing your keyboard layout was a matter of switching from KCHR number 2, the standard British layout, to number 26, which was Dutch, for example. Adding your own custom keyboard layout was then merely a matter of defining a new KCHR resource with a unique ID number, here 128, and editing it to do what you wanted.

Many resource types had custom editors: here that for KCHR has a simple keyboard layout and tabular form. The highlighted (black) keys indicate that the Euro character € being shown would be generated by holding the Opt (Alt) modifier key and pressing the 2 key on the top row of the main keyboard, but not the 2 on the numeric keypad.

Mac OS X changed all that, almost for the worse. Traditional Unix at that time didn’t have or support resource forks, and it was proposed that they should be done away with on Macs as well. This is reflected today in application bundles: instead of a Classic app consisting of a single file with its collection of resources, in Mac OS X this became a hierarchy of folders containing flat files without resources. In other parts of Mac OS X, former resources like keyboard layouts were also flattened into normal data-only files.

Although Mac OS X still supported resource and other named forks, Apple wanted to move away from them, and in Mac OS X 10.4 introduced extended attributes, one of which, com.apple.ResourceFork, supports what used to be in resource forks. Since then, forks have died away and been replaced by extended attributes. As far as I can tell, although APFS does still retain some support for resource forks, use of resource and other named forks was removed in about 10.8.

Resource forks and the thinking behind them are ingenious and empowering, but open up all sorts of security issues. In the days when there was Classic Mac OS malware, it almost invariably lived in resource forks, and took great advantage of their features. Although it was possible to lock and protect individual resources, armed with a copy of ResEdit you could soon change that, and almost anything else that you wished to, just as malware could and did.

One of the first of these new extended attributes, introduced in Mac OS X 10.5, was what we commonly refer to as the quarantine flag, or the extended attribute named com.apple.quarantine. It’s so old that the ‘flag’ itself, a hexadecimal number, is encoded as UTF-8 text rather than in hex. Like resource forks, extended attributes aren’t stored in the file’s data, ensuring that a quarantine flag can be attached to any file without changing that file’s data.

This is important for quarantine flags, in particular, so they can be attached to files that are signed. The process of signing requires computation of a hash of the file’s data, so that its integrity can be verified. If the quarantine flag had to be stored in a file’s data, adding it would then change the file’s modification date and the hash of its data. Because the quarantine flag is kept separately, in file system metadata, it’s simple to add and change.

Most recently, Apple has taken to adding other extended attributes for security, including the notorious com.apple.macl, which is thought to play a role in privacy protection. One of the weaknesses of quarantine flags is that they’re easily stripped; that wasn’t good enough for com.apple.macl, though, which macOS protects using System Integrity Protection (SIP), making it more robust to attack. That’s a trick you can only pull when that data is stored separately from the main file data, using an extended attribute.

Today’s quarantine flags are thus implemented using extended attributes, which originated as a replacement for resource forks, for which we have to thank Bruce Horn and other Apple engineers in the dim and distant past. Without the vision of those pioneers, macOS wouldn’t have such rich and capable features.

References

Wikipedia on resource forks.
Wikipedia on ResEdit.
Apple reference manual for ResEdit 2.1.