Despite the distraction of macOS updates last week, I’ve been trying to get to grips with the more enduring issue of preserving metadata in iCloud. The last time that I visited this, over eighteen months ago, it all looked a bit of a disaster: if you shared documents between Macs, or with iOS devices, those metadata which weren’t actually embedded in a document were likely to be stripped. Apple’s concept of using iCloud for this purpose looked in disarray, to say the least.

There are a great many good reasons for wanting to store information about the data in a file, its metadata. In some cases, standard types of metadata become part of the data in a file, such as all those natty details about a photo which you took, including your location at the time. The same goes for author details in PDF documents.

Back in the days of Classic Mac OS, whenever you reopened a document it appeared just as when it was last open, with the same window size and location, and in exactly the same state. This was accomplished by saving that information as resources in that document’s file metadata, then the ‘resource fork’ of the file. When Mac OS X came along, those from the Classic Mac tradition wanted to retain these metadata, which were anathema to those with a pure Unix background. The compromise solution of extended attributes (xattrs) which we now have in macOS is actually more powerful than existed in old Mac OS, but less accessible to users and less popular with developers too.

Apple’s attitude to xattrs remains oddly ambivalent. In its flagship language Swift, there’s no xattr class or general support for access to xattr data. In my several apps which use xattrs, I access the C functions provided in the API through a wrapper. This isn’t a particularly high hurdle in development, but coupled with the rudimentary support of command tools seems sufficient to keep scripters and other advanced users well away. This compares with Classic Mac OS, in which Apple provided users with a free extensible resource editor, the famous ResEdit. Xattrs today are definitely not the resources of old.

But one of the most serious objections to doing anything serious with xattrs was the discovery early in 2018 that by default almost all xattrs are stripped when documents are passed through iCloud Drive. Any Mac app which writes metadata to custom xattrs is likely to hit this problem: the metadata will work fine on the Mac on which the document is authored, but the moment that you pass that document through your iCloud Drive to another Mac or iOS device, that metadata will have vanished.

The canonical illustration of this is the open source PDF reader and annotation app Skim. Rather than writing its annotations into the document data, it respects the integrity of that and saves them as custom xattrs. That’s the way that all good PDF apps should work, of course. However, because it necessarily uses its own types of xattr to do this, every one is stripped when its annotated PDFs are passed through iCloud Drive. The first time that I saw this happen, a shiver ran down my spine.

At the end of June this year, someone kindly added a comment to my original article drawing attention to this issue, providing a clue which led me to xattr flags, controlling the preservation of xattrs during various types of copy operation. Specifically included in its controls is a flag which can prevent the stripping of custom xattrs in transit through iCloud. This has been available for around six years, but documented only in brief in an old WWDC presentation, I gather, in a fairly obscure man page, and in the Open Source code provided by Apple.

Michael Tsai kindly republished some excerpts from my recent articles here in which I have tried to document this xattr flag system, and point out how it could be used by developers to ensure their metadata are preserved through iCloud. Among the comments which that elicited is an insightful thread of tweets from Pierre Habouzit @pedantcoder who is one of the team which designed and implemented xattr flags. In addition to underlining some of the important and valuable features in this system, he points out that, for iCloud file sharing, “in general we prefer packages with metadata in a plist inside the document or in a Resource/subdir of the bundle”.

This is indeed what Apple has been doing fairly consistently. The flat file standard of RTF has grown into RTFD, originally in NeXTSTEP but now only extensively supported on macOS and iOS. Some of Apple’s proprietary file formats have followed the same route. When using common file formats, though, this is seldom available, and would often be a liability. Much as we might like the flexibility and power of RTFD, the fact is that it’s almost inaccessible on any other platform, and even on the Mac there are several major apps which open RTF happily but can’t grok RTFD.

So xattrs are here to stay, and can be used in ways which work well when kept in, or transiting through, iCloud. The irony is that Apple itself now uses xattrs more than at any time in the past. Have a crawl through the xattrs in your Home folder, for example, and you’ll find a great many aren’t third party types, but quarantine flags on documents – a standard behaviour whenever any sandboxed app opens a document, even when it doesn’t save it.

What this now needs is some purposeful Swift (and Objective-C) support, documentation, and example code.