Early last year, I announced here that iCloud didn’t preserve file metadata in the form of extended attributes (xattrs) – an article which has proved one of the most popular of all those published here. I’m pleased at last to be able to report that this can be fixed, as rather than being a bug, it turns out to be a feature. This article takes an even deeper dive into xattrs to explain one of their most remarkable properties, one which seems to have escaped even the eagle eye of the great Jonathan Levin and his series of books on *OS internals.
When I looked at this issue 18 months ago, I found that passing files through iCloud Drive resulting in the stripping of some extended attributes, including (at that time):
- com.apple.FinderInfo
- com.apple.metadata:_kMDItemUserTags
- com.apple.ResourceFork
- com.apple.serverdocs.markup
- net_sourceforge_skim-app_ series
- net_sourceforge_skim-app_notes
- net_sourceforge_skim-app_rtf_notes
- net_sourceforge_skim-app_text_notes
- co.eclecticlight.[any]
Xattrs which were preserved included:
- com.apple.TextEncoding
- com.apple.metadata:kMDItemDownloadedDate
- com.apple.metadata:kMDItemWhereFroms
- com.apple.metadata:kMDLabel_ series
- com.apple.quarantine
This suggested some form of whitelist was being used, and we speculated that this may have been for security purposes. With no clue as to the location of that whitelist, I didn’t see a way to alter this behaviour. Then at the end of June this year, a comment added to the original article offered a way ahead. This clue was given by John Rodo (to whom I am eternally grateful), who referred me to xattr_flags.h, one of the less well-known header files for macOS. Coupled with the even more informative open source file xattr_flags.c, this article explains what I have pieced together.
In 2013, as part of its enhancements for iCloud in particular, Apple added support for flags on xattrs to indicate how those xattrs should be treated when the file is copied in various ways. Rather than change the file system, Apple opted for what is perhaps best seen as an elegant kludge: appending characters to the end of the xattr’s name.
If you work with xattrs, you’ve probably already seen this in xattrs whose name ends with a hash # then one or more characters: that’s actually the flags, not part of the name, what Apple refers to as a ‘property list’. To avoid confusion I won’t use that term here, but refer to them as xattr flags. A common example of this is com.apple.lastuseddate#PS, which is seen quite widely.
Flags can be upper or lower case letters C, N, P and S, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets (enables) that property, whilst lower case clears (disables) that property. The properties are currently:
- C:
XATTR_FLAG_CONTENT_DEPENDENT, which ties the flag and the file contents, so the xattr is rewritten when the file data changes. This is normally used for checksums and hashes, text encoding, and position information. The xattr is preserved for copy and share, but not in a safe save. - P:
XATTR_FLAG_NO_EXPORT, which doesn’t export the xattr, but normally preserves it during copying. - N:
XATTR_FLAG_NEVER_PRESERVE, which ensures the xattr is never copied, even when copying the file. - S:
XATTR_FLAG_SYNCABLE, which ensures the xattr is preserved even during syncing. Default behaviour is for xattrs to be stripped during syncing, to minimise the amount of data to be transferred, but this will override that default.
These must operate within another general restriction of xattrs: their name cannot exceed a maximum of 127 UTF-8 characters.
The sought-for ‘whitelist’ is actually baked into the xattr flag code, where as of 2013 the following default flags are set for different types of xattr (* here represents the wild card):
com.apple.quarantine– PCScom.apple.TextEncoding– CScom.apple.metadata:*– PScom.apple.security.*– Scom.apple.ResourceFork– PCScom.apple.FinderInfo– PCS
Which explains some of my original observations from 18 months ago, although it would appear that some default flags have changed since 2013, or the C flag has special meaning in iCloud.
If you want a xattr preserved when it passes through iCloud, you therefore need to give it a name which ends in the xattr flag S, such as co.eclecticlight.MyTest#S. Sure enough, when xattrs with that flag are passed through iCloud Drive, those xattrs are preserved even if the default rule would treat them differently. Similarly, to have a xattr which is stripped even when you just make a local copy of that file, append #N to it.
In the next article, to appear very shortly, I’ll show how these xattr flags work in practice with iCloud Drive in Mojave, and provide a test app which can preserve all the xattrs on files which are passed through iCloud Drive.
