How iCloud marks the place of documents stored remotely

Today’s little insight into iCloud internals comes by way of a unique extended attribute (xattr) attached to files that aren’t really there at all:

I have been surveying my Macs for all the different types of xattrs which are present, and came across one which didn’t exist on my Sierra 10.12.6 system, but was present in quite large numbers on my High Sierra 10.13.2/10.13.3 system. It looked as if it was used in High Sierra and possibly iOS 11 to address issues over document naming in iCloud.

An example might be the xattr attached to the file /Users/hoakley/Library/Mobile Documents/com~apple~CloudDocs/backup1/.201605-299manageStorage.txt.icloud, which is shown in the Finder as being iCloud Drive/backup1/201605-299manageStorage.txt: this contains the text 201605-299manageStorage.txt, which is the file name.

At least it does when examined on my MacBook Air running High Sierra. I have been unable to find any such xattrs from my iMac running Sierra.

Worse still, the iMac and MacBook Air seem to be looking at quite different iCloud Drives, although they are connected to the same one. When run on the High Sierra system, my xattr scanner finds that the backup1 folder contains 235 files, of which 152 have xattrs, and they have a total of 362 xattrs of 26 different types. On the Sierra system, the same app finds five more files, of which 155 have xattrs, with a total of 392 xattrs of 36 different types.

Is iCloud Drive some sort of chameleon?

Like my mystery xattrs and files, it is and it isn’t. I think that I can now explain what is happening, the purpose of the xattr, and how iCloud Drive handles local and remote storage.

When your Mac is connected to iCloud Drive, macOS tries to cache iCloud Drive’s contents locally, to the volume seen in the Finder as iCloud Drive, but represented in Terminal’s Unix-style path of ~/Library/Mobile Documents. On my iMac running Sierra, that local caching is normally maintained for the entire contents of my iCloud Drive, as it has ample free disk space and a wired internet connection.

My MacBook Air has less free disk space, and a more intermittent Wi-Fi network connection. There, macOS opts to cache much of the contents of iCloud Drive locally, but leaves some files in the cloud, with local placeholders. In the Finder, these are represented by the small iCloud icon next to the filename.

In Terminal, those placeholders consist of a stub file of about 174 bytes, whose name starts with a stop/period. Those stub files contain that small amount of data and a xattr of type which contains the real name of the original file, still stored remotely. When you then fetch that file, for example by opening it, iCloud gets the file of that name, caches it locally, and the stub file with its xattr is removed.

If you’re snooping around in the file system, as I am when seeking out extended attributes, any stub files found there can cause confusion. Apart from the stop/period in front of their name, and their lack of content, they appear to shell scripts and apps just like real files, as they are. If you have scripts or apps which are likely to examine the contents of ~/Library/Mobile Documents or its Finder analogue of the iCloud Drive volume, your code needs to be mindful of the fact that it may encounter stub files and get thoroughly misled.

So xattrs of type are the only ones which exist solely in iCloud. They can’t be passed to a client system (unless by an unfortunate accident), and iCloud is careful to strip them from files which are copied or moved into cloud storage.