When it first came out with macOS Sierra, the full-on iCloud option of putting your Desktop & Documents Folders into the cloud was often a disaster. For many users, iCloud became an unseen hand which snatched their precious files away as fast as their internet connection could carry them. It only takes a few well-known examples of users losing control, or even worse losing documents, to put most of us off.

Since that troubled launch, iCloud has allowed many variations, but for general file-sharing comes in two main flavours:

• plain iCloud Drive, in which the user controls what is put into their shared folders;
• the full Desktop & Documents Folders configuration, in which macOS and iCloud manage all files in those folders.

These have the further option to Optimize Mac Storage. Apple’s description of that is:
The full contents of iCloud Drive will be stored on this Mac if you have enough space. Older Documents will be stored only in iCloud when space is needed.
However, as I have reported here, that is not the way that it works in Sierra 10.12.6 on some Macs at least, nor in High Sierra 10.13.4, and whatever optimisation macOS is supposed to perform can be overridden by user actions, although not normally under user control.

Given the rather opaque behaviour of these different variations, I have been looking at how they work out, with the aim of discovering whether the full Desktop & Documents Folders configuration is now worth considering, and how it works.

iCloud Drive

This is, I believe, by far the most popular choice for iCloud users, as it gives the user most control, and works well with smaller iCloud capacities. If, like me, your main Mac has a 2 TB internal drive with over 400 GB in its ~/Documents folder, you only have 50 GB of iCloud storage, and you are still on a slow rural internet connection, this is the option which makes more sense.

In theory, it should be irrelevant as to whether you enable Optimize Mac Storage provided that you keep ample free space on your startup volume: iCloud Drive should keep local copies of all the files that you put into the cloud, and never see any need to evict any from local storage. As there’s no way of evicting files using the Finder or any other tool bundled with macOS, most iCloud Drive users potter along with local and cloud copies of the files they move or copy to iCloud Drive.

The exceptions seem to come with High Sierra, which can underuse local storage by evicting files from there even when there is ample free space. I discovered that one of my High Sierra systems checks its local storage quota correctly, but still evicts files which are not accessed often when Optimize Mac Storage is enabled.

Folders/Containers

In the Finder, your iCloud Drive is shown as a special item in Favourites, and doesn’t exist anywhere else: it isn’t a shared item, a device, or a volume, but a special illusion of the Finder and GUI. As far as Terminal and the file system are concerned, your iCloud Drive is based on a folder/directory in your Home folder, in the path ~/Library/Mobile Documents/com~apple~CloudDocs, and the Finder and related GUI interface elements won’t let you past ~/Library/Mobile Documents; the only way to gain access to items in iCloud using Open File and similar gateways is via the iCloud Drive Favourite.

Other folders exist in ~/Library/Mobile Documents, including all the app-specific folders or Containers, but the GUI effectively prevents you from seeing inside them, even as an admin user, unless you enter via Terminal. Except for some which are shown as if they were inside ~/Library/Mobile Documents/com~apple~CloudDocs even though they are not.

For example, my ~/Library/Mobile Documents/com~apple~CloudDocs folder contains just five folders, but in the Finder that same iCloud Drive contains another dozen folders which are actually stored alongside com~apple~CloudDocs in ~/Library/Mobile Documents. This is all extremely confusing and thoroughly inconsistent.

These inconsistencies worsen, particularly in Sierra. In the Finder, my iCloud Drive folder contains 17 folders from Automator to Word, which includes the Desktop and Documents folders from another Mac, and one folder that I created for my own use. In an Open File dialog displaying hidden items, there are 37 folders shown, including those for apps such as Mellel, which are not listed by the Finder even when set to show hidden items. In fact, Mellel has not one hidden folder but two, apparently with identical names.

When listed using the macOS file system, ~/Library/Mobile Documents contains no less than 54 folders, and the apparently duplicated folder named Mellel turns out to be two folders with very different names, iCloud~com~redlex~mellel and DS2QH574BD~com~redlex~mellel, even though both are shown in the GUI has having the same name.

There are three different folder/Container naming systems working side by side:

• com~apple~Pages format, which tend to be displayed in normal Finder windows as (e.g.) Pages,
• iCloud~com~apple~iBooks format, which may only appear in File Open dialogs, etc., which show hidden items,
• F3LWYJ7GM7~com~apple~garageband10 format, which are not shown in the GUI at all, only in Terminal and the file system.

It appears that the Finder and Cocoa hide and move folders/Containers on the basis of a ruleset similar to that above. When you do want to check the contents of any of these folders/Containers, this makes it thoroughly confusing and frustrating.

These folders/Containers exist in local storage even when the files inside them have all been evicted to cloud storage.

Files

Files which are being stored locally exist as normal files within the folder/Container hierarchy, and to all intents and purposes are indistinguishable from normal local files. When a file is evicted from local storage, its place is marked by a stub file bearing a different name. This is formed by prefacing the regular file name with a stop/period to hide it from the GUI, and appending the extension .icloud.

So the downloaded file named myFile.text is replaced by a stub file .myFile.text.icloud when it is evicted from local storage. However, the GUI never displays stub file names, only the names of the downloaded files, even when the file has been evicted.

This can cause errors. For example, if an app uses a FileManager call to deliver the contents of a folder containing evicted files, the file names will be those of the stubs. If the app then calls for one of those files after it has been downloaded – perhaps triggered automatically by a GUI interaction with that file – the stub will have been replaced by the downloaded file, with its original and different name.

The risk of errors is reduced by referencing the files by URL rather than path. However, that may in turn force a long wait until a file has been downloaded before it can be accessed. As such delays are peculiar to iCloud and their duration can extend to many minutes, catering for these behaviours properly in apps introduces an added layer of complexity, and ample opportunity for bugs.

Dealing with files stored in iCloud is an even greater problem in shell commands and scripts. If a file had been evicted from local storage, it is no longer accessible through its original name, but accessing it through the stub name – which is what is returned by ls, for instance – returns the binary property list of the stub file, not the original. I know of no shell command which will force the downloading of files which have been evicted to iCloud storage, which could be used to ensure that there were no stub files in an iCloud Drive folder.

Desktop & Documents Folders

My new MacBook Pro is an obvious candidate to use the full-on Desktop & Documents Folders option, as it has relatively small local storage (512 GB) and very little cluttering up its Documents folder yet. One trick worth thinking about is moving items which you don’t want given the full iCloud treatment into another folder before enabling this option. I created a folder ~/OtherDocuments into which I moved a large and complex folder of Xcode projects, for example.

Once you have enabled Desktop & Documents Folders in the iCloud pane, macOS makes some rearrangements for you, whether you like them or not. The Documents and Desktop folders vanish from your Home folder, and reappear in iCloud Drive. All other devices which connect to that same iCloud Drive will then also see that user’s Documents and Desktop folders too, which can be a benefit, or a problem.

What you then get is iCloud Drive, with its quirks as detailed above, and special rules applied to your Documents and Desktop folders, which are now also based in iCloud.

Folders and files

The Finder now treats three items – iCloud Drive, Desktop and Documents – as sidebar items in their own right, classified under iCloud. Once handed over to iCloud, these are now your only normal means (in the Finder) of accessing your Desktop and Documents folders.

Unfortunately, macOS does not incorporate your Desktop and Documents folders into its folder hierarchy in ~/Library/Mobile Documents, and the behaviour of Finder windows and GUI interfaces such as the Open File dialog are extremely confusing.

In Finder windows, you will find ~/Documents in the special iCloud item Documents and in the iCloud Drive item, but not in your Home folder unless you opt to display hidden files, when the Documents folder does appear in your Home folder as if nothing had happened. The same applies to ~/Desktop.

In the GUI interface more generally, such as Open File dialogs, ~/Documents can appear in four places:

• in the special iCloud item Documents,
• in your iCloud Drive as Documents,
• in your iCloud Drive as an alias (not a Finder alias, but a symbolic link) also named Documents (normally hidden),
• in your Home folder as Documents (normally hidden).

Desktop behaves similarly, although no alias is made to it in iCloud Drive.

In the file system (e.g. FileManager calls), and in Terminal, ~/Documents hasn’t moved at all, but can now contain iCloud stub files when items are evicted. These follow the stop/period suffix and .icloud extension as described above, and have exactly the same issues with access from apps and in Terminal. There is also a symbolic link (of all things) in ~/Library/Mobile Documents/com~apple~CloudDocs to the ~/Documents folder, but none to ~/Desktop.

If the situation with plain iCloud Drive is confusing and tough to handle in shell commands, that with Desktop & Documents Folders seems impossible if a user continues to store most of their documents in their Documents folder. I’d be very interested to hear of solutions which work with this iCloud option, and cope with evicted items, stub files, and their naming system.

At this stage, I think that I’ll go and lie down in a quiet, darkened room for a few hours…