Trying to make sense of iCloud: Progress with Cirrus

I have switched back to working on Cirrus, the start of a tool to diagnose and attempt to fix iCloud Drive problems. At this stage, I was hoping to have a new beta-release to post here for use, but I have run into some rather strange problems which give insight into how iCloud Drive works, and its limitations.

Cirrus 1.0b2 provides regular users with an easy way of uploading a small test file to iCloud Drive, and of removing it once done. For advanced users with experience of reading the macOS unified log, it also provides tailored log access to help diagnose problems occurring during such uploads.

Working through the macOS developer documentation, I have come across information which can be obtained from files stored in iCloud Drive, and two potentially useful functions to download and evict items (I will explain these shortly). These gave me the idea of tackling problems with individual files, and of getting more information about their status, and controlling it.

Files in iCloud Drive are quite unlike ordinary files. They exist in two states: downloaded to local storage, and held in iCloud.

When a file is downloaded to local storage, there are two copies of it, one on your Mac, the other on Apple’s iCloud servers. When you change that file, you are working with the local copy, which appears to be in a special Finder location named iCloud Drive. As far as Terminal’s command line and the file system goes, there is no such volume or directory: folders in iCloud Drive are actually stored in ~/Library/Mobile Documents/com~apple~CloudDocs. You are blocked from being able to access that in the Finder, though.

So any app or other software which inspects or opens this dual file does in fact use the local copy, which has extended attributes and everything else you would expect of a Mac file.

Change your local copy of a file held in iCloud Drive, and macOS tells iCloud’s servers that changes have occurred, and uploads the changes to the servers. Other connected devices, such as another Mac or iPad, are then notified of the changes, and any local copies which they might have of that file are then updated, to keep them in sync.

If you disable Optimize Mac Storage in your iCloud pane, that is all you get: each file held in iCloud will also be stored as a local copy.

When that optimisation is enabled, macOS will optimise the use of local storage by only keeping ‘active’ files locally, and others will display the iCloud icon next to them, to indicate that no local copy is available. The process of turning a file from being stored locally and in iCloud, to being stored only in iCloud is called eviction (from local storage).

Although it may not appear so, you can’t do anything by way of editing or even previewing evicted files. macOS doesn’t stream these files live from iCloud’s servers to your app, but has to download them first. So it keeps stub files, of less than 1 KB size, in ~/Library/Mobile Documents/com~apple~CloudDocs. When anything tries to access that stub, it triggers macOS to request the iCloud servers to supply the file they hold remotely.

cirrus20

My first task in changing Cirrus to tackle individual files was to split the advanced user section to access the log into its own window.

cirrus21

Then each open document is a file in iCloud, with indicators to show its uploading and downloading status, and to let the user evict and download the local copy of that file.

This all works very nicely now, apart from two significant problems. First, most of the special iCloud attributes associated with a file return spurious results, at least some of the time, in both Sierra and High Sierra. Those may be related to the second, and more serious, problem: once a file or folder has been evicted, macOS finds it very difficult to access that file!

cirrus22

Here I have evicted a complete folder of screenshots, which are now shown with the iCloud icon next to them. Note the QuickLook thumbnail next to the selected image has a white border around it: this is because it isn’t a normal thumbnail, but is associated with the stub file which is left in local storage. With this file evicted, the Open button is disabled, as macOS can’t open the file while it is stored remotely.

cirrus23

I then clicked on another file’s iCloud icon to download it to local storage. Note how it gains a normal QuickLook thumbnail, without the white border, and now the Open button is enabled.

cirrus24

The same applies to folders containing evicted files: although this version of Cirrus can open folders, until the contents are at least being downloaded, macOS won’t let you open the enclosing folder.

cirrus25

This is what you see if an app is quick enough to try to open an evicted file: macOS looks at the stub and recognises that it can’t access the file (yet), so tells you that it doesn’t exist. At least, not until it has been downloaded.

cirrus26

Here I have managed to open a stub file using xattred, before macOS has downloaded a copy from iCloud, and it is only 714 bytes in size.

For Cirrus to do anything with files which haven’t been downloaded yet, it has to be able to open them. But it can’t open them until they have been downloaded.

Having found this catch-22, I now need to find a way around it. And that’s my long-winded apology for not being able to offer you Cirrus version 1.0b3. Yet.