A lot of macOS is designed to ‘just work’, and generally it does. In order for things to just work, macOS often creates an illusion, like Time Machine’s local snapshots and backups. Unfortunately when several systems that just work through illusions are brought together, the result can just fail, and the illusions prevent explanation. In this article, I look at such a collision which can have serious consequences: iCloud and backups.

The basics of iCloud are complicated enough on their own. What appears to be a single cloud service actually handles quite different types of data:

• user documents stored in user folders in iCloud Drive,
• app documents stored in app-specific folders in iCloud Drive,
• shared databases for apps like Contacts and Notes,
• specially-secured shared databases like Keychain.

In this article, I deal only with the first two of those, and defer examination of iCloud’s databases until I’m feeling strong enough.

Basic model

At its simplest, documents stored in iCloud Drive consist of two copies, one kept locally, and the other in iCloud. When you make a change in the local copy, background processes upload the changed document to update the copy in iCloud Drive. When you change the copy in iCloud Drive using a different Mac or device, background processes download the changed document to update your local copy.

This synchronisation becomes more complicated when storage optimisation is enabled, for instance by ticking the box to Optimise Mac Storage in the iCloud section of the Apple ID pane. Apple claims then:
“The full contents of iCloud Drive will be stored on this Mac if you have enough space. Older documents will be stored only in iCloud when space is needed.”
As this appears to make good sense, and in some cases becomes essential to cope with all the files stored in iCloud, it hands over control of iCloud Drive to macOS. As we know well, that means that some docs will no longer be stored locally, but ‘evicted’ to iCloud. When that happens, the Finder marks them with the familiar icon consisting of a cloud with a downward arrow.

Evicted documents don’t exist locally, but macOS keeps a stub file in their place. Instead of MyDocument.extn, your Mac’s local file system contains the stub named .MyDocument.extn.icloud and, in some cases, another stub file named .MyDocument.extn-tef.icloud (which may be the remains of a stub file from an older version of iCloud). Stub files are typically less than 200 bytes in size, so result in the freeing of significant storage space. However, as the stub file is just a special form of link to the full file in iCloud, it can’t fulfil many of the original’s functions. In particular, if it’s copied into a backup, it can’t recreate the data in the document. So as a backup it’s useless, and is normally not backed up at all.

The first and most important lesson is that evicted documents can’t be backed up locally.

If you want instant access to any documents in iCloud Drive, you shouldn’t enable storage optimisation, although in the past (at least) that hasn’t made much difference, in that iCloud evicted old documents even when optimisation wasn’t enabled.

Another important consideration is that your view of the need to evict documents will always be different from that of macOS. Even when you keep ample free space on local storage, hundreds of GB perhaps, you’ll find that macOS quietly evicts files which haven’t been accessed for a long time. Apple doesn’t explain how macOS makes such decisions.

The illusion

Access to files in iCloud Drive is made very simple in the Finder.

For example, all the Numbers spreadsheets I have stored in iCloud are listed in the Numbers folder in iCloud Drive, almost all showing the icon which indicates they have been evicted from local storage. If Time Machine or any other backup software were to try backing this folder up, there’s only one file, Untitled 2.numbers, which would be copied into its backup. But when you look for the location of iCloud Drive in Terminal, for example, it simply doesn’t exist. It’s a figment of the Finder’s imagination, an illusion.

If you want to follow this along live on your Mac, open my free app Cirrus, use the Open command in the File menu and select iCloud Drive. You’ll see the same contents as in the Finder. Now, from the app’s Window menu, open a browser. This opens in a folder named com~apple~CloudDocs which is similar to the iCloud Drive folder in the Finder, but missing many of its contents, including the Numbers folder. Scroll down to the bottom of the list, and double-click the up one folder entry at the end.

The browser now lists the contents of ~/Library/Mobile Documents, which is the local root for the iCloud virtual file system. Note that isn’t always the case: in some beta-releases of Catalina, Apple moved it elsewhere.

To navigate to the Numbers folder in iCloud Drive, double-click the com~apple~Numbers folder.

Now double-click the Documents folder.

Here are all the stub files for my Numbers documents which have been evicted, and at the end the single file which remains stored locally. Numbered folders contain Numbers’ databases.

Disentangling the layouts, what the Finder does is synthesise the contents of its iCloud Drive folder from any user folders you have created, document folders for third-party apps, and those for Apple’s special apps like Numbers. The true file system paths to those are very different, as shown in the following diagram.

Third party app folders can appear within com~apple~CloudDocs, iCloud~com~dev~name, and 1ABCDE1234~com~dev~name class folders. That depends on whether that app is run from a sandbox, which version of iCloud is running, the Mac or device which created the folder, the app’s own implementation, and other factors. What appears so simple in the Finder is a complex illusion.

The second lesson is that trying to identify and specify folders within iCloud Drive is a nightmare, and is at high risk of failure or error. Cirrus’s browser can be helpful in identifying paths.

Snapshots and backups

Apple doesn’t explain how Time Machine backups work with the complex structure of iCloud folders on local storage. I report here my own experience, based on an M1 Pro NBP running macOS 12.2.1 with a vast amount of free disk space, always more than 1.5 TB, backing up the whole of its Data volume to an external SSD using Time Machine to APFS. Your mileage may vary, of course.

Time Machine makes copies of documents from different paths differently, and they’re handled differently according to whether the copy is retained by a local snapshot, or a formal Time Machine backup.

Documents stored in user folders created by the user in iCloud Drive appear to be backed up most reliably, and appear in both snapshots and backups. That assumes, of course, that they aren’t evicted and left as stub files.

Documents stored in folders created and maintained by apps are less reliable. Those of third-party apps appear to behave as user documents, in both snapshots and backups, so long as they aren’t evicted. However, documents in the folders managed by Apple’s ‘special’ apps using paths of the form com~apple~specialapp, including Numbers, Pages and Keynote, aren’t reliably backed up by Time Machine.

I have already shown you the contents of my Numbers folder in iCloud Drive, both in the Finder and using Cirrus’s browser. This is all that a Time Machine backup has to show for that folder. The only document that’s available is the one which hadn’t been evicted by another Mac. Quite why the eviction of files on a different Mac should seem to determine whether documents can be backed up elsewhere remains a mystery.

Note in this case the long path shown at the foot of the window, which confirms that this view is of the Time Machine backup created at 1717 on 18 February 2022.

The Time Machine app reveals that just two hours later all the documents in that Numbers folder are available from backup. However, as the path shown at the foot of the window reveals, those copies of the documents aren’t in one of the backups, but in a local snapshot made at 1917 the same day.

The third lesson, then, is that you can’t rely on Time Machine to back up documents stored in iCloud Drive. It might do a good job, or it could omit them completely from its backups.

Could you do better using a different backup utility? Maybe, but you’ve still got to negotiate the same problems, that evicted files can’t be backed up, and you’ll have to explicitly include directories in ~/Library/Mobile Documents if you want to include documents stored by Apple’s special apps, and perhaps some third-party products too. If you’re interested in using Carbon Copy Cloner, Mike Bombich has just added further detail to his online documentation for CCC version 6.

Conclusions

• Evicted documents can’t be backed up.
• Identifying and specifying paths to iCloud contents is complicated, and often results in errors.
• Documents in user-created folders on iCloud Drive are backed up most reliably.
• Documents in folders created by third-party apps on iCloud Drive also appear to be reliably backed up.
• Documents in folders created by Apple’s special apps should appear in local snapshots, but are not likely to be backed up.
• To ensure that important documents kept in iCloud Drive are backed up, you should keep a local copy of them in a local folder which is included in your backups. Keeping that in sync with the copy held in iCloud is essential.
• Backups must be inspected and tested regularly to ensure that they contain what you expect, and that older versions of files can be retrieved reliably. It’s no use discovering a problem six months later.

Further reading

iCloud File Management for developers
iCloud Drive User guide from Apple
iCloud Security overview from Apple
Archiving and copying from iCloud (Apple)
Security of iCloud Backup from Apple Platform Security guide.