hoakley July 17, 2023 Macs, Technology, Updates

How does iCloud work? Plus a new version of Cirrus

Even if you don’t sign into your Mac using an Apple ID, it still connects to iCloud so long as it has an open internet connection. For many of us, though, iCloud and iCloud Drive are essential features of using Macs and Apple’s devices. They make it easy to share passwords, photos, documents we’re working on, and much more. This article explains the basics of how iCloud and particularly iCloud Drive now work, and provides a new version of my free utility Cirrus to help you investigate them.

iCloud is descended from a long series of features in older versions of Mac OS, going back to iTools in January 2000. It’s currently divided into two main functional services: CloudKit to handle databases, and iCloud Drive for storage of files. These appear to be implemented in the same vast distributed databases served by Apple’s own data centres and those of third-parties including Amazon’s AWS. macOS also includes some specialist tasks relying on iCloud such as the Find My service, and App Store apps are able to synchronise small shared dictionaries of key-value pairs such as preferences in an Ubiquitous Key-Value store.

CloudKit

This supports structured cloud-based databases, including the synchronisation of Core Data containers through Persistent CloudKit Container private databases. This is exceedingly complex, and uses the com.apple.cloudkit sub-system and the cloudd service in macOS.

When used to synchronise a Core Data container for an app’s database, changes to a local container are scheduled to be synchronised up to iCloud by the DAS-CTS dispatch system, and that’s performed in the background when conditions allow. Because of the time elapsed between the app making a change and it being synchronised up by CloudKit, tracing entries in the log can be difficult.

CloudKit is used extensively by bundled apps like Notes, by many third-party apps that keep their data in iCloud so it can be shared between different Macs and devices, and more. It also hosts some important system services for macOS, including the online app notarization database, one of the most important iCloud services to work without an Apple ID.

CloudKit services are controlled for macOS features in iCloud settings, in System Settings > [Apple ID] > iCloud. Third-party apps may place their control in the app’s settings, but they’re also combined with iCloud Drive access in the iCloud Drive settings: open those and click on the Apps syncing to iCloud Drive item at the foot of that dialog to see a full list and control their access to iCloud as a whole.

iCloud Drive

In contrast to CloudKit, iCloud Drive is far easier to deal with, and to understand, as it’s largely in the control of the user, and its actions are readily observable. When you copy or move a file to your Mac’s iCloud Drive, the com.apple.clouddocs sub-system and bird service upload that file to your iCloud Drive, then synchronise the information held in iCloud with that on each connected Mac and device.

Another sub-system in macOS is responsible for dividing blobs of data into the chunks used in iCloud storage: com.apple.mmcs reveals its ancient origins in its name of MobileMe Chunk Storage, or MMCS.

iCloud Drive local paths

If you’re using iCloud Drive without putting Desktop & Documents Folders into iCloud, your own iCloud Drive folders will be found in ~/Library/Mobile Documents/com~apple~CloudDocs. This roughly corresponds to the top level of iCloud Drive as seen in the Finder. However, macOS adds other folders that are actually located in ~/Library/Mobile Documents/.

If you have enabled Desktop & Documents Folders with iCloud Drive, those folders don’t include your Documents and Desktop folders. There are several ways to locate them, including following the symlink placed in ~/Library/Mobile Documents/com~apple~CloudDocs (useful for Documents only, as no symlink is created for the Desktop), or by opening those folders from the top level of your Home folder. This can become extremely confusing because of discrepancies between what’s shown in the Finder, and what can be seen elsewhere. This is all part of the illusion created by macOS with iCloud Drive.

iCloud Drive controls

There are four controls available for iCloud Drive, in addition to that giving access to third-party apps (above):

iCloud Drive switch, in iCloud > iCloud Drive;
Desktop & Documents Folders switch, immediately below that;
Optimise Mac Storage switch, at the top of the main iCloud settings;
the Finder’s contextual menu for the contents of iCloud Drive, allowing you to Download Now or Remove Download.

Storing the whole of the contents of the Desktop & Documents folders is an attractive solution for those with limited space for their Home folders, particularly when used in combination with Optimise Mac Storage. It has two potential drawbacks, though. It’s easy for the total stored in the two folders to exceed available free space, which prevents the contents from being properly backed up and can cause a crisis should you need to download most or all of what’s stored in iCloud Drive. Should you want to turn off Desktop & Documents Folders, you must plan that carefully to cope with the resulting removal of local files.

Enabling Optimise Mac Storage lets macOS determine which files in iCloud Drive will be stored locally as well (downloaded), and which will be removed from local storage and only kept in iCloud (evicted). While the wording displayed might suggest that eviction is only performed when local free space becomes low, that isn’t accurate, and macOS may decide to evict files that haven’t been accessed for a long while, and others seemingly at random.

Immediately after a file is added to iCloud Drive, it’s both stored locally and kept in sync with the copy uploaded to iCloud storage. When that file is evicted, a stub file is kept in its local place, whose name is identical with the addition of a prefix of a stop/period. The stub file is very small by comparison, and because its name starts with a stop/period, the stub file itself is hidden, although the Finder shows the original file name with an evicted icon next to it. Stub files also retain the extended attributes of the original file, together with the special extended attribute of type com.apple.icloud.itemName containing the name of the original file. Evicted files behave differently, as they can’t be searched by Spotlight, and can’t be backed up by Time Machine, as I’ll discuss in future articles. They also lack a QuickLook preview.

Logs

Custom log browsers for iCloud are available in Cirrus and Mints. These distinguish between:

com.apple.clouddocs reporting transactions for iCloud Drive,
com.apple.cloudkit reporting transactions for CloudKit,
com.apple.mmcs chunking data for iCloud storage,
cloudd and bird, as the services for CloudKit and iCloud Drive respectively.

You can obtain a similar extract using the predicate
subsystem == "com.apple.clouddocs" OR subsystem == "com.apple.cloudkit" OR subsystem == "com.apple.mmcs" OR processImagePath CONTAINS[c] "cloudd" OR processImagePath CONTAINS[c] "bird" OR processID = 0
with log show.

Cirrus

One of the problems with using the Finder to try to understand what’s going on with iCloud Drive is that it has a habit of changing the status of documents there. If you open a folder of evicted documents in the Finder, macOS starts downloading some or all of them if it can. One way to work around this is to use the custom browser built into my free utility Cirrus, which also features a simple test of uploading a 1 MB file to iCloud Drive.

I’m pleased to provide a new version of Cirrus, 1.13, which is now fully updated to work with recent versions of macOS including Sonoma. It’s available from here: cirrus113
from Downloads above, from its Product Page, and via its auto-update mechanism.

Its menu bar companion Bailiff is no longer as useful as it has been in the past, now that you can evict items in the Finder. I will be assessing it further to see if there is any value in updating it as well.

17Comments

Add yours

1

EcleX on July 17, 2023 at 6:58 am

Thanks. Is there a way to prevent iCloud asking for the password each time that the Mac is booted from two different Macs (two times in a row), as well as when opening “Apple – System Settings” (three times in a row)?

No problem when other Mac is always booted from the same (internal) disk. The issue arises when booting two different Macs from the same external disk, as said.

System: two intel iMacs (iMac18,3; mid 2017) booted from the same external SSD with macOS 13.4.1 (c) (22F770820d) Ventura.

LikeLiked by 1 person
- 2
  
  hoakley on July 17, 2023 at 7:15 am
  
  “The issue arises when booting two different Macs from the same external disk”
  Think about this for a moment. If you were automatically logged into your iCloud account when you booted any Mac from that external disk, that would be shockingly bad security, as it would bypass the protection of your Apple ID altogether.
  So the answer is no, I’m afraid, and this has been one of the known penalties of booting from external disks in this way, and so it should be to protect your iCloud account.
  Howard.
  
  LikeLike
  - 3
    
    EcleX on July 17, 2023 at 8:02 am
    
    Thanks. I understand, but as said it also asks for the Mac admin/user password when opening “Apple – System Settings” (asks three times in a row!) each time that it is opened in the very same Mac (without booting any other Mac).
    
    So, it you open System Settings, it asks for the iCloud (Apple ID) password three times in a row. And if you close System Settings and open it again even immediately, it asks again for the password three times in a row. It should be possible to prevent it. And to ask just once, not two or three times in a row.
    
    LikeLiked by 1 person
    - 4
      
      hoakley on July 17, 2023 at 8:38 am
      
      Now I’m confused. You wrote “it also asks for the Mac admin/user password when opening” System Settings, but also “it asks for the iCloud (Apple ID) password three times in a row”.
      Which is it asking for – your admin password, or your Apple ID password?
      Do you have any third-party panes installed in System Settings? Does this also occur when that Mac is booted from its internal drive, or only from the external?
      Howard.
      
      LikeLike
    - 5
      
      EcleX on July 17, 2023 at 1:46 pm
      
      Oops! Sorry. It asks for the iCloud-Apple ID password (not the Admin/user password). Actually, they are two Macs used by other person at home and work which are booted from the same external Samsung Portable SSD T5 2 TB. No third-party panes installed in System Settings. The internal disk only contains Time Machine backup in both Macs.
      
      LikeLiked by 1 person
    - 6
      
      hoakley on July 17, 2023 at 5:08 pm
      
      Thank you.
      I don’t know why that is, but suspect it’s because it’s booting from an external disk used by multiple Macs.
      Howard.
      
      LikeLike
7

SarahB on July 17, 2023 at 11:43 am

I always keep reminding myself iCloud isn’t a backup. I wish it worked differently sometimes. Like how it limits the size we can upload for one file. I remember wanting to store things there but would have to break them up into smaller chunks of files.

I still appreciate it and use it. I don’t use the desktop and documents sync thing though. I still remember paying for a .Mac account back in 2004 and it was so expensive. I found my old receipt and cost me $139 or $162.63 Canadian after tax. iCloud has never been that bad price wise for me thankfully.

I really am thankful for your apps. I only just discovered them recently but I’ve been trying to put them to use since I do like things that give me more info. iStat Menus has always been a life saver for me.

I wish apple would let us change our iCloud email but I understand why they don’t let us. A lot of stuff is tied to my iCloud ID. I’ve just been struggling with my mom trying to get her off google and onto iCloud but made a few mistakes. Since she already had a iCloud account but with a gmail address. I will keep working on creating a new account where she has no ties to google anymore.

LikeLiked by 1 person
- 8
  
  hoakley on July 17, 2023 at 5:07 pm
  
  Thank you.
  I thought the maximum file size was 50 GB now, which seems more than adequate for cloud storage. But no, it certainly isn’t intended for backups.
  Howard.
  
  LikeLike
9

DV Henkel-Wallace on July 17, 2023 at 6:38 pm

I find the opacity of icloud drive infuriating. I have some folders that show the “waiting to upload” (dotted cloud) icon and nothing happens…sometimes for months. I have one folder that shows that icon but when you open it nothing in it displays the icon — so what’s waiting to upload? (I don’t have “optimize space” enabled because I want every file (including in icloud) to appear on my backups.

This also makes me wonder about the reliability of cloudkit synchronisation.

What’s crazy is that a small company like Dropbox can get this right but a big company can’t.

LikeLiked by 1 person
- 10
  
  hoakley on July 17, 2023 at 9:18 pm
  
  There are various suggestions as to how best to address this relatively common problem, which I will be covering in a future article.
  CloudKit works quite differently and can have problems of its own – commonly seen when Macs and devices don’t seem able to sync shared databases like address books or calendars.
  Dropbox is far simpler, and smaller in scale. It’s also less integrated into macOS, which actually makes it task easier in some respects.
  Howard.
  
  LikeLike
11

Bill Stanford on July 18, 2023 at 12:59 am

Dear Howard, you wrote:

“Bailiff is no longer as useful as it has been in the past, now that you can evict items in the Finder. I will be assessing it further to see if there is any value in updating it as well.”

Your utilities are invaluable! I run Silent Knight every time I log on to a Mac; and Bailiff is such a comfort — much better than facilitating iCloud Drive from the Finder! If you can keep supporting it a lot of us will be grateful!

LikeLiked by 1 person
- 12
  
  hoakley on July 18, 2023 at 7:45 am
  
  Thank you.
  My concern with Bailiff is that it only gives easy access to Download and Evict commands, which are now readily accessible in the Finder’s contextual menu. I see no reason that Bailiff shouldn’t continue to work, but it really needs a rewrite to fit in better with more recent macOS. I’ll have a look and think about this.
  Howard.
  
  LikeLike
13

Joe Hanna on July 18, 2023 at 4:55 am

I’m reluctant to post here for support for my personal gripes, but this feels like the right time and place, so please forgive me. Are you aware of a way to temporarily halt iCloud drive syncing without fully disconnecting from the internet? This would have been a huge benefit to me several times in 2019 when I used to travel for work, and would accidentally save a large file in Desktop or Documents while connected to airplane Wi-Fi, only to have it gobble up what little bandwidth I had for the rest of the flight.

Otherwise, I live and breathe in the world of iCloud, particularly since the introduction of the Files app in iOS, and this article has been hugely helpful in my understanding of all how it all works, so Thank You!

LikeLiked by 1 person
- 14
  
  hoakley on July 18, 2023 at 7:49 am
  
  You’re always most welcome.
  In order of preference, I’d try:
  – remove download in the Finder’s contextual menu
  – Bailiff or Cirrus to evict the file.
  Because you’ve got Documents & Desktop enabled, I’d be very hesitant to try anything else.
  Howard.
  
  LikeLiked by 1 person
15

John on September 19, 2023 at 1:54 pm

I used iCloud, Time Machine, and a remote backup service called “Backblaze.” My question is: Is Blackblaze really effective if 100% of my files are not residing on my Mac hard drive? Thanks, Howard for your reply, in advance.

LikeLiked by 1 person
- 16
  
  hoakley on September 19, 2023 at 2:47 pm
  
  Like TM, Backblaze will only back up what’s stored locally. If a file has been evicted to iCloud, then it can’t get backed up by either.
  Howard.
  
  LikeLike
17

John on September 19, 2023 at 2:49 pm

Thanks, Howard. I so appreciate your rapid responses.

LikeLiked by 1 person