hoakley April 2, 2018 Macs, Technology

Inside iCloud Drive: Pushed updates, that syncing feeling

The two common scenarios in which iCloud Drive appears to become stuck, and cause long waits for users, are uploading files, and propagating such uploads (and other changes) to other clients connected to the same account.

In my previous article I looked at the steps involved in uploading. This article looks at propagating those changes to other clients, but stops short of them downloading new or changed files.

To investigate this, I ran my tool Cirrus simultaneously on two Macs connected to the same iCloud account, on the same local network. I ran the standard 1 MB upload test in Cirrus on one, then watched for that new file to be propagated via iCloud Drive to the other. I did this with privacy disabled, and examined both log extracts, and a full log extract on the recipient Mac, using Consolation. I’ll refer to the uploading Mac as A (running 10.12.6), and the syncing Mac as B (running 10.13.4).

Mac A’s progress icon went very quickly to over 90% complete, as it usually does, then paused before completing within 5 seconds. The last delay coincides with the phase identified by CloudDocs as sync down, which runs after the actual upload is complete, and is described by CloudKit as FetchRecordZoneChanges.

This phase started at 4.207 seconds time elapsed, and completed at 4.462 seconds, at which point Mac A was finished.

Receipt of the push notification from APS to Mac B of changes to iCloud Drive was logged at 4.310 seconds time elapsed, well before the completion of the final phase on Mac A.

The log entry first recording the push notification is worth noting:
04.310 com.apple.apsd courier Stream processing: complete yes, invalid no, length parsed 249, parameters { APSProtocolCommand = 10; APSProtocolMessageExpiry = "1970-01-01 18:12:15 +0000"; APSProtocolMessageID = <e9c74616>; APSProtocolMessagePriority = 5; APSProtocolMessageTimestamp = 1522602544155347565; APSProtocolPayload = <…>; APSProtocolToken = <…>; APSProtocolTopicHash = <…>; }
I hope that the expiry given means that the message never expires.

This was then interpreted by apsd as:
04.311 com.apple.apsd courier-oversized Received message for enabled topic 'com.apple.icloud-container.clouddocs.com.apple.CloudDocs' onInterface: NonCellular with payload '{ ck = { ce = 1; cid = "com.apple.clouddocs"; fet = { dbs = 1; sid = "subscription/com.apple.CloudDocs"; zid = "com.apple.CloudDocs"; zoid = "…"; }; nid = "…"; }; }' with priority 5

Key events in its processing include:
04.311 apsd asked to store incoming message <…> with guid … environment …
04.311 com.apple.apsd courier Stream processing: complete no, invalid no, length parsed 0, parameters (null)

The next entry is important, and records that low-priority notifications are then deleted, which could of course explain some problems:
04.311 apsd deleting low-priority incoming messages for topic com.apple.icloud-container.clouddocs.com.apple.CloudDocs environment production 04.311 apsd APSMessageStore - APSIncomingMessageRecordDeleteMessagesForTopic com.apple.icloud-container.clouddocs.com.apple.CloudDocs 1 environment production 04.316 apsd APSMessageStore - New message record [<…>{}] has ID [8726]. 04.316 apsd calling completion block for incoming message … 04.316 com.apple.apsd courier Sending acknowledgement message with response 0 and messageId <…> (3922150934) 04.317 com.apple.apsd courier _notifyForIncomingMessage … with guid … 04.317 com.apple.apsd connectionServer Dispatching low priority message: <xpc object> 04.317 apsd Looking up connection on peer: … found … 04.318 com.apple.apsd connection bird ApplePushService … Delivering message from apsd: … … com.apple.icloud-container.clouddocs.com.apple.CloudDocs 04.318 com.apple.apsd connection bird ApplePushService … responding with an ack for message with guid …

At this stage, bird has received the pushed notification, and this is announced by CloudDocs at the start of the sync sequence:
04.318 CloudDocs [NOTICE] received a push for client zone com.apple.CloudDocs
CloudDocs then starts the sync down, which is announced by bird.

This triggers a long dialog between cloudd and CloudKit to perform the sync down:
04.322 CloudKit starts CKFetchRecordZoneChangesOperation
which is then echoed by cloudd as fetch-record-changes.

Next, CloudKit starts CKDFetchRecordZoneChangesOperation, announced by cloudd as db-operation, and CloudKit starts CKDRecordFetchAggregator.

cloudd announces url-request then c2-request-task, and at this stage DuetActivityScheduler (DAS), one of the main activity scheduling and dispatching services, becomes involved, submitting activities for this to one of its activities groups, com.apple.cloudkit.BackgroundConnectionPool. Note that this is scheduled to a pool of background connection activities, which may explain why sync down can become greatly delayed.

After cloudd announces url-request-transmission, there is a loop to fetch the new/changed records for iCloud Drive. This runs in sequence:
04.830 cloudd announces fetch-records
CloudKit starts CKDFetchRecordsOperation
cloudd announces decrypt-records
CloudKit starts CKDDecryptRecordsOperation
cloudd announces db-operation, then starts its state transition sequences
04.849 CloudDocs [INFO] got a record
CloudKit finished CKDFetchRecordsOperation
and then recycles with cloudd announcing fetch-records again, until all the records are fetched.

Once those are completed:
04.863 bird announces did-sync-down
CloudDocs [NOTICE] received 2 edited items from the cloud for com.apple.CloudDocs
CloudKit finished CKDFetchRecordsOperation, CKDRecordFetchAggregator, CKDFetchRecordZoneChangesOperation
04.868 bird finish-changes
CloudKit finished CKFetchRecordZoneChangesOperation – sync down (push triggered)

This completes the sync-down, and the first run through the iCloud progress icon. At this stage, iCloud Drive shows the new/changed items, but they have not yet been downloaded to Mac B. That sequence starts with the announcement:
00:04.871 CloudDocs [NOTICE] we will now download new documents automatically because the account contains less than 5.37 GB 00:04.913 CloudDocs [NOTICE] downloading 1 documents in com.apple.CloudDocs
then bird calls dl-content and client/fetch-records, for CloudKit to download (opportunistic). I will analyse the download process in another article shortly.

The sync down sequence on Mac B is summarised in the following chart. Note that MMCS is not involved at all (judging by its complete lack of log entries) in this sequence, as no bulk transfers take place.

This suggests several good reasons for a Mac or iOS device failing to sync promptly:

iCloud may not push the notification promptly
the pushed notification may not be received, relying on iCloud to send another when that first message passes unacknowledged
the pushed notification may get deleted by higher priority messages
bird, CloudDocs, or CloudKit may stall part way through the sync down sequence.

It should be fairly straightforward to distinguish between these using the log obtained with Cirrus.

These also explain why uploading a file to iCloud Drive could clear a sync down failure, by forcing a fresh sync down of its own. That should include all earlier changes too.

My concern over performance is the involvement of DAS and the use of its pool of background connection activities. If that pool is busy, iCloud Drive sync down could be delayed considerably. There doesn’t appear to be any way to influence that, which is a design decision on Apple’s part.

12Comments

Add yours

1

Thomas Lichy on April 3, 2018 at 1:38 pm

Hi Howard, I love your iCloud investigations, and also your art articles (being a person of multiple interests myself). I have a question / challenge for you, which could be a topic for an article by you:

How would one keep a local copy of multiple iCloud Drive accounts?

For example, a family using iCloud Family Shared Storage over various mac laptops with various Apple ID accounts, or another example, a couple each with their own Apple ID & laptop. Due to the wonders of Optimise Storage in High Sierra and Apple’s rather small SSDs in laptops, many files and photos only exist in iCloud Drive online. Therefore to be safe, one should keep a local copy on, if possible, a spare Mac with a large HDD.

That’s exactly the situation I’m in. And Apple Tech support have just told me ‘it can’t be done’! Grr. Gaping hole in Apple’s iCloud provision, and a bit of a red flag. Maybe there’s a way to have OSX (or OS Server) rotate between iCloud accounts and syc the local storage?

LikeLiked by 1 person
- 2
  
  hoakley on April 3, 2018 at 1:54 pm
  
  Thank you. That sounds a serious challenge – particularly if Apple can’t suggest a solution!
  Apple’s assumption is, of course, that iCloud Drive is infallible. But if you do lose anything from it, they also expect you to have kept a backup.
  In theory, Time Machine backups should include what you have on iCloud Drive, and anything which has been moved to iCloud using the Optimize Storage option. But several users have complained that they cannot find those backups when they needed them.
  Have you tried looking in Time Machine backups?
  Howard.
  
  LikeLike
- 3
  
  Damian on April 4, 2018 at 12:20 am
  
  What if you created separate user accounts on the “spare Mac with a large HDD”, and then logged each user account into its relevant iCloud account? Theoretically, that Mac should keep its iCloud Drive files synced and stored on the HDD, and you could connect it to a Time Machine drive for backup and versioning.
  
  That’s basically what I’m doing with my setup, but with one iCloud account. My iMac has a large Fusion drive and is set to keep downloaded all iCloud Drive and iCloud Photo files, and acts (if all is working as it should) as the central archive for all my digital assets. My laptop with its tiny 128 GB SSD is relieved of the burden, and just holds whatever I happen to be working on at the moment.
  
  LikeLiked by 1 person
  - 4
    
    hoakley on April 4, 2018 at 7:21 am
    
    Unfortunately, the spare Mac can only be logged into a single user account and iCloud account at one time. You can’t keep it in sync with multiple user accounts simultaneously, can you?
    In which case, that solution works fine for one iCloud Drive, but not at all for others.
    I’d also note that versioning is not part of Time Machine – versions are operated separately by the local file system onto that Mac’s local storage, and aren’t shared across iCloud Drive, as they are stored separately from the working copy of each document.
    I think that you have separate control over Photos libraries (as well as iTunes movies), which are also confusingly known as Optimize Storage.
    Howard.
    
    LikeLike
  - 5
    
    Damian on April 5, 2018 at 10:54 pm
    
    Why could the spare Mac not be logged into multiple user accounts at a time? I have mine logged in to two user accounts right now. Although this account is active, apps and services are also running in the other, which I can see through Activity Monitor, and can readily switch over via Fast User Switching at any time. I *believe* iCloud (and other) syncing is happening in my other account even as this one is active — and even if it wasn’t, would’t it just be a matter of bringing up the relevant account once in a while to sync files so they can be stored on the Mac’s hard drive? Thomas above asked how one could keep a “local copy of mutiple iCloud Drive accounts” and I am pretty certain what I described will work — or will work as well as iCloud Drive works in the first place, anyway.
    
    LikeLiked by 1 person
    - 6
      
      hoakley on April 5, 2018 at 11:06 pm
      
      OK, maybe I was wrong, maybe you can.
      I eagerly await a demonstration, as I haven’t heard of anyone doing it yet.
      Howard.
      
      LikeLike
  - 7
    
    Damian on April 5, 2018 at 10:56 pm
    
    (and, I’d add, multiple user accounts can absolutely be pointed to different iCloud accounts — in fact the first thing you see when you log into a new user account is a dialog asking you to input iCloud credentials)
    
    LikeLike
    - 8
      
      hoakley on April 5, 2018 at 11:07 pm
      
      In fairness, I think that part was self-evident. I’ll wait and see whether you can have several concurrent users each connected to their individual iCloud accounts and receiving updates to their iCloud Drives.
      It would make quite a spectacular demo – and prove a very important point.
      Howard.
      
      LikeLike
  - 9
    
    Damian on April 6, 2018 at 6:21 am
    
    Just tested this. I have a secondary user account on my iMac (High Sierra), and it’s logged into an extra Apple ID I use to administer a shared iPad in our household. I dropped a goodly number of files into its iCloud Drive folder, 3 GB or so, and of course they synced and showed up on the iPad’s “Files” app (why wouldn’t they?) I even put another GB of files in iCloud Drive on the Mac and then quickly switched over to my main user account before they had a chance ot finish uploading. After a while, the files were eventually visible on the iPad, even though the user account associated with its Apple ID was running in the background. As far as I know, this is all normal “it does what it says on the box” behavior from MacOS and iCloud.
    
    LikeLiked by 1 person
    - 10
      
      hoakley on April 6, 2018 at 6:39 am
      
      Thank you. It would be interesting to know how well that worked the other way around (multiple uploading devices -> single Mac with multiple concurrent users and accounts), and how it scaled up to several shared iCloud accounts within a family, which was I think the starting question.
      I’m not sure which box you’re reading this info from, but would appreciate a link to it. Getting information about what precisely iCloud is supposed to do, and how it does it, hasn’t been easy. Is there a spec which states how many simultaneous accounts can be open in this way on a single Mac, for example? Could this be used in a whole class, perhaps?
      Howard.
      
      LikeLike
    - 11
      
      Damian on April 6, 2018 at 6:59 pm
      
      I think what I tested does what the original poster was asking about. Files from the iPad show up on the corresponding MacOS user account, syncing in both directions as expected. One can readily test this using a Mac, a spare Apple ID, and another iOS or MacOS device. I don’t know how much syncing the Mac is going to do in the background, but surely switching over to the appropriate account once in a while is going to keep it active. I’m not an engineer or anything and I just ran a pretty casual (and very unscientific) test and found it all works as advertised. No idea how far up you could scale it, but I’d *imagine* system resources would be taxed once you hit some kind of threshold of logged-in user accounts.
      
      LikeLiked by 1 person
    - 12
      
      hoakley on April 6, 2018 at 10:21 pm
      
      Thank you.
      Going back to the original problem posed – and apparently without any solution from Apple – it was:
      How would one keep a local copy of multiple iCloud Drive accounts?
      
      To make this feasible, I think you need a system which can be run unattended, even headless, perhaps an older iMac or mini, which will sit there automatically harvesting changes to iCloud Drive and synced services to local storage, which would then be backed up every hour to Time Machine. It sounds like you may have the basis for a solution, but someone would need to implement it fully and see whether it worked with ‘typical’ use and, most important of all, could actually recover data from its backup.
      
      I’m very grateful for what you have done and reported – it shows a potential solution which needs proving in practice, something which from Apple support’s answers to the OP doesn’t appear to have been done yet. Which should be surprising really – that should be part of product testing and user documentation.
      Howard.
      
      LikeLike

·Comments are closed.

Share this:

Related