hoakley October 13, 2020 Macs, Technology

How macOS schedules and dispatches background tasks using CTS 1

In most traditional Unix-family operating systems, running a task every hour, day, or week is accomplished using cron and configured using a crontab, a scheduling table which could, for example, be used to run Time Machine’s timed backups. Although Mac OS X and now macOS have retained support for cron and crontab, they have long been deprecated in favour of launchd, which is the most important process running in macOS after the kernel itself.

Using launchd isn’t difficult, and a matter of setting up a property list in one of the LaunchAgents or LaunchDaemons folders, following the instructions detailed in Apple’s guide. There are plenty of tools to help you do this, from Peter Borg’s generic Lingon to specialist apps which use this to customise Time Machine backup scheduling.

Much older versions of macOS have suffered problems with this system (and with cron), but many Apple and third-party products now put their trust in launchd working, and it very seldom disappoints.

Some time after 2011, it appears that Apple started moving its own scheduled and background services, like Time Machine backup, to use a novel dispatching system involving two services, Duet Activity Scheduler (DAS) and Centralized Task Scheduling (CTS), the latter being intimately related or a part of the lightweight communication and dispatching system XPC. No one outside Apple seems to know when this happened, as DAS and CTS are almost completely undocumented.

By Sierra’s release, the DAS and CTS dispatching system managed more than seventy activities at most times, one of which is Time Machine’s scheduled backups. However, in Sierra this system has a bug which results in its breakdown: backups suddenly become irregular or stop altogether, and the other activities also become unreliable. That was fixed in High Sierra.

Among the heaviest users of CTS in Catalina (on my desktop iMac Pro, at least) are the following:

com.apple.accounts.cleanup
com.apple.AddressBook.ScheduledSync – Contacts syncing
com.apple.backupd-auto – Time Machine automatic backups
com.apple.bird.stage.gc – iCloud
com.apple.calendar.AlarmScan – Calendar event warnings
com.apple.cloudkit.assetHandleExpiration – iCloud
com.apple.CoreAnalytics.Daily
com.apple.coreduetd.people
com.apple.FileProvider.cache-delete.push – cache maintenance
com.apple.HMFoundation.Message.Dispatcher.Index
com.apple.ical.sync.x-coredata://C3502861-F6AE-44DA-AAE8-9318B23D56A7/CalDAVPrincipal/p5 – Calendar syncing
com.apple.knowledgestore.sync
com.apple.mds.Health check (2).0
com.apple.mediaanalysisd.fullanalysis
com.apple.mediaanalysisd.photosanalysis – Photos library image analysis
com.apple.messages.messageSyncing – Messages syncing
com.apple.parsec-fbf.flush
com.apple.photoanalysisd.backgroundanalysis – Photos library image analysis
com.apple.photolibraryd.curatedlibraryprocessing
com.apple.routined.locationAwareness.heartbeat
com.apple.Safari.SafeBrowsing.BrowsingDatabases.Update – Safari Safe Browsing updates
com.apple.SafariShared.WBSPeriodicActivityScheduler
com.apple.searchd.expirations
com.apple.suggestd.persist-stats
com.apple.suggestions.harvest
com.apple.syncdefaultsd
com.apple.timed.ntp.wanted – checking and setting the clock.

You’ll notice that not one of those listed is a third-party service, despite the fact that Apple provides a developer interface to CTS which is quite easy to use. As far as I’m aware, though, the user can’t schedule their own tools or scripts to be run by CTS, in the way we can with launchd.

CTS aims to ‘intelligently’ decide when to perform its tasks, according to provided criteria. It’s typically used for background tasks which need to be performed repeatedly, where the exact interval between runs isn’t critical. Time Machine backups are a good example: they don’t need to be run at precisely one hour intervals, so CTS can decide the best time, according to energy usage, thermal conditions, and loading on the processor cores.

Background tasks which are registered with CTS are in fact scheduled by a different subsystem, DAS. This maintains lists of activities, with their priorities and the time that each is next due to be run. It periodically re-computes a priority score for each, on a scale from 0 to 1, and compares those against a threshold. Activities which are due and exceed the threshold are then dispatched for execution by CTS.

CTS records in the unified log when it has been instructed to run an activity by DAS in a distinctive entry beginning DAS told us to run. If you have a copy of Mints version 1.0b9, you can see these by obtaining a log excerpt for DAS scheduling covering the last ten minutes or so. Once CTS has been told to run an activity, it does so using XPC, a lightweight cross-process communication system, rather than using launchd.

The dialog between DAS and CTS resumes once the activity is complete, which is reported back to CTS by XPC. CTS closes its connection with the completed activity, and schedules the next run with DAS, which adds that to a list of activities.

The bug in Sierra is severe, and renders DAS incapable of dispatching activities reliably. CTS works fine for the first few days of continuous running, then DAS – presumably affected by a memory leak or failure to maintain its activity lists – falls apart. Instead of performing its frequent reassessments of activities and their priority scores, it runs intermittently, and may forget to dispatch a scheduled backup for several hours.

In the next article in this series, I’ll provide you with a utility which can use CTS to schedule your own activities, and combining that with Mints, I’ll walk through how CTS works in detail.

7Comments

Add yours

1

stevetrease (@stevetrease) on October 13, 2020 at 6:37 am

Is it possible to use these mechanisms to schedule an event to run once after each reboot?

LikeLiked by 1 person
- 2
  
  hoakley on October 13, 2020 at 6:47 am
  
  You should be able to by configuring it to run just once and terminate. You could then make it a LaunchDaemon or LaunchAgent, or run it once when CTS starts up. Depending on whether it’s run as a system or user item, that will determine whether it’s run before login or afterwards.
  For LaunchDaemons/LaunchAgents, you’d need to study the Apple documentation carefully to configure its property list.
  Howard.
  
  LikeLike
- 3
  
  yeechang on October 13, 2020 at 8:11 am
  
  In addition to Howard’s answer, crontab can do what you want with @reboot as the time.
  
  LikeLiked by 1 person
4

Grumpy on April 10, 2021 at 6:31 am

Howard, really like your website for all the effort you put into trying to grok the actual implementation details of Apples software — not sure about the painting/art stuff but having a diversity or breadth of interests is always good – I wonder what your demographics are for people who are interested in both?

The documentation for anything you need to know when there is a problem is usually non existent or so badly out of date it’s almost worse than no documentation…and most all this stuff is ever changing of course.

That said would appreciate any deeper dive or holistic insight you could give into all the background processes and conditions for execution/success/failure/progress and indications thereof when it comes to the modern photos.app on big sur.

I have a massive photos.app library that simply won’t finish photoanalysisd and never starts the face recognition phase as best I can tell at cursory examination. A cursory look at the various logs showed progress for photoanalysisd and mediaanalysis full scan seems to always be stopped due to activity when trying to watch the log, or thermals even when obviously not hot and the Mac is completely idle.

Long story slightly shorter I wrote a shell script to parse all my jpegs and found only three damaged files out of 120000 plus of these (truncated images I was able to easily re-air it turned out) and used powerphotos to make a “virgin library”containing all my originals (this eliminated any legacy issues having come through numerous versions of iPhoto, aperture and photos.app over the years) this worked up until it did not — the scanning worked for a few days until it got to about 50% on the curating progress bar in the photos app. So much better than the about 35% it made it too in my original library! Video files pass rudimentary checks out but I haven’t found any great software for attempting to low level verify the data in the various codecs and container formats.

Alas the face scanning never started in the new virgin library either and it would appear that I could never get a fullmediaanalysis to start let alone complete which seems to be a good symptom of where the problem lies…

I intend to come back to this issue and deal with it in the fall when the next version of photos.app and macOS are around — there is quite a complex set of daemons, xpc, and scheduling underlying all of this stuff and Apple has seen fit to not document anything for its users nor have they provided any kind of repair feature that works. (oh yes I have tried photos repair library at startup feature to no avail along with checking permissions – library is on an ignore ownership APFS external SSD volume). How hard would it have been for Apple to actually log error conditions and make note of any bad or corrupt files etc… there are hundreds of “normal end users” having these questions online so what hope do they have when the developers and nerds can’t even figure it out after significant efforts. Reverse engineering levels of effort should not be needed just to get basic functionality to work or to try to gauge detailed task progress in photos.app! Ah software quality in 2021, ever more (“oversimplified”) complexity, ever lesser documentation and QA. If stuff “just worked” then only us nerds would be annoyed that there is intentionally no documentation of any of the implementation and architectural details, however when stuff doesn’t actually work it like trying to use a version of Windows…

Huge data sets like this are hard to find problems in when said problems only become apparent days into background scanning and it’s not practical to repeatedly partition your data into separate libraries until you find problem items if that is indeed the root problem…

LikeLiked by 1 person
- 5
  
  hoakley on April 10, 2021 at 8:39 am
  
  Thank you.
  I’m afraid your solution lies in your last paragraph. As I keep trying to explain to users, Photos and its libraries is primarily designed to store snapshots (in the sense of photos) for regular social and other purposes. If you want a media database, then you need to use a third-party product which is designed to handle such large numbers of images. Photos will work comfortably with extensive collections, but only when you divide them up into different libraries, aiming to have no more than about 10,000 images in each. It’s actually a miracle that your vast library works at all.
  Trying to dig into library and process internals isn’t going to help, as the next time that Apple updates Photos it will all change, as it’s a consumer not pro app.
  Howard.
  
  LikeLike
6

Grumpy on April 12, 2021 at 12:01 am

Thanks for the reply Howard,

Apologize for typos, using ios software keyboard here at the moment.

In general I agree with you that just using smaller libraries would help avoid issues that might be caused by large libraries… the problem with this approach to reality is this workaround is not for me and my data but for a “normal user” and their data — normal/casual users cannot (or just consistently have a whole lot of trouble) manage the extra complexities and side effects of dealing with multiple libraries (and yet only one iCloud photos library) —in fact in this case they are trying to move from n=9 separate libraries with lots of duplications and having to regularly switch libraries and search for stuff and not being able to leverage an iCloud Photo Library. [still definitely not risking using iCloud Photo Library until sometime after I can get stuff working right without it, but it is what they need to use going forward with multiple iOS devices and a Mac]

For this particular use case I can’t “fix the user” and throw or silo away their data because it would be simple to do so – they have a largish size problem but nothing on an industrial scale, it really only gets its complexity by having multiple libraries, not fitting locally on their mobile devices, and no cloud truth for the multiple devices…

Decades of painful experience has reinforced that, For all the functionality in the Apple ecosystem to “work right” across macOS and iOS , tvOS etc you need to stick within Apple’s ever changing implementation of “media (mis)management”. (although using iTunes/podcasts libraries for your AV media management became increasingly untenable over the last few years as there really is no great solution when you want full platform integration and proper control and support for distributed storage locations and full metadata support).

I too used to think that it is stupid to even try to use photos.app with large libraries but really aside from the original data being somewhat large in aggregate it’s 2021 and I figured I would try to reevaluate my logic+conclusions on this topic given we are now several major release versions into photos.app and iCloud photos. Also I am not sure that largish libraries, say 50k+ items are all that uncommon for “normal” users anymore – would be nice to have some decent data on this.

Compute power is now relatively decent as it relates to this task and with fast local modern ssd storage the I/O problem is far less constraining. I really do not see why modern apps with sufficiently sophisticated architectures should not be able to easily handle a quarter million+ media items from a databasing and item manipulation point of view. yes the ML processing is very compute limited on now legacy intel hardware but this functionality is arguably somewhat secondary (as long as it eventually completes in the background within a few weeks say — surely Apple Silicon goes along way to solving this particular bottleneck) and iCloud supports 2TB of storage (yes it should be more in 2021 but this is another area they continue to shall we say over optimize costs vs ARPU in). Furthermore photos.app seems to run quite acceptably and is very usable (such as it is metadata wise) with ~175,000 items using about 700GB for original media.

Do you have any official info on photos app limits or detailed opinion/reasoning as to why it is unreasonable to expect photos.app libraries to handle this kind of use case? I am thinking I have must have some sort of original media data corruption or incompatibility issue and that photos’ app simply is not telling me where it is failing to make progress once it stops making progress during the background ML processing. is it your opinion that the problem I (and apparently many others with much smaller libraries) am seeing is somehow just from the fact I have ~1.5 orders of magnitude more items in the photos library or ?

Apple’s “library” apps have always been shall we say, “suboptimal” once you have more than one library and they have never properly implemented the necessary abstractions for distributed storage (and management thereof) of items except for a brief while — Aperture had an overly complex partial implementation of such. [I am not aware of how they dealt with these issues in the Final Cut/Logic/GarageBand etc realms].

What I always hoped for [yeah not very realistic was it…] was Apple would implement the core library databasing, full metadata support, local/cloud media storage management and provide a truly complete set of API’s and affordances for scripting/automation access to it — this way the platform keeps control of all the key functionality including fine grained access that allows for security and privacy whilst enabling third party developers and sophisticated end users to do all the stuff that Apple never could or would bother to address — All while keeping everything compatible/integrated and consistent to a much greater extent. Having every developer who wants to do anything novel or “pro-ish” in this space have to reinvent the wheel for all of this is just a horrible idea and it pretty much stops indie and small developers from even starting, whilst ensuring that basic platform features don’t work or are unavailable if you do attempt to roll your own solutions. Living in an Adobe ecosystem is not something I want to subject any normal end user too and it still would not ever get you back the Apple ecosystem benefits.

LikeLiked by 1 person
- 7
  
  hoakley on April 12, 2021 at 7:26 am
  
  Thank you.
  You’re correct: very large Photos libraries are becoming increasingly common, as are users complaining of problems with their very large Photos libraries. I wonder if there’s some connection?
  Photos has never been intended as a ‘pro’ app. Apple had a superb ‘pro’ solution, which it killed.
  I don’t think that Lightroom is any better, either. You have to go up to a proper Digital Asset Management system if you want fully-integrated management of more than around 50-100K images – something like Extensis Portfolio.
  Howard.
  
  LikeLike

·Comments are closed.

Share this:

Related