Last Week on my Mac: Stories of gods and Grand Central Dispatch

Stories are arguably the most enduring cultural phenomenon of humans. However they’re told, we are as obsessed by them today as we were at the dawn of recorded history. Every group, society, culture, nation has its own evolving collection of stories – even the nerds among us.

Last week I have been immersed in two anthologies of stories: the ancient Greek and Roman myths expertly told and crafted by the Roman poet Ovid into his Metamorphoses, and a much murkier story of how Time Machine backups are run in macOS Sierra. Before dismissing this as an absurd comparison, there are some valuable comparisons which can be made.

In Ovid’s case, I have a wealth of literature written by some of the brightest minds of the time, eminent scholars who have studied the many manuscripts which have survived, not from the Augustan age when Ovid wrote the fifteen books which make up his epic, but from their descendants during the Middle Ages. The entire source is available in its original Latin, and several English (and other) translations, free of charge, thanks to the efforts of the Perseus Project at Tufts University, and others.

My task is to understand, explain, and then illustrate each of the 250 or so myths, using an array of apps and tools, including Safari, GraphicConverter, BBEdit, Google’s search engine, Wikimedia Commons, and more. At the heart is Storyspace, which I am using to structure words and images into hypertext. Not in HTML, which is too crude and generic a tool for this task, but in bespoke XML within a powerful and sophisticated authoring environment.

macOS has its own myths, and its own strange pantheon of gods. At its heart is the kernel, the first major code to be run after starting up, without which nothing else happens. Following that is launchd, the master controller, which is responsible for starting everything else (apart from the kernel and itself), and which initiates tasks which run at regular time intervals.

We tend to assume that this has always been the way that Mac OS X worked, but launchd didn’t arrive until Mac OS X 10.4 Tiger in the spring of 2005. Prior to that we had a cobbled-together system using tools such as rc, SystemStarter, inetd, cron, and others. launchd was maintained as open source code until OS X 10.10 Yosemite, but most of it appears now to have moved to libxpc, which is proprietary and little-documented.

The myth persists that key parts of macOS, such as Time Machine’s periodic backups, continue to be controlled by launchd, something which I for one had assumed was still true. But looking at the evidence suggests something quite different: set anything else to be run every hour using launchd, and it is launched within a second of the scheduled time. Time Machine’s backups normally occur within five or ten minutes of expectation, often drift steadily later, and may become completely irregular.

Documentation seems still in the Middle Ages, with fragments of uncertain relevance to Sierra, and no coherent account of any of this. The only way to work out what is going on is to examine the logs, a task which now seems to be reserved for those of priestly caste, or Apple’s own oracles.

When you do finally manage to see what is being recorded in those logs, most of the more valuable information has been removed. Entries such as
(DuetActivityScheduler) [com.apple.duetactivityscheduler.client] STARTING: <private>
are as meaningful as the wailing of a Bacchante. And the great god launchd remains silent throughout those logs, never uttering a single entry.

What is becoming clear, though, is a different story from the traditional myth of launchd. Instead of – or, more accurately, as well as – launchd there are new daemons at work. Components known as CTS (Centralised Task Scheduling), Duet (as in Duet Activity Scheduler, DAS) which used to be mainly associated with Handoff features, and XPC (lightweight interprocess communication), which are part of Grand Central Dispatch, GCD.

What looks to be happening when it is about time to run a Time Machine backup is that the DAS daemon maintains a list of activities and their ‘scores’ (Quality of Service, or QoS, measures), which determine the probability of their being run. Periodically, DAS rescores the activities, and may then decide to instruct CTS to run the backupd-auto periodic task, which in turn kicks backupd into action to perform the backup.

The story which emerges is as complex as any of Ovid’s elaborately interwoven myths. It explains why a simple, quick test task handed to launchd is run at almost exactly the right time, every hour, but a more complex and drawn-out task like making a backup may have to be held for the right moment, and runs much less regularly.

Most importantly, understanding Apple’s metamorphosed task scheduling system, with GCD, CTS, DAS, and XPC, is the key to explaining how Time Machine can start behaving erratically when launchd still seems to be ticking away accurately. To know the right answer, you have to be in the right myth, and to have the right tools.