It’s nearly eleven years since Apple last shipped a Mac with a single processor core, and all its current iOS, watchOS and tvOS devices now have two or even four processor cores.

We now take for granted the fact that more cores means better performance, but before Mac OS X 10.6 (August 2009) that often wasn’t the case. Since then, macOS and Apple’s other operating systems have been quietly transformed so that they make best use of multiple cores, work far better on mobile devices, and are responsive to changes in location, power supply, even the quality of local network conditions.

Take a look in /System/Library, and there are whole folders full of guff which never used to be there: DuetActivityScheduler, DuetKnowledgeBase, and UserEventPlugins, for example. In Activity Monitor, there are processes which have similarly cryptic names like coreduetd and DuetHeuristic-BM. Browse your logs and you’ll see those and other processes in action, with messages about Duet Activity Scheduler (DAS) and Core Task Scheduler (CTS). Look in books about macOS from an advanced user’s or developer’s view, and you’ll be none the wiser. At present, they’re essentially undocumented – almost complete blanks in our understanding.

Most of the better books and sites go into detail about the kernel, its extensions, I/O, file systems, networking, and loads more. But amazingly they don’t actually explain how macOS, iOS, etc., actually work. Not now, as it has evolved over the last few years.

One of the problems with the old Mac OS X model and multiple processors/cores was that getting the most out of those cores required careful design of apps to use threads/tasks/processes. Done well, it worked excellently. But developers had to work quite hard to do well, and many didn’t. You could watch heavyweight apps hogging a single core, and the other cores almost sat idly.

You could feel that in the user interface too: start some rendering or similarly demanding task, and everything else started to grind to a halt. When Time Machine first came out in 2007, things were improving, but it was still wise to avoid trying to do too much when a backup was in progress.

What has changed over those years has been the way in which macOS (and iOS, etc.) handle tasks and cores – a mechanism named by Apple Grand Central Dispatch (GCD), which reaches new heights in macOS Sierra. In many respects, Sierra doesn’t seem that different from El Capitan, or Yosemite before. Look at what GCD does, though, and it is substantially changed.

At its heart, GCD is just that: a dispatcher. It manages queues of tasks, activating those which need most to be run, and leaving the less pressing to wait a bit longer. It has its own queues, and queues which are assembled by apps. Some are run as straight queues, first in first out, others using sophisticated heuristics to determine relative priorities.

Time Machine backup is a most obvious task which is managed by GCD: it occurs at roughly hourly intervals, but its precise timing is determined by all sorts of other factors. If your Mac is very busy doing other things when the hour comes round, the backup may be deferred by five or ten minutes. If everything else is idling, then the backup will start close to the expected time.

Even on macOS, factors which are taken into account by GCD’s heuristics extend well beyond its current capabilities. Look in some of the CoreDuet configuration files in /System/Library/DuetKnowledgeBase, for example, and you might think that you were dealing with an iPhone with its voice communications features, or another iOS device which is sensitive to precise location change, or its orientation. These demonstrate how GCD is common across Apple’s products, from desktop Macs to the Apple Watch.

So, fascinating though this might be, what does this have to do with any Mac user?

More and more apps now use GCD’s dispatch queues to handle tasks within the app, and more and more services are run not by the old launchd scheduling system, but are managed by GCD’s heuristics. Great though GCD is much of the time, it has its own issues: until Sierra 10.12.4, it contained a serious bug which eventually led it to cease working properly, for example.

When simple things go wrong, we know how to fix their preference settings, how to remove badly-behaved kernel extensions and tweak LaunchAgent settings, and can often work around issues which might otherwise stop us from doing what we need to. When things go wrong with GCD, few of us even know how to diagnose the problem, and there are almost no tools available to help us.

One of the well-known problems in multi-tasking and concurrency is deadlock, when one task sits waiting for another, and the other task cannot proceed until the first task is complete. GCD is not immune from deadlock, and there are some curious issues which have arisen in El Capitan and Sierra which look suspiciously as if deadlock may have been their root cause.

I am also beginning to wonder if some of the persistent problems which we have experienced with Bluetooth disconnects, in both El Capitan and Sierra, are the result of issues within GCD, rather than in the Bluetooth drivers themselves. Even now, with Sierra 10.12.4, seldom a day passes without my Magic Trackpad 2 spontaneously disconnecting, then reconnecting.

These are all made the more complex by GCD’s heuristics, its ‘smart’ dispatching system. When trying to identify and diagnose problems, predictability is one of the most important properties, but because of those complex heuristics nothing about GCD follows clear and simple rules.

Don’t get me wrong: I think that these changes in macOS (iOS, etc. too) are some of the most important that Apple has made. I see fewer and fewer spinning beachballs, and GCD has enabled Apple to have a strong campaign both internally and with third-party developers to eliminate them. The days of waiting for tasks to complete before you can get on with your work are largely past: Macs remain responsive even when put under heavy load. But fixing problems without documentation or tools is fiendishly difficult.

Note: as you’ll see from the long series of comments below, in this article, and elsewhere, I use the name Grand Central Dispatch to describe a group of technologies and services which is far wider than Apple’s developer documentation from five years ago does. As Apple has still not published a reference to Grand Central Dispatch, it is impossible to know whether my usage corresponds with Apple’s, or whether subsystems like DAS and CTS are intended to be part of another system. GCD is here a convenient blanket term for ‘old GCD’, CTS, DAS, and a few other related bits too. If you have a problem with that, please suggest a better name.