Judging by widespread comments about the Big Sur betas, one major new feature being tested in them is a version of Time Machine which can at last make backups which are stored on an APFS volume. Up to and including Catalina, Time Machine has only supported two backup destinations: local storage in HFS+ format, and networked storage via AFP or SMB. As it is nearly three years since Apple released its new file system in macOS High Sierra, for many users APFS support is high on the wishlist for Big Sur.

From its release in Mac OS X 10.5, Time Machine backups have been dependent on features of the HFS+ file system to create their Finder illusion. Every hour the backup service would examine the record of changes made to the file system since the last backup was made, using its FSEvents database. It worked out what had changed and needed to be copied into the backup. During the backup phase itself, it only copied across those files which had been created or changed since the last backup was made.

It still does this in Catalina by using hard links in the backup, and Apple added a new feature to its HFS+ file system to support this: directory hard links. Where an entire folder has remained unchanged since the last backup, Time Machine simply creates a hard link to the existing folder in that backup. Where an existing file has been changed, though, the new file is written to the backup inside a changed folder, which in turn can contain hard links to unchanged contents.

This preserves the illusion that each backup consists of the complete contents of the source, but only requires the copying of changed files, and creation of a great many hard links to files and folders. It’s also completely dependent on the backup volume using the HFS+ file system, to support those directory hard links.

Without the directory hard link, backups would quickly be overwhelmed by hard links to files. If you had a million files and folders on the backup source volume, every hourly backup would have to create a total of a million copied files or hard links for those which remained unchanged. Directory hard links enable the efficiency needed for this scheme to work.

Apple later introduced what it called Mobile Time Machine, intended for laptops which could be away from their normal backup destination for some time. In around 10,000 lines of code, Mac OS X came to create something like a primitive snapshot, but on HFS+.

When Apple released the first version of APFS on Mac OS X, in High Sierra, its new snapshot feature was incorporated into Time Machine. They were initially used instead of the FSEvents database to determine what should be backed up.

Since then, making each backup of an APFS volume has involved creating a snapshot, but that is stored locally on the APFS volume being backed up. The structure of backups themselves didn’t change: they still required an HFS+ volume and used directory hard links.

Catalina introduced a more complicated scheme to replace snapshots as the normal means for determining what to back up. This was presumably because computing a snapshot delta proved slow, and the introduction of the Volume Group, with specialist types of APFS volume for which snapshot deltas would be inappropriate or impossible.

Determining what gets backed up in Catalina can thus depend on snapshots, and snapshots are made during each backup, but because the backup destination remains in HFS+ format it can’t use snapshots itself, and instead still has to rely on directory hard links.

Apple hasn’t (yet) made any announcement concerning Time Machine in macOS 11, but those testing the early betas report that Time Machine can now create and maintain backups on a destination volume in APFS format. As this is unannounced, it’s perfectly possible that this feature won’t ship in the first full release of Big Sur. However, it does appear to work well.

For Big Sur’s Time Machine to be able to back up to APFS destinations, those backups need a new structure which isn’t reliant on directory hard links, but would have the same effect in terms of creating the Finder illusion, and minimising space required to store what appears to be a complete copy of the volume being backed up.

Several beta testers have reported that Big Sur’s backups maintain their Finder illusion as they are snapshots, which are mounted on demand when the user selects them in the Finder. Although a plausible explanation, there’s clearly a little more to it. Other reports confirm another feature high on the wishlist for Time Machine, that it can back up changed blocks from large files, rather than always having to copy the whole file whenever anything changes within it. These are interrelated: if backups are going to rely on snapshots, then it’s essential to use those file system metadata to minimise the amount of duplicated storage space by backing up blocks whenever possible. However, HFS+ had no support for doing that.

One explanation of how Big Sur might make a backup is to copy the changed files and blocks into a folder, make a snapshot, mount that snapshot, then present the user with just the backup folder within that snapshot. This doesn’t explain how individual blocks can become incorporated: they need to be associated in the snapshot with the previous version of that changed file, something which doesn’t normally occur in a snapshot. There’s an even bigger problem too: a snapshot can’t of itself replace directory hard links. As it stands, the section of a snapshot containing a fresh backup in a folder has no references to the original folders which it needs to replicate.

To understand this, assume that Time Machine has just backed up a Data volume consisting of just one very large file, and an unchanged folder with a large and full directory tree within it. The changed data copied to the backup folder consists of a single changed block from that large file. Before any snapshot can be made of that, the changed block needs to be associated with the other blocks of that large file, which might be in the previous backup. The unchanged folder must also be associated with its last backup, a process which could have been achieved in HFS+ using a directory hard link. For a snapshot containing the new backup folder to maintain the Finder illusion, Time Machine and APFS must therefore do rather more than making a simple snapshot.

In Big Sur, copying items to the backup is more complex than in Catalina. Not only are blocks copied (‘Files Delta Copied’ in the log), but unchanged directories are also ‘move skipped’ before the snapshot is made of the backup. The ‘move skipped’ directory appears to be a synthetic form of directory hard link which propagates unchanged directories into the backup. To understand how Big Sur can show a backup using a snapshot, we need to understand how it handles those two tasks, at the least.

I’m very grateful for comments posted here by Thomas, Sebby and others.