Why nothing else can back up to APFS like Time Machine does

What Time Machine used to do, before Big Sur, wasn’t that remarkable. Every hour it would check to see which files had changed since the last time it backed up. Then it would copy across all those files to the backup storage, together with directory and file hard links to create the illusion that the backup was a complete replica of the source.

When Time Machine in Big Sur backs up to an APFS volume, what it does is totally different. It has to be, for one thing because the directory hard links which it relied on making on HFS+ disks aren’t available in APFS. So it adopts a completely different model: first, it looks at changed files in more detail, so that it only copies across as much it needs, and then it creates a snapshot, complete with all the file data required, on the backup storage.

There are several clever twists in what it does that we still don’t understand. One good example of this is how is copes with APFS clone files.

Clone-copying a file in APFS is simple to do in the Finder: just select a file, and either duplicate it, or Option-drag a copy to another folder or location on that same volume (and the clone has to be on the same volume as the original). If you’d like to do that with a huge file of 10 GB or so, and clone it a few times before Time Machine makes its hourly backup to an APFS volume, you’ll see that it doesn’t copy across those clones, but identifies them and only makes a single copy in your backup. You can repeat that by making small changes in those large files: only the changed blocks of the file will be copied in the next backup.

That might not seem a big deal, but in reality a lot of files on many APFS volumes are now either partial or full clones. A quick check on one of my working folders, just a part of my extended Home folder, found almost 1600 clone files totalling over 10 GB in size. If I were still backing up to HFS+, every one of those files would have to be saved in full into my backup, and in every subsequent backup it would have to be present either in the form of a complete copy (if it had changed), a hard link, or one to a directory above it. Instead, my APFS backup just contains a directory entry for the clone.

The effect on backup health is remarkable. After a few months backing up to it, an HFS+ backup store often had millions of hard links, and its file system started to choke and grow errors, requiring an appointment with a tool like DiskWarrior to bring it back up to scratch again. No such maintenance or repair is needed with these new APFS backups, though.

At present, we have almost no insight into APFS cloning. We’ve only just been able to detect it, and working out which files on a volume are clones, and how much of each is still cloned, remains beyond us. Yet Time Machine uses that to reduce the quantity of data which it has to copy in each backup, so optimising time and space used.

Creating the snapshot to form each backup requires even more sleight of hand. What happens when you, or a third-party tool like Carbon Copy Cloner, make a snapshot is fairly easy to understand. For an instant, the file system of that volume is frozen, and a copy is made of it to form the snapshot itself. That’s stored not in the volume, but in its container. As the contents of that volume change, instead of the old blocks being erased and made free for re-use, they’re kept, so they still hook up to the snapshot, which thus retains all its original data, until it’s deleted and those blocks are freed again.

Time Machine not only makes local snapshots of the volume it’s backing up, and stores them in the volume’s container for up to 24 hours, but it also uses a snapshot to create the backup, on a completely different disk. So it must copy across the frozen file system metadata and assemble the requisite file data around it. As far as I know, Time Machine is the only software which can do anything like this, and in theory at least should be capable of copying those backup snapshots to a different disk, allowing the user to copy their backups, for example.

Such are the secret tricks used by Time Machine. Sadly, Apple hasn’t seen fit to reveal any of its magic spells in the APFS documentation, which was last revised over a year ago, and lists many of the ingredients, but stops short of detailing any useful spells. To think that, in its early days, Apple said it might even consider making APFS open source at some time in the future.

If you’re interested in further details of how Time Machine backs up to APFS, the following series should help:
Time Machine to APFS: Evolution
Time Machine to APFS: Initiating an auto backup
Time Machine to APFS: Understanding backups
Time Machine to APFS: Backing up
Time Machine to APFS: How processes have changed
Time Machine to APFS: Backup structure and access
Time Machine to APFS: Maintenance and repair
Time Machine to APFS: Changing disks
Time Machine to APFS: How efficient are backups?
Time Machine to APFS: First full backup
Time Machine to APFS: Using a network share
Time Machine to APFS: When a network backup goes wrong