Last week, once I had upgraded my iMac with its large Fusion Drive to APFS in Mojave, I stumbled across the major change which had apparently taken place in Time Machine backups in High Sierra. As I have been unable to find any clear account as to what has changed, this article compares making a backup from an HFS+ disk in Sierra, with the same event from an APFS disk in Mojave.
In the absence of any documentation from Apple, I presume that these mechanisms are file-system-specific, so an HFS+ volume in Mojave should still work much as it did in Sierra. This is because HFS+ doesn’t offer the same local snapshot feature that APFS does.
When Time Machine was first included in macOS, backups were scheduled using a fixed-time scheduled LaunchDaemon, I believe. Apple later introduced a more complex but flexible scheduling scheme using two systems, DAS and CTS, which attempts to run backups at times when they will have less impact on the whole system. There was a bug in Sierra’s implementation of DAS which seems to have been fixed only recently as part of a security update, which periodically led to this mechanism falling apart.
This means that backups are seldom started exactly 60 minutes after the last, but should be called off by DAS after an interval of around 55-65 minutes. It is CTS which actually kicks them off, and the
backupd service which goes into action.
Its first task is to determine the destination backup folder, then to work out what needs to be backed up on that occasion. For HFS+ volumes, it examines the hidden FSEvents records stored at the root of each volume, which record the changes made to its files and folders. If it can’t find a ‘proper’ FSEvents record, then it performs a deep traversal of the volume to construct one. Deep traversals can take a long time, and in many cases may take over an hour, leading to cancellation of the next scheduled backup.
backupd knows what needs to be backed up, it calculates the disk capacity required. This has to allow for overhead or ‘padding’. This is compared with the free space on the backup destination: if there is room, the backup proceeds, otherwise this throws an error.
backupd then copies changed items to the backup destination. In order to maintain the impression that each backup is a complete copy of the source volume, it then makes hard links to all the unchanged files and folders. It is able to do this as, unlike many file systems, HFS+ supports directory hard links as well as those to files.
With the backup itself complete,
backupd turns to maintenance tasks, and identifies which old backups have expired, according to its rules. Those are then deleted in a process which can take longer than making the backup itself. Once that task is done,
backupd reports that the backup is complete, which is signalled back to CTS and DAS so that the next automatic backup can be scheduled by DAS.
These are summarised in the diagram below.
backupd is scheduled using the same mechanism, which now isn’t as prone to failure as it was in the past. This starts in the normal way, by determining the destination for the backup. Having recognised that it has an APFS volume to back up to that, it then follows a different sequence.
The preparatory sequence identifies and deletes expired local snapshots. According to Apple’s Support Note, these local snapshots are kept for 24 hours; although the log entries below indicate a shorter period, later backups confirm that this is normally the case, and you should expect to find a full 24 hours of snapshots at any time.
Once that maintenance is complete,
backupd makes a fresh snapshot, which is saved to the backup folder. This is set as the ‘stable’ snapshot, and mounted ready for access. The last snapshot, made an hour earlier during the Time Machine backup, is then mounted too as the ‘reference’, and
backupd then determines which files and folders should be backed up by comparing those two snapshots, as stored in their backup folders.
Although the two Macs making these backups are not exactly the same, they should be fairly comparable in performance: identifying 239 files for backup on a large Fusion Drive in HFS+ (FSEvents) took about 1.6 seconds; identifying 1447 files for backup on a large Fusion Drive in APFS (snapshots) took about 23 seconds, suggesting that the new procedure is not being used for speed.
backupd then checks that there is sufficient free space on the backup destination, and if there is, performs that same process as with HFS+, copying changed items and making hard links to the rest. That is followed by new steps, which save a clone family cache to the new backup folder, and back-up-later caches there too. The precise purpose of these isn’t yet clear, although the latter may well list files which changed as the backup was being made.
The two snapshots are then unmounted, their job complete, and the latest snapshot is marked as the next to be used as the reference snapshot when
backupd is next run.
A second local snapshot is then made, which shows the state of the file system on completion of the backup. This appears to be saved to the local snapshot store and not copied to the backup, and is intended for use when the full Time Machine backup isn’t available. It is accessible in the Time Machine app: for the previous 24 hours, for an APFS volume, you should find two backups made every hour instead of one. The first represents that in the backup destination, and the second (a few minutes later) is the local snapshot.
backupd finally performs the same maintenance routines on the backup folder, to remove old expired backups, on completion of which it reports that the backup is complete.
The equivalent diagram for APFS volumes is shown below.
How much space do these snapshots take?
Although some users consider that 24 hours worth of snapshots consumes a significant amount of disk space, Apple claims that snapshots are kept as available storage. In other words, when there is pressure for free space, macOS should automatically delete the oldest, as required. For Time Machine backups to function properly, only the latest two snapshots are needed, and are those stored in the backup anyway.
If you want to force older snapshots to be removed from the startup volume, Apple recommends turning Time Machine off for a few minutes to allow the old snapshots to be removed. When turned back on,
backupd should only need to access the snapshot stored in the last backup in order to determine what needs to be backed up.
I have been writing that Time Machine has fallen behind macOS, at least in respect of its reliance on the HFS+ file system for backups, which results from its use of directory hard links. This implementation of Time Machine for APFS is perhaps best viewed as version 1.5: it now takes best advantage of the new file system as its source, but has yet to find a new backup method and format appropriate to an APFS backup destination.
My best guess is that Time Machine 2.0, under development at the moment, will base its backups on the snapshots currently stored in each backup folder. Whether it will be ready for release in macOS 10.15 is unclear, but we should know more at WWDC 2019.
Example log extracts
All entries are written by
El Capitan/Sierra with HFS+:
47:17.053 com.apple.backupd: Starting automatic backup
47:17.268 Backing up to /dev/disk3s2: /Volumes/PROMISE PEGASUS/Backups.backupdb
47:18.834 Will copy (304.1 MB) from Macintosh HD
47:18.834 Found 239 files (304.1 MB) needing backup
47:18.863 14.63 GB required (including padding), 3.23 TB available
51:01.273 Copied 360 items (304 MB) from volume Macintosh HD. Linked 9332.
51:08.237 Created new backup: 2016-07-07-075106
51:10.048 Starting post-backup thinning
54:15.363 Deleted /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2016-07-06-065322 (27.1 MB)
54:15.363 Post-backup thinning complete: 1 expired backups removed
54:19.280 cBackup completed successfully.
Deep Traversal with HFS+:
09:44.382003 Forcing deep traversal on source: "Macintosh HD" (device: /dev/disk2 mount: '/' fsUUID: BE50387F-1302-31FB-B567-68F50F8ADE7A eventDBUUID: FDAC9A01-ECB1-4AFA-8F58-B018F81CF569)
09:44.398792 Deep event scan at path:/ reason:must scan subdirs|require scan|
09:44.398917 Running event scan
09:44.399022 periodic consistency scan for '/'
57:54.162626 Finished scan
57:54.244922 Saved event cache at /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2017-02-18-080944.inProgress/6C87A4AF-8E4C-41C5-B8FE-62BCA79A27F4/.BE50387F-1302-31FB-B567-68F50F8ADE7A.eventdb
Mojave with APFS:
37:13.324510 Starting automatic backup
37:13.944141 Backing up to /dev/disk3s2: /Volumes/PROMISE PEGASUS/Backups.backupdb
37:15.303692 Starting age based thinning of Time Machine local snapshots on disk '/'
37:15.306357 Age based thinning deleted Time Machine snapshot 'com.apple.TimeMachine.2018-10-12-074443' on disk '/
37:20.814376 Created Time Machine local snapshot with name 'com.apple.TimeMachine.2018-10-12-093715' on disk '/'
37:20.821185 Declared stable snapshot: com.apple.TimeMachine.2018-10-12-093715
37:21.222744 Mounted stable snapshot: com.apple.TimeMachine.2018-10-12-093715
at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s iMac/2018-10-12-093715/Macintosh HD source: Macintosh HD
37:21.823102 Mounted reference snapshot: com.apple.TimeMachine.2018-10-12-083723
at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s iMac/2018-10-12-083723/Macintosh HD source: Macintosh HD
37:44.916779 Will copy (637.5 MB) from Macintosh HD
37:44.917419 Found 1447 files (637.5 MB) needing backup
37:44.953124 7.97 GB required (including padding), 4.49 TB available
41:12.017287 Copied 1593 items (623.3 MB) from volume Macintosh HD. Linked 19601.
41:12.037146 Saved clone family cache for 'Macintosh HD' at /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2018-10-12-093721.inProgress/14F6D6F4-4BCA-4EA1-B4F5-67A5B9623254/.BE50387F-1302-31FB-B567-68F50F8ADE7A.clonedb
41:12.688615 Saved back-up-later cache at /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2018-10-12-093721.inProgress/14F6D6F4-4BCA-4EA1-B4F5-67A5B9623254/.reservations.plist
41:14.014047 Saved back-up-later cache at /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2018-10-12-093721.inProgress/14F6D6F4-4BCA-4EA1-B4F5-67A5B9623254/.reservations.plist
41:14.259024 Unmounted local snapshot: com.apple.TimeMachine.2018-10-12-093715 at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s iMac/2018-10-12-093715/Macintosh HD source: Macintosh HD
41:14.507748 Unmounted local snapshot: com.apple.TimeMachine.2018-10-12-083723 at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s iMac/2018-10-12-083723/Macintosh HD source: Macintosh HD
41:15.039030 Marked as reference snapshot: com.apple.TimeMachine.2018-10-12-093715
41:15.161260 Completed snapshot: 2018-10-12-094114
41:16.127934 Starting post-backup thinning
42:01.489936 Deleted /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2018-10-11-090416 (12.2 MB)
42:01.489947 Post-backup thinning complete: 1 expired backups removed
42:01.580128 Backup completed successfully.