hoakley Macs, Technology

Time Machine has changed again in Catalina

Before Apple introduced its new file system APFS, Time Machine worked out what to back up using the hidden database of file system events, FSEvents. In High Sierra, this changed so that, when backing up APFS volumes, Time Machine made a snapshot of that volume, compared it with the previous snapshot made for the last backup, and worked out what had changed, thus what needed to be backed up. This new system continued through Mojave, but appears to have changed yet again in Catalina. This article examines those changes.

One obvious change in Catalina’s Time Machine is that it’s no longer backing up a single volume such as Macintosh HD, but has both the boot System and Data volumes to include, unless you opt for backups to exclude the read-only System volume. For good measure, Catalina also makes and maintains backups of the Recovery volume, including boot.efi, the Base System, diagnostics, and its other components.

Although the structure of the Backups.backupdb folder remains essentially the same as in Mojave, inside each backup folder you will find event-based data in files with names of .[UUID].eventdb as well as those for clones named .[UUID].clonedb. A typical lower size for the first full backup is around 56 GB, which includes one folder for each of the volumes Macintosh HD (System volume, if included in backups), Macintosh HD – Data (Data volume) and Recovery.

The sequence of events in a normal successful backup, excluding housekeeping of old backups and snapshots which is performed at the end, then runs:

  1. Starting automatic backup.
  2. Look for sparsebundle of backups on the backup volume. This can be identified by host UUID or MAC address.
  3. Mount the Recovery volume.
  4. Create local snapshots on the root disk / and the Data volume at /System/Volumes/Data, and declare these as stable.
  5. Mount latest (‘stable’) and previous snapshots (‘reference’).
  6. For each volume in turn, enumerate possible strategies for determining the files to back up, then determine those and save in a cache database.
  7. For each volume in turn, using the cache database, determine the total size of files to be backed up.
  8. Total these to estimate the size required for all the backups.
  9. For each volume in turn, copy the files to be backed up onto the backup volume.
  10. Unmount the snapshots.
  11. Mark the last snapshots as ‘reference’ for the next backup.
  12. Confirm the path to the backup volume is still correct.
  13. Backup completed successfully.

Strategies available for determining the files to back up include:

  • first backup – all files and folders, other than those in the exclude list, are copied;
  • deep scan – a scan is performed of the volume looking at modification dates, which can be very slow on large volumes, but is simplest on small volumes like Recovery;
  • FSEvents – the FSEvents database for that volume is used, as is the default for HFS+ volumes;
  • snapshot diffing – two snapshots are compared to determine the differences, as was the default for APFS volumes in High Sierra and Mojave.

Currently, there appears to be one significant omission in the log entries from backupd: it doesn’t check and return the amount of free space on the backup volume, something which has been logged consistently by backupd in Mojave and earlier. An annoying side effect of this is that my free tool T2M2 no longer reports free space on the backup volume, something which I need to rectify.

Summary of changes

  • Backups now include three volumes: the read-only System volume, the Data volume, and the Recovery volume.
  • Strategies for determining what needs to be backed up now include both FSEvents and snapshot differences for APFS volumes.
  • backupd doesn’t log the free space available on the backup volume.
  • Backups made by macOS 10.15 are no longer compatible with 10.14.6 and earlier.

Log extracts

Start of backup sequence:
10:56.951016 backupd TimeMachine Starting automatic backup

Look for sparsebundle to back up to:
10:56.953108 backupd TimeMachine Looking for sparsebundle for host UUIDs '(
"F579E47E-F6C9-5D32-BE1E-66E17E83B339"
)' or MAC addresses '(
88e9fe4e8bca
)' in directory '/Volumes/ExternalSSD2'

Mount the Recovery volume:
10:57.067409 backupd TimeMachine Mounted recovery volume at /Volumes/Recovery

Announce backuo destination:
10:57.067435 backupd TimeMachine Backing up to ExternalSSD2 (/dev/disk2s2): /Volumes/ExternalSSD2

Create local ‘stable’ snapshots:
10:58.961719 backupd TimeMachine Created Time Machine local snapshot with name 'com.apple.TimeMachine.2019-11-11-141058.local' on disk '/'
10:58.968146 backupd TimeMachine Created Time Machine local snapshot with name 'com.apple.TimeMachine.2019-11-11-141058.local' on disk '/System/Volumes/Data'
10:58.969220 backupd TimeMachine Declared stable snapshot: com.apple.TimeMachine.2019-11-11-141058.local

Mount previous ‘reference’ and latest ‘stable’ snapshots:
10:58.978608 backupd TimeMachine Mounted stable snapshot: com.apple.TimeMachine.2019-11-11-141058.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-141058/Macintosh HD - Data source: Macintosh HD - Data
10:58.989240 backupd TimeMachine Mounted stable snapshot: com.apple.TimeMachine.2019-11-11-141058.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-141058/Macintosh HD source: Macintosh HD
10:58.996564 backupd TimeMachine Mounted reference snapshot: com.apple.TimeMachine.2019-11-11-131153.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-131153/Macintosh HD - Data source: Macintosh HD - Data
10:59.006041 backupd TimeMachine Mounted reference snapshot: com.apple.TimeMachine.2019-11-11-131153.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-131153/Macintosh HD source: Macintosh HD

For the Recovery volume, determine which method to use to determine what will be backed up, and generate the event database:
10:59.105896 backupd TimeMachine Possible strategies for "Recovery" (device: /dev/disk1s3 mount: '/Volumes/Recovery' fsUUID: 9E45515F-B169-4338-831B-CCE4B5657CC9 eventDBUUID: (null)) (<EventMarker 2019-11-11 13:11:56 +0000 0 (null) 0>): first backup, deep scan,
10:59.106547 backupd TimeMachine Deep scan required for source: "Recovery" (device: /dev/disk1s3 mount: '/Volumes/Recovery' fsUUID: 9E45515F-B169-4338-831B-CCE4B5657CC9 eventDBUUID: (null))
10:59.107539 backupd TimeMachine Running deep scan - looking for changes after 2019-11-11 13:10:56 +0000
10:59.125428 backupd TimeMachine Trusting source modification times for complete event history.
10:59.127671 backupd TimeMachine Found 0 perfect clone families, 0 partial clone families. Zero KB physical space used by clone files. Zero KB shared space.
10:59.128534 backupd TimeMachine Saved event cache at /Volumes/ExternalSSD2/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-141059.inProgress/75510780-9057-4097-A56E-EB646413C690/.9E45515F-B169-4338-831B-CCE4B5657CC9.eventdb

Do the same for the Data volume:
10:59.130764 backupd TimeMachine Possible strategies for "Macintosh HD - Data" (device: /dev/disk1s1 mount: '/System/Volumes/Data' fsUUID: A86D5F30-2372-44D6-BA86-E92B079DDF56 eventDBUUID: 8B03C842-BB4D-4484-B81C-297A3F6AB325) (<EventMarker 2019-11-11 13:12:05 +0000 7162257443518969687 8B03C842-BB4D-4484-B81C-297A3F6AB325 245746>): first backup, deep scan, FSEvents, snapshot diffing,
10:59.131520 backupd TimeMachine Using FSEvents for source: "Macintosh HD - Data" (device: /dev/disk1s1 mount: '/System/Volumes/Data' fsUUID: A86D5F30-2372-44D6-BA86-E92B079DDF56 eventDBUUID: 8B03C842-BB4D-4484-B81C-297A3F6AB325)
10:59.133036 backupd TimeMachine Read 342 clone families from /Volumes/ExternalSSD2/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-131207/.A86D5F30-2372-44D6-BA86-E92B079DDF56.clonedb
10:59.142412 backupd TimeMachine Buffering events...
10:59.153898 backupd TimeMachine Buffered 680 events.
10:59.155953 backupd TimeMachine Collected 1 events. Last path seen: 'private/var/log/system.log'
10:59.249625 backupd TimeMachine Trusting source modification times for complete event history.
10:59.249658 backupd TimeMachine Found 0 perfect clone families, 0 partial clone families. Zero KB physical space used by clone files. Zero KB shared space.
10:59.252695 backupd TimeMachine Saved event cache at /Volumes/ExternalSSD2/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-141059.inProgress/75510780-9057-4097-A56E-EB646413C690/.A86D5F30-2372-44D6-BA86-E92B079DDF56.eventdb

Do the same for the System volume:
10:59.256351 backupd TimeMachine Possible strategies for "Macintosh HD" (device: /dev/disk1s5 mount: '/' fsUUID: D9350CE9-7741-4BEA-B0D0-0A8A7420443D eventDBUUID: 0027C297-BF1E-440C-AB89-E8471634F57A) (<EventMarker 2019-11-11 13:12:06 +0000 7162257443518969687 0027C297-BF1E-440C-AB89-E8471634F57A 245745>): first backup, deep scan, FSEvents, snapshot diffing,
10:59.256959 backupd TimeMachine Using FSEvents for source: "Macintosh HD" (device: /dev/disk1s5 mount: '/' fsUUID: D9350CE9-7741-4BEA-B0D0-0A8A7420443D eventDBUUID: 0027C297-BF1E-440C-AB89-E8471634F57A)
10:59.257811 backupd TimeMachine Read 0 clone families from /Volumes/ExternalSSD2/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-131207/.D9350CE9-7741-4BEA-B0D0-0A8A7420443D.clonedb
10:59.260950 backupd TimeMachine Buffering events...
10:59.263519 backupd TimeMachine Buffered 0 events.
10:59.276853 backupd TimeMachine Trusting source modification times for complete event history.
10:59.276892 backupd TimeMachine Found 0 perfect clone families, 0 partial clone families. Zero KB physical space used by clone files. Zero KB shared space.
10:59.277670 backupd TimeMachine Saved event cache at /Volumes/ExternalSSD2/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-141059.inProgress/75510780-9057-4097-A56E-EB646413C690/.D9350CE9-7741-4BEA-B0D0-0A8A7420443D.eventdb

Calculate space required to back up the Recovery volume:
10:59.280027 backupd TimeMachine Estimating size of changes for "Recovery" (device: /dev/disk1s3 mount: '/Volumes/Recovery' fsUUID: 9E45515F-B169-4338-831B-CCE4B5657CC9 eventDBUUID: (null))
10:59.282800 backupd TimeMachine (fsk:0,dsk:1,fsz:0,dsz:0)(1/0)
10:59.283917 backupd TimeMachine Sizing by file event iteration for source Recovery completed. Changed content estimate: (bytes: 0 (Zero KB), items: 2)
10:59.285932 backupd TimeMachine Estimated 2 files (Zero KB) need to be backed up from 'Recovery'

Repeat for the Data volume:
10:59.288410 backupd TimeMachine Estimating size of changes for "Macintosh HD - Data" (device: /dev/disk1s1 mount: '/System/Volumes/Data' fsUUID: A86D5F30-2372-44D6-BA86-E92B079DDF56 eventDBUUID: 8B03C842-BB4D-4484-B81C-297A3F6AB325)
10:59.293063 backupd TimeMachine (fsk:0,dsk:1,fsz:0,dsz:0)(3/106)
11:00.282070 backupd TimeMachine Sizing by file event iteration for source Macintosh HD - Data completed. Changed content estimate: (bytes: 31586906 (31.6 MB), items: 208)
11:00.284420 backupd TimeMachine Estimated 208 files (31.6 MB) need to be backed up from 'Macintosh HD - Data'

Repeat for the System volume:
11:00.286609 backupd TimeMachine Estimating size of changes for "Macintosh HD" (device: /dev/disk1s5 mount: '/' fsUUID: D9350CE9-7741-4BEA-B0D0-0A8A7420443D eventDBUUID: 0027C297-BF1E-440C-AB89-E8471634F57A)
11:00.288815 backupd TimeMachine Sizing by file event iteration for source Macintosh HD completed. Changed content estimate: (bytes: 0 (Zero KB), items: 1)
11:00.291145 backupd TimeMachine Estimated 1 files (Zero KB) need to be backed up from 'Macintosh HD'

Arrive at a grand total number of files and their size for this backup:
11:00.292344 backupd TimeMachine Estimated 211 files (31.6 MB) need to be backed up from all sources
Note that in Catalina, this isn’t seen to be compared with free space available, as it was in Mojave and earlier.

Copy the files to be backed up:
11:00.407950 backupd TimeMachine Copied 1 items (Zero KB) from volume Recovery. Linked 1. Moved 0
11:00.408435 backupd TimeMachine Found 0 perfect clone families, 0 partial clone families. Shared space Zero KB (0 bytes).
11:06.654761 backupd TimeMachine Copied 285 items (31.5 MB) from volume Macintosh HD - Data. Linked 2310. Moved 0
11:06.655632 backupd TimeMachine Found 0 perfect clone families, 0 partial clone families. Shared space Zero KB (0 bytes).
11:06.657546 backupd TimeMachine Saved clone family cache for 'Macintosh HD - Data' at /Volumes/ExternalSSD2/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-141059.inProgress/75510780-9057-4097-A56E-EB646413C690/.A86D5F30-2372-44D6-BA86-E92B079DDF56.clonedb
11:06.949677 backupd TimeMachine Copied 5 items (69 bytes) from volume Macintosh HD. Linked 7. Moved 0
11:06.950550 backupd TimeMachine Found 0 perfect clone families, 0 partial clone families. Shared space Zero KB (0 bytes).

Unmount all the local snapshots:
11:07.324930 backupd TimeMachine Unmounted local snapshot: com.apple.TimeMachine.2019-11-11-141058.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-141058/Recovery source: Recovery
11:07.617359 backupd TimeMachine Unmounted local snapshot: com.apple.TimeMachine.2019-11-11-141058.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-141058/Macintosh HD - Data source: Macintosh HD - Data
11:07.917901 backupd TimeMachine Unmounted local snapshot: com.apple.TimeMachine.2019-11-11-141058.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-141058/Macintosh HD source: Macintosh HD
11:07.919147 backupd TimeMachine Unmounted local snapshot: com.apple.TimeMachine.2019-11-11-131153.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-131153/Recovery source: Recovery
11:08.217592 backupd TimeMachine Unmounted local snapshot: com.apple.TimeMachine.2019-11-11-131153.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-131153/Macintosh HD - Data source: Macintosh HD - Data
11:08.517588 backupd TimeMachine Unmounted local snapshot: com.apple.TimeMachine.2019-11-11-131153.local at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s MacBook Pro/2019-11-11-131153/Macintosh HD source: Macintosh HD

Mark the last snapshots as reference for the next backup:
11:09.740827 backupd TimeMachine Marked as reference snapshot: com.apple.TimeMachine.2019-11-11-141058.local

Backup completed:
11:09.741046 backupd TimeMachine Completed backup: 2019-11-11-141108
11:09.741366 backupd TimeMachine Mountpoint '/Volumes/ExternalSSD2'is still valid
11:10.211622 backupd TimeMachine Mountpoint '/Volumes/ExternalSSD2'is still valid
11:10.516239 backupd TimeMachine Backup completed successfully.