hoakley October 17, 2018 Macs, Technology

How Time Machine makes backups

Last week, once I had upgraded my iMac with its large Fusion Drive to APFS in Mojave, I stumbled across the major change which had apparently taken place in Time Machine backups in High Sierra. As I have been unable to find any clear account as to what has changed, this article compares making a backup from an HFS+ disk in Sierra, with the same event from an APFS disk in Mojave.

In the absence of any documentation from Apple, I presume that these mechanisms are file-system-specific, so an HFS+ volume in Mojave should still work much as it did in Sierra. This is because HFS+ doesn’t offer the same local snapshot feature that APFS does.

When Time Machine was first included in macOS, backups were scheduled using a fixed-time scheduled LaunchDaemon, I believe. Apple later introduced a more complex but flexible scheduling scheme using two systems, DAS and CTS, which attempts to run backups at times when they will have less impact on the whole system. There was a bug in Sierra’s implementation of DAS which seems to have been fixed only recently as part of a security update, which periodically led to this mechanism falling apart.

This means that backups are seldom started exactly 60 minutes after the last, but should be called off by DAS after an interval of around 55-65 minutes. It is CTS which actually kicks them off, and the backupd service which goes into action.

HFS+

Its first task is to determine the destination backup folder, then to work out what needs to be backed up on that occasion. For HFS+ volumes, it examines the hidden FSEvents records stored at the root of each volume, which record the changes made to its files and folders. If it can’t find a ‘proper’ FSEvents record, then it performs a deep traversal of the volume to construct one. Deep traversals can take a long time, and in many cases may take over an hour, leading to cancellation of the next scheduled backup.

Once backupd knows what needs to be backed up, it calculates the disk capacity required. This has to allow for overhead or ‘padding’. This is compared with the free space on the backup destination: if there is room, the backup proceeds, otherwise this throws an error.

backupd then copies changed items to the backup destination. In order to maintain the impression that each backup is a complete copy of the source volume, it then makes hard links to all the unchanged files and folders. It is able to do this as, unlike many file systems, HFS+ supports directory hard links as well as those to files.

With the backup itself complete, backupd turns to maintenance tasks, and identifies which old backups have expired, according to its rules. Those are then deleted in a process which can take longer than making the backup itself. Once that task is done, backupd reports that the backup is complete, which is signalled back to CTS and DAS so that the next automatic backup can be scheduled by DAS.

These are summarised in the diagram below.

TMbackupHFS

APFS

backupd is scheduled using the same mechanism, which now isn’t as prone to failure as it was in the past. This starts in the normal way, by determining the destination for the backup. Having recognised that it has an APFS volume to back up to that, it then follows a different sequence.

The preparatory sequence identifies and deletes expired local snapshots. According to Apple’s Support Note, these local snapshots are kept for 24 hours; although the log entries below indicate a shorter period, later backups confirm that this is normally the case, and you should expect to find a full 24 hours of snapshots at any time.

Once that maintenance is complete, backupd makes a fresh snapshot, which is saved to the backup folder. This is set as the ‘stable’ snapshot, and mounted ready for access. The last snapshot, made an hour earlier during the Time Machine backup, is then mounted too as the ‘reference’, and backupd then determines which files and folders should be backed up by comparing those two snapshots, as stored in their backup folders.

Although the two Macs making these backups are not exactly the same, they should be fairly comparable in performance: identifying 239 files for backup on a large Fusion Drive in HFS+ (FSEvents) took about 1.6 seconds; identifying 1447 files for backup on a large Fusion Drive in APFS (snapshots) took about 23 seconds, suggesting that the new procedure is not being used for speed.

backupd then checks that there is sufficient free space on the backup destination, and if there is, performs that same process as with HFS+, copying changed items and making hard links to the rest. That is followed by new steps, which save a clone family cache to the new backup folder, and back-up-later caches there too. The precise purpose of these isn’t yet clear, although the latter may well list files which changed as the backup was being made.

The two snapshots are then unmounted, their job complete, and the latest snapshot is marked as the next to be used as the reference snapshot when backupd is next run.

A second local snapshot is then made, which shows the state of the file system on completion of the backup. This appears to be saved to the local snapshot store and not copied to the backup, and is intended for use when the full Time Machine backup isn’t available. It is accessible in the Time Machine app: for the previous 24 hours, for an APFS volume, you should find two backups made every hour instead of one. The first represents that in the backup destination, and the second (a few minutes later) is the local snapshot.

backupd finally performs the same maintenance routines on the backup folder, to remove old expired backups, on completion of which it reports that the backup is complete.

The equivalent diagram for APFS volumes is shown below.

TMbackupAPFS

How much space do these snapshots take?

Although some users consider that 24 hours worth of snapshots consumes a significant amount of disk space, Apple claims that snapshots are kept as available storage. In other words, when there is pressure for free space, macOS should automatically delete the oldest, as required. For Time Machine backups to function properly, only the latest two snapshots are needed, and are those stored in the backup anyway.

If you want to force older snapshots to be removed from the startup volume, Apple recommends turning Time Machine off for a few minutes to allow the old snapshots to be removed. When turned back on, backupd should only need to access the snapshot stored in the last backup in order to determine what needs to be backed up.

Where next?

I have been writing that Time Machine has fallen behind macOS, at least in respect of its reliance on the HFS+ file system for backups, which results from its use of directory hard links. This implementation of Time Machine for APFS is perhaps best viewed as version 1.5: it now takes best advantage of the new file system as its source, but has yet to find a new backup method and format appropriate to an APFS backup destination.

My best guess is that Time Machine 2.0, under development at the moment, will base its backups on the snapshots currently stored in each backup folder. Whether it will be ready for release in macOS 10.15 is unclear, but we should know more at WWDC 2019.

Example log extracts

All entries are written by backupd.

El Capitan/Sierra with HFS+:
47:17.053 com.apple.backupd[7985]: Starting automatic backup 47:17.268 Backing up to /dev/disk3s2: /Volumes/PROMISE PEGASUS/Backups.backupdb 47:18.834 Will copy (304.1 MB) from Macintosh HD 47:18.834 Found 239 files (304.1 MB) needing backup 47:18.863 14.63 GB required (including padding), 3.23 TB available 51:01.273 Copied 360 items (304 MB) from volume Macintosh HD. Linked 9332. 51:08.237 Created new backup: 2016-07-07-075106 51:10.048 Starting post-backup thinning 54:15.363 Deleted /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2016-07-06-065322 (27.1 MB) 54:15.363 Post-backup thinning complete: 1 expired backups removed 54:19.280 cBackup completed successfully.

Deep Traversal with HFS+:
09:44.382003 Forcing deep traversal on source: "Macintosh HD" (device: /dev/disk2 mount: '/' fsUUID: BE50387F-1302-31FB-B567-68F50F8ADE7A eventDBUUID: FDAC9A01-ECB1-4AFA-8F58-B018F81CF569) 09:44.398792 Deep event scan at path:/ reason:must scan subdirs|require scan| 09:44.398917 Running event scan 09:44.399022 periodic consistency scan for '/' 57:54.162626 Finished scan 57:54.244922 Saved event cache at /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2017-02-18-080944.inProgress/6C87A4AF-8E4C-41C5-B8FE-62BCA79A27F4/.BE50387F-1302-31FB-B567-68F50F8ADE7A.eventdb

Mojave with APFS:
37:13.324510 Starting automatic backup 37:13.944141 Backing up to /dev/disk3s2: /Volumes/PROMISE PEGASUS/Backups.backupdb 37:15.303692 Starting age based thinning of Time Machine local snapshots on disk '/' 37:15.306357 Age based thinning deleted Time Machine snapshot 'com.apple.TimeMachine.2018-10-12-074443' on disk '/ 37:20.814376 Created Time Machine local snapshot with name 'com.apple.TimeMachine.2018-10-12-093715' on disk '/' 37:20.821185 Declared stable snapshot: com.apple.TimeMachine.2018-10-12-093715 37:21.222744 Mounted stable snapshot: com.apple.TimeMachine.2018-10-12-093715 at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s iMac/2018-10-12-093715/Macintosh HD source: Macintosh HD 37:21.823102 Mounted reference snapshot: com.apple.TimeMachine.2018-10-12-083723 at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s iMac/2018-10-12-083723/Macintosh HD source: Macintosh HD 37:44.916779 Will copy (637.5 MB) from Macintosh HD 37:44.917419 Found 1447 files (637.5 MB) needing backup 37:44.953124 7.97 GB required (including padding), 4.49 TB available 41:12.017287 Copied 1593 items (623.3 MB) from volume Macintosh HD. Linked 19601. 41:12.037146 Saved clone family cache for 'Macintosh HD' at /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2018-10-12-093721.inProgress/14F6D6F4-4BCA-4EA1-B4F5-67A5B9623254/.BE50387F-1302-31FB-B567-68F50F8ADE7A.clonedb 41:12.688615 Saved back-up-later cache at /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2018-10-12-093721.inProgress/14F6D6F4-4BCA-4EA1-B4F5-67A5B9623254/.reservations.plist 41:14.014047 Saved back-up-later cache at /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2018-10-12-093721.inProgress/14F6D6F4-4BCA-4EA1-B4F5-67A5B9623254/.reservations.plist 41:14.259024 Unmounted local snapshot: com.apple.TimeMachine.2018-10-12-093715 at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s iMac/2018-10-12-093715/Macintosh HD source: Macintosh HD 41:14.507748 Unmounted local snapshot: com.apple.TimeMachine.2018-10-12-083723 at path: /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/Howard’s iMac/2018-10-12-083723/Macintosh HD source: Macintosh HD 41:15.039030 Marked as reference snapshot: com.apple.TimeMachine.2018-10-12-093715 41:15.161260 Completed snapshot: 2018-10-12-094114 41:16.127934 Starting post-backup thinning 42:01.489936 Deleted /Volumes/PROMISE PEGASUS/Backups.backupdb/Howard’s iMac/2018-10-11-090416 (12.2 MB) 42:01.489947 Post-backup thinning complete: 1 expired backups removed 42:01.580128 Backup completed successfully.

25Comments

Add yours

1

David PR Bailin on October 17, 2018 at 7:40 pm

Any easy way to delete a whole year of backups on a Time Capsule? Am down to 14 GB on a 2 TB Tiome capsule. Or will Time machine just delete what it needs as required. The original 150 GB free keeps going down slowly. Or will I need to erase the time capsule soon? Thanks

LikeLike
- 2
  
  hoakley on October 17, 2018 at 7:45 pm
  
  Although Time Machine does perform maintenance, it cannot recover such substantial amounts of space.
  If you want to keep those old backups, I recommend that you make a cloned copy of that drive onto another 2 TB hard disk. That will take a lot of time, because of the elaborate hard links, but should be achievable using Carbon Copy Cloner or similar.
  I would then, keeping that archive safely, reformat that disk and start with a fresh full backup to it. That should give you maximum free space for the future.
  Howard.
  
  LikeLike
3

Manoli on October 18, 2018 at 11:14 am

Hello Howard. You did amazing research on Time Machine backup process and make it a lot more clear what’s going on there, thank you.

I just missed something here, or maybe I misunderstood. The destination disk where Time Machine backups go has still to be HFS+ or could be APFS, too? Or which one You are using?

You say, that “Time Machine has yet to find a new backup method and format appropriate to an APFS backup destination”. It means you cannot use APFS volume as destination for Time Machine backups? I don’t know, I didn’t try, yet.

Thanks, if you make it a bit more clear, have good day, bye.

LikeLiked by 1 person
- 4
  
  hoakley on October 18, 2018 at 11:26 am
  
  Thanks, Manoli.
  No, in the current version of TM in Mojave (and earlier), the backups must be stored on an HFS+ volume. TM still relies on being able to make directory hard links in the backup, and they are not supported in APFS. So before TM can make backups to an APFS volume, Apple will have to change it to use a different system. I think that will be based on the snapshot feature in APFS, and hopefully will come in macOS 10.15 next year.
  Howard.
  
  LikeLiked by 1 person
  - 5
    
    Manoli on October 18, 2018 at 11:58 am
    
    All clear, thank you.
    
    LikeLiked by 1 person
6

EcleX on October 19, 2018 at 10:06 am

Are you sure that Time Machine 2.0 is under development at the moment? Do you have proof of it? Thanks.

LikeLiked by 1 person
- 7
  
  hoakley on October 19, 2018 at 10:18 am
  
  No. Even if someone were to ‘prove’ that it was under development, that wouldn’t ensure that it was released.
  However, APFS is now the default file system for macOS. It is preferred for all media to which Time Machine makes backups, including hard disks, Fusion Drives, SSD, and should now be coming in for RAID systems too (AppleRAID and SoftRAID). To have TM require a different file system for its backups can only be temporary.
  Not only that, but TM doesn’t work well now on HFS+. Most users can only maintain backups for 1-2 years, and pushing TM to go much longer makes backing up very slow and restoring painful too. It was never designed for modern use and high storage capacities.
  If Apple doesn’t introduce TM 2, then it may as well kill it, as users will continue drifting away from using TM altogether.
  Howard.
  
  LikeLike
8

EcleX on October 19, 2018 at 12:15 pm

Thanks for the quick and informative reply. I have found that Time Machine backups using macOS 10.12.6 (16G1510) Sierra and previous versions generates many disk issues all the time (identified when rebuilding its directory with DiskWarrior, which fixes them).

Hopefully, Apple will release Time Machine with full APFS support, as you say. Otherwise, do you know or recommend any other backup application capable of incremental backups as Time Machine does (without deleting older item versions)? I know SuperDuper! and Carbon Copy Cloner, but they delete such older items versions. Thanks again.

LikeLiked by 1 person
- 9
  
  hoakley on October 19, 2018 at 12:21 pm
  
  I start with fresh TM backups every 2-3 years now, which tends to keep these problems at bay.
  I’m afraid that I haven’t tried using any other backup software since TM came out, and wouldn’t know where to start now. Perhaps someone else can suggest something?
  Howard.
  
  LikeLike
10

Recent Links (weekly) | Symesposium on October 21, 2018 at 7:31 am

[…] How Time Machine makes backups […]

LikeLike
11

Cyril Scetbon on October 29, 2018 at 10:52 pm

When you say “determine files to back up from snapshot differences”, how does it do it ? the FSEvents in HFS+ was there to avoid scanning the filesystems but now that it’s gone and that it relies on local snapshots, how does it do it ? Does it mean it does a deep traversal every time now ?
I didn’t find a way to compare apfs snapshots other than traversing them and comparing each side.

LikeLiked by 1 person
- 12
  
  hoakley on October 29, 2018 at 11:03 pm
  
  Apple doesn’t tell us.
  The fact is that when making a Time Machine backup from an APFS volume, macOS makes a snapshot which is stored in the backup, which is referred to as the Stable snapshot. It then takes the snapshot from an hour before, which it refers to as the Reference, and mounts both. With those two snapshots mounted and open, backupd then works out what it is going to back up. This new process takes significantly longer than when using FSEvents.
  I think the evidence points to backupd comparing the two snapshots, and using the differences between them to determine which files to back up. I also suspect that Apple knows how best to do that using its own file system, and maybe pulls tricks which only it knows.
  I’m very welcome to better suggestions which fit the evidence from the logs.
  FSEvents are still there, and apparently still used to back up HFS+ volumes, which macOS can’t create snapshots for in the same way.
  Howard.
  
  LikeLike
  - 13
    
    Cyril Scetbon on October 30, 2018 at 8:41 pm
    
    Yeah I really can’t tell but like you said it probably knows how to query its filesystems to know the blocks that don’t have anymore references pointing to them to know what changed.
    
    Do you know how it handles local snapshots ? more precisely how it does to keep one snapshot per hour for the last 24 hours then 1 per day for the last 30 days then 1 per week etc.. ? I tried to use local snapshots (without a time backup) and local snapshots gets cleaned out after 24 hours. So I’m wondering if there is a way to tell it to do the same as if there was a backup drive configured.
    
    Thanks
    —
    Cyril S.
    
    LikeLiked by 1 person
    - 14
      
      hoakley on October 30, 2018 at 9:59 pm
      
      Cyril,
      The snapshots made during TM backups are stored in two locations: the normal snapshot store on the boot disk, where I think they are only kept for 24 hours before being weeded by backupd, and a copy is stored in the backup folder itself. I don’t think that there’s any easy way of using the latter, although those are what TM uses to work out what to back up, and it doesn’t delete them until it removes that entire backup.
      In theory, there might be a way to copy one of those backed up snapshots back to the boot disk, where it could be used (e.g. to roll back). But I don’t know how.
      Howard.
      
      LikeLike
- 15
  
  David PR Bailin on October 29, 2018 at 11:43 pm
  
  What do you do when the Time Capsule runs out of room? I know that Time machine makes space as needed by deleting old backups, but the free space still keeps decreasing (in my case from 150 GB to 12.8 GB, have 2 TB Time Capsule). It is very difficult to erase space manually. Is the only recourse an ERASE?
  
  LikeLiked by 1 person
  - 16
    
    hoakley on October 29, 2018 at 11:50 pm
    
    Yes.
    This is inevitable with Time Machine. Most users actually find that TM backups (and any attempts to restore from them) become so clogged and slow that they have to do this anyway.
    TM is wonderful, but wasn’t designed for today’s Macs with TB of main drive and millions of files. Eventually, even on much larger backup disks, the backups either fill the disk, or become too slow to use. One neat trick then is to make an archive copy of the backup (or, with a removable drive, to remove and store the disk), then to reformat it and start the whole process again.
    Howard.
    
    LikeLike
17

Alex on November 1, 2018 at 9:48 am

Hi Howard
First thanks for your insights.
I’ve been trying to understand time machine’s patterns for a long time.
I noticed that when running high cpu graphic intensive tasks time machine isn’t triggered for hours waiting for me to finish tasks to finally run an incremental backup which I think is fine to minimise the impact both on I/O and cpu.
Sometimes this isn’t happen for some reason and TM starts even if those tasks are running
What bothers me is that I don’t find a common pattern for this time machine triggering.
Do you have any clue from your experience ?

I’m using rmbp late-2013 HS 10.13.6 backing up to external HDD

LikeLiked by 1 person
- 18
  
  hoakley on November 1, 2018 at 10:22 am
  
  The despatching system used to trigger backups, DAS, is very complex and has elaborate heuristics. It wouldn’t surprise me if heavy CPU load was preventing it from calling for an automatic backup when your Mac is heavily loaded. Trying to work out a common pattern is probably a bit optimistic – no one outside Apple seems to known how DAS scores priorities on such routine tasks.
  So long as, when your Mac is less loaded, the auto backups occur fairly regularly, I don’t think that there’s anything to be done, or to worry about.
  Howard.
  
  LikeLike
19

How Time Machine makes backups – The Eclectic Light Company – Jerry's Mac Blog on November 1, 2018 at 9:16 pm

[…] Source: How Time Machine makes backups – The Eclectic Light Company […]

LikeLike
20

Philip Noguchi on December 13, 2018 at 7:14 am

Howard,

I have used Carbon Copy Cloner, SuperDuper and Chronosync successfully for backups. Based on your Time Machine articles, I thought it worthwhile to start using the built-in TM app. Unfortunately, the initial backup was quite a bit less than the size of my boot disk, something like 160 GB out of 200 GB. There were subsequent backups of lesser amounts around 5-10 GB roughly per hour; I had no idea if the totality of the backups would equal the original boot disk.

Searching the internet revealed that a whole lot of people were saying that they couldn’t use TM on Mojave with symptoms varying to mine to TM simply refusing to work correctly. The most common fixes were to remove Nektony App Uninstaller and to reformat the TM backup drive to HFS+, journaled, case sensitive.

Turned out that Nektony was inserting an invisible character in one of the support folders, and TM went nuts on that folder. The reformatting to case sensitive seemed to work much better for me; the current initial backup is now about 205 GB, about the same as my boot drive. Several articles mentioned that after selecting a disk in TM preference the disk would be reformatted. This did not happen with my case insensitive partition, so II reformatted my TM backup disk to ExFAT with MBR partitioning. Sure enough when I selected this disk in TM preferences, TM did note that the formatting was not compatible and then proceed to ask if I wanted to reformat. After it did its thing, I checked the partition with Disk Utility, and the partition had indeed been changed to HFS+, journaled, case sensitive! The TM menu item said that backup would begin in about 120 seconds, which was not the experience I had before and it now putting along just fine in the background.

One would think that Apple could have figured this out and issued a support notice. Better yet to document in its support articles that the drive needs to be case-sensitive. Apparently, the old Apple Time Capsule did use some sort of case sensitive formatting, but of course for the unwashed masses such a detail is apparently too “technical”. If there is a TM 2.0 I hope it has a menu item to format the backup disk.

-Phil

LikeLiked by 1 person
- 21
  
  hoakley on December 13, 2018 at 7:24 am
  
  Thank you, Phil.
  I don’t know Nektony App Uninstaller but it’s always possible it has a bug which is causing problems.
  Apple doesn’t say that HFS+ case-sensitive is required for the backup volume, because it isn’t. The great majority of Mojave TM users get along perfectly well without switching to case-sensitive. All that is required is HFS+ journalled. I’m using my old Pegasus RAID which still has all my old backups from Sierra on it, and haven’t had a glitch.
  It may be that the reformatting is the important step – not the case-sensitivity.
  Howard.
  
  LikeLike
22

Philip Noguchi on December 14, 2018 at 3:40 am

Howard,
After ranting my polemic last night I did realize your stories of the Pegasus RAID. I really can’t explain why for my setup with an ordinary external hard drive seems to do better with case-sensitive formatting.

Regarding your statement regarding formatting being in and of itself a fix did not pertain in my case. Multiple repeat formatting of the TM drivein HFS+ case insensitive did not fix my problems.

Phil

LikeLiked by 1 person
- 23
  
  hoakley on December 14, 2018 at 9:07 am
  
  Phil,
  What if your boot drive still has detritus from that incompatible app, which is only fixed by case-sensitive HFS+?
  Wouldn’t that explain it?
  Howard.
  
  LikeLike
24

Philip Noguchi on December 14, 2018 at 10:37 pm

Howard,
Thanks for the insight. That could be it.
Phil

LikeLiked by 1 person
25

Michael Tsai - Blog - How Time Machine Makes Backups With APFS on March 7, 2019 at 8:59 pm

[…] Howard Oakley: […]

LikeLike