Several features in macOS rely on keeping a close watch on the changes which occur in files and folders. The most obvious and important are Time Machine’s automatic backups, and Spotlight’s metadata indexer. Changes in storage are also part of the journaling system which tries to prevent damage to Mac Extended (HFS+) volumes: that is a separate issue which I will explain later.
FSEvents
When the time comes for Time Machine to make an automatic backup, it checks with the database of File System Events (FSEvents) kept on each volume to determine which items have changed since the last update. This quickly generates the list of files which it then backs up.
If that database of FSEvents is missing, or Time Machine suspects that it is incomplete, or perhaps periodically as a check, before it performs an automatic backup, Time Machine carries out a deep event scan. That checks the last modified timestamp of all the files and folders on that volume, and builds a list of those which have changed since the last backup, and thus need to be backed up now. Deep event scans are reported in Sierra’s log, and detected by my free tool T2M2 (available from Downloads above); they can easily take an hour or two, but do not necessarily indicate a problem.
Spotlight watches for FSEvents which change content that it has already indexed, or which add new content. When such changes are detected, its metadata indexer mdworker updates the metadata for the new or changed files. This doesn’t appear to depend on the same database of FSEvents on which Time Machine backups rely, and on backup volumes metadata re-indexing – now essential for Time Machine – occurs automatically after each backup.
To avoid filling the disk with FSEvent data, files and folders in your backup which change when backups are made are not recorded in the FSEvent database of that backup volume.
.fseventsd
The FSEvent database for each volume is tucked away in a hidden top-level folder named .fseventsd on that volume. This is potentially confusing, because fseventsd is also the name of the process responsible for maintaining the database, which runs as root. Gain root privileges and peek inside the folder using
sudo ls -la /.fseventsd
and you will see a very long list of FSEvents files, typically 10-20 per day, with hex names like 0000000035fa46b6, stored in compressed format. These should go back to when FSEvents were last flushed, typically when macOS last underwent a major upgrade.
When a volume on removable storage is properly unmounted, one of the tasks of macOS is to ensure its FSEvents database is fully updated before the volume is removed. If the storage device is disconnected before that can occur, then the database may be left incomplete, and those FSEvents lost. This is one reason why you should always eject removable storage properly. Some volume formats, such as FAT32 and EXFAT, appear to be unreliable at keeping proper FSEvents databases.
You can turn off FSEvent recording for a volume, by creating a .fseventsd folder at the root level of that volume, and within that hidden folder making an empty file named no_log. This is most unwise if you want it backed up by Time Machine, of course.
Data files
If you copy one of the database files from its hidden folder to a more accessible location, you’ll see that it contains a little binary content, and a very long list of the full pathnames of every file and folder which has changed over a period. Nicole Ibrahim has examined the file format in detail, discovering that each entry contains three components:
- the full path of the item which changed,
- flags to detail how that item changed,
- the FSEvent identifier, a 64-bit integer serial number.
Note that the database contains all changes made by all users of that volume, that the user ID responsible for the change is not recorded, and there is no timestamp either. Time Machine simply keeps track of the last FSEvent identifier which it handled when making the last backup, and has no need for timestamps. Thus one reason for forcing a deep event scan might be a conflict in FSEvent identifiers, such as a stored identifier which is later than the most recent identifier in the FSEvents database.
Apps can also access FSEvent data which is relevant to them, on a much smaller scale, and with appropriate security and privacy protection. Few seem to do so, though.
The FSEvent system is not part of the file system, but operates on top of it. It is thus not expected to change with High Sierra’s new file system APFS, and Time Machine, Spotlight, and other software will continue to rely on it.
HFS+ journaling
Journaling was introduced as an option for Mac Extended (HFS+) volumes, and has long been the recommended default.
It addresses a problem which, in the past, could result in unrepairable damage to the file system on a volume. This commonly occurred when the Mac operating system crashed, forcing a restart. With each restart, minor damage would occur, because not all changes which should have been made to the contents of the volume had been completed when the Mac was restarted. After a few restarts, the cumulative damage reached the stage where the volume malfunctioned and could longer be repaired.
When journaling is enabled on an HFS+ volume, the file system keeps a record of the changes which should have been made, in a circular store of fixed size. As new events are added, the oldest ones are removed from the store. Journaling operates at a very low level, dealing with collections of disk blocks which are to be updated in a series of transactions.
Each block transaction handled by the journal consists of a sequence of five steps:
- a copy of all pending metadata changes is written to the journal;
- the journal file is written out to disk;
- the transaction is noted in the journal header;
- the required changes to the file system metadata are made;
- the transaction is marked as complete in the journal.
When an HFS+ volume is mounted, macOS checks the journal to see if any of the actions listed in it have not been completed. For those which are nearly complete, with step 3 completed but not step 5, the actions listed in the journal will be replayed to maintain consistency in the file system. Such journal replays are recorded in Sierra’s log. Note that journaling and journal replays are not intended to recover lost data, as such, but to ensure that the file system remains as consistent as possible.
APFS uses a different strategy to preserve its integrity, based on copy on write, which should make journaling unnecessary. Hence APFS volumes do not have a journalling option.
Reference
Apple’s File System Events Programming Guide explains how third-party software can use the FSEvents system, but hasn’t been updated since 2012.
