Skip to content

The Eclectic Light Company

Macs, painting, and more
Main navigation
  • Downloads
  • M1 & M2 Macs
  • Mac Problems
  • Mac articles
  • Art
  • Macs
  • Painting
hoakley September 12, 2017 Macs, Technology

Watching macOS file systems: FSEvents and volume journals

Several features in macOS rely on keeping a close watch on the changes which occur in files and folders. The most obvious and important are Time Machine’s automatic backups, and Spotlight’s metadata indexer. Changes in storage are also part of the journaling system which tries to prevent damage to Mac Extended (HFS+) volumes: that is a separate issue which I will explain later.

FSEvents

When the time comes for Time Machine to make an automatic backup, it checks with the database of File System Events (FSEvents) kept on each volume to determine which items have changed since the last update. This quickly generates the list of files which it then backs up.

If that database of FSEvents is missing, or Time Machine suspects that it is incomplete, or perhaps periodically as a check, before it performs an automatic backup, Time Machine carries out a deep event scan. That checks the last modified timestamp of all the files and folders on that volume, and builds a list of those which have changed since the last backup, and thus need to be backed up now. Deep event scans are reported in Sierra’s log, and detected by my free tool T2M2 (available from Downloads above); they can easily take an hour or two, but do not necessarily indicate a problem.

Spotlight watches for FSEvents which change content that it has already indexed, or which add new content. When such changes are detected, its metadata indexer mdworker updates the metadata for the new or changed files. This doesn’t appear to depend on the same database of FSEvents on which Time Machine backups rely, and on backup volumes metadata re-indexing – now essential for Time Machine – occurs automatically after each backup.

To avoid filling the disk with FSEvent data, files and folders in your backup which change when backups are made are not recorded in the FSEvent database of that backup volume.

.fseventsd

The FSEvent database for each volume is tucked away in a hidden top-level folder named .fseventsd on that volume. This is potentially confusing, because fseventsd is also the name of the process responsible for maintaining the database, which runs as root. Gain root privileges and peek inside the folder using
sudo ls -la /.fseventsd
and you will see a very long list of FSEvents files, typically 10-20 per day, with hex names like 0000000035fa46b6, stored in compressed format. These should go back to when FSEvents were last flushed, typically when macOS last underwent a major upgrade.

When a volume on removable storage is properly unmounted, one of the tasks of macOS is to ensure its FSEvents database is fully updated before the volume is removed. If the storage device is disconnected before that can occur, then the database may be left incomplete, and those FSEvents lost. This is one reason why you should always eject removable storage properly. Some volume formats, such as FAT32 and EXFAT, appear to be unreliable at keeping proper FSEvents databases.

You can turn off FSEvent recording for a volume, by creating a .fseventsd folder at the root level of that volume, and within that hidden folder making an empty file named no_log. This is most unwise if you want it backed up by Time Machine, of course.

Data files

If you copy one of the database files from its hidden folder to a more accessible location, you’ll see that it contains a little binary content, and a very long list of the full pathnames of every file and folder which has changed over a period. Nicole Ibrahim has examined the file format in detail, discovering that each entry contains three components:

  • the full path of the item which changed,
  • flags to detail how that item changed,
  • the FSEvent identifier, a 64-bit integer serial number.

Note that the database contains all changes made by all users of that volume, that the user ID responsible for the change is not recorded, and there is no timestamp either. Time Machine simply keeps track of the last FSEvent identifier which it handled when making the last backup, and has no need for timestamps. Thus one reason for forcing a deep event scan might be a conflict in FSEvent identifiers, such as a stored identifier which is later than the most recent identifier in the FSEvents database.

Apps can also access FSEvent data which is relevant to them, on a much smaller scale, and with appropriate security and privacy protection. Few seem to do so, though.

The FSEvent system is not part of the file system, but operates on top of it. It is thus not expected to change with High Sierra’s new file system APFS, and Time Machine, Spotlight, and other software will continue to rely on it.

HFS+ journaling

Journaling was introduced as an option for Mac Extended (HFS+) volumes, and has long been the recommended default.

It addresses a problem which, in the past, could result in unrepairable damage to the file system on a volume. This commonly occurred when the Mac operating system crashed, forcing a restart. With each restart, minor damage would occur, because not all changes which should have been made to the contents of the volume had been completed when the Mac was restarted. After a few restarts, the cumulative damage reached the stage where the volume malfunctioned and could longer be repaired.

When journaling is enabled on an HFS+ volume, the file system keeps a record of the changes which should have been made, in a circular store of fixed size. As new events are added, the oldest ones are removed from the store. Journaling operates at a very low level, dealing with collections of disk blocks which are to be updated in a series of transactions.

Each block transaction handled by the journal consists of a sequence of five steps:

  1. a copy of all pending metadata changes is written to the journal;
  2. the journal file is written out to disk;
  3. the transaction is noted in the journal header;
  4. the required changes to the file system metadata are made;
  5. the transaction is marked as complete in the journal.

When an HFS+ volume is mounted, macOS checks the journal to see if any of the actions listed in it have not been completed. For those which are nearly complete, with step 3 completed but not step 5, the actions listed in the journal will be replayed to maintain consistency in the file system. Such journal replays are recorded in Sierra’s log. Note that journaling and journal replays are not intended to recover lost data, as such, but to ensure that the file system remains as consistent as possible.

APFS uses a different strategy to preserve its integrity, based on copy on write, which should make journaling unnecessary. Hence APFS volumes do not have a journalling option.

Reference

Apple’s File System Events Programming Guide explains how third-party software can use the FSEvents system, but hasn’t been updated since 2012.

Share this:

  • Twitter
  • Facebook
  • Reddit
  • Pinterest
  • Email
  • Print

Like this:

Like Loading...

Related

Posted in Macs, Technology and tagged APFS, backup, copy on write, file system, FSEvents, fseventsd, HFS+, journal, journalling, metadata, Spotlight, Time Machine. Bookmark the permalink.

Quick Links

  • Downloads
  • Mac Troubleshooting Summary
  • M1 & M2 Macs
  • Mac problem-solving
  • Painting topics
  • Painting
  • Long Reads

Search

Monthly archives

  • February 2023 (17)
  • January 2023 (74)
  • December 2022 (74)
  • November 2022 (72)
  • October 2022 (76)
  • September 2022 (72)
  • August 2022 (75)
  • July 2022 (76)
  • June 2022 (73)
  • May 2022 (76)
  • April 2022 (71)
  • March 2022 (77)
  • February 2022 (68)
  • January 2022 (77)
  • December 2021 (75)
  • November 2021 (72)
  • October 2021 (75)
  • September 2021 (76)
  • August 2021 (75)
  • July 2021 (75)
  • June 2021 (71)
  • May 2021 (80)
  • April 2021 (79)
  • March 2021 (77)
  • February 2021 (75)
  • January 2021 (75)
  • December 2020 (77)
  • November 2020 (84)
  • October 2020 (81)
  • September 2020 (79)
  • August 2020 (103)
  • July 2020 (81)
  • June 2020 (78)
  • May 2020 (78)
  • April 2020 (81)
  • March 2020 (86)
  • February 2020 (77)
  • January 2020 (86)
  • December 2019 (82)
  • November 2019 (74)
  • October 2019 (89)
  • September 2019 (80)
  • August 2019 (91)
  • July 2019 (95)
  • June 2019 (88)
  • May 2019 (91)
  • April 2019 (79)
  • March 2019 (78)
  • February 2019 (71)
  • January 2019 (69)
  • December 2018 (79)
  • November 2018 (71)
  • October 2018 (78)
  • September 2018 (76)
  • August 2018 (78)
  • July 2018 (76)
  • June 2018 (77)
  • May 2018 (71)
  • April 2018 (67)
  • March 2018 (73)
  • February 2018 (67)
  • January 2018 (83)
  • December 2017 (94)
  • November 2017 (73)
  • October 2017 (86)
  • September 2017 (92)
  • August 2017 (69)
  • July 2017 (81)
  • June 2017 (76)
  • May 2017 (90)
  • April 2017 (76)
  • March 2017 (79)
  • February 2017 (65)
  • January 2017 (76)
  • December 2016 (75)
  • November 2016 (68)
  • October 2016 (76)
  • September 2016 (78)
  • August 2016 (70)
  • July 2016 (74)
  • June 2016 (66)
  • May 2016 (71)
  • April 2016 (67)
  • March 2016 (71)
  • February 2016 (68)
  • January 2016 (90)
  • December 2015 (96)
  • November 2015 (103)
  • October 2015 (119)
  • September 2015 (115)
  • August 2015 (117)
  • July 2015 (117)
  • June 2015 (105)
  • May 2015 (111)
  • April 2015 (119)
  • March 2015 (69)
  • February 2015 (54)
  • January 2015 (39)

Tags

APFS Apple AppleScript Apple silicon backup Big Sur Blake bug Catalina Consolation Console diagnosis Disk Utility Doré El Capitan extended attributes Finder firmware Gatekeeper Gérôme HFS+ High Sierra history of painting iCloud Impressionism iOS landscape LockRattler log logs M1 Mac Mac history macOS macOS 10.12 macOS 10.13 macOS 10.14 macOS 10.15 macOS 11 macOS 12 macOS 13 malware Mojave Monet Monterey Moreau MRT myth narrative OS X Ovid painting Pissarro Poussin privacy realism Renoir riddle Rubens Sargent scripting security Sierra SilentKnight SSD Swift symbolism Time Machine Turner update upgrade Ventura xattr Xcode XProtect

Statistics

  • 13,815,564 hits
Blog at WordPress.com.
Footer navigation
  • About & Contact
  • Macs
  • Painting
  • Language
  • Tech
  • Life
  • General
  • Downloads
  • Mac problem-solving
  • Extended attributes (xattrs)
  • Painting topics
  • Hieronymus Bosch
  • English language
  • LockRattler: 10.12 Sierra
  • LockRattler: 10.13 High Sierra
  • LockRattler: 10.11 El Capitan
  • Updates: El Capitan
  • Updates: Sierra, High Sierra, Mojave, Catalina, Big Sur
  • LockRattler: 10.14 Mojave
  • SilentKnight, silnite, LockRattler, SystHist & Scrub
  • DelightEd & Podofyllin
  • xattred, Metamer, Sandstrip & xattr tools
  • 32-bitCheck & ArchiChect
  • T2M2, Ulbow, Consolation and log utilities
  • Cirrus & Bailiff
  • Taccy, Signet, Precize, Alifix, UTIutility, Sparsity, alisma
  • Revisionist & DeepTools
  • Text Utilities: Nalaprop, Dystextia and others
  • PDF
  • Keychains & Permissions
  • LockRattler: 10.15 Catalina
  • Updates
  • Spundle, Cormorant, Stibium, Dintch, Fintch and cintch
  • Long Reads
  • Mac Troubleshooting Summary
  • LockRattler: 11.0 Big Sur
  • M1 & M2 Macs
  • Mints: a multifunction utility
  • LockRattler: 12.x Monterey
  • VisualLookUpTest
  • Virtualisation on Apple silicon
  • LockRattler: 13.x Ventura
Secondary navigation
  • Search

Post navigation

Wet in wet: a brief history of watercolour – 4, After Impressionism
Edgar Degas: Landscapes

Begin typing your search above and press return to search. Press Esc to cancel.

  • Follow Following
    • The Eclectic Light Company
    • Join 3,138 other followers
    • Already have a WordPress.com account? Log in now.
    • The Eclectic Light Company
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
%d bloggers like this: