Skip to content

The Eclectic Light Company

Macs, painting, and more
Main navigation
  • Downloads
  • M1 & M2 Macs
  • Mac Problems
  • Mac articles
  • Art
  • Macs
  • Painting
hoakley November 2, 2020 Macs, Technology

Controlling metadata: Tricks with persistence

File data is permanent but mutable: when you open a document, you expect its content to be exactly the same as when you last saved it. Metadata is different in this respect: file attributes such as the date the file was last opened should be updated, and extended attributes (xattrs) have varying requirements for their persistence. Some metadata are disposable, and you don’t want them to propagate to copies of that file, while others need to remain sticky and go with it as long as possible.

As far as I’m aware, macOS is the only general-purpose operating system which has inbuilt mechanisms for managing the persistence of its metadata, and those are supported by its two native file systems, HFS+ and APFS. This article explains how this, one of the best-kept secrets of macOS, works.

Now you see it…

To see how to make xattrs vanish magically, you’ll need a minimum of my free app Metamer, which edits them one at a time. If you’re already experienced with xattrs, then you’ll find xattred even better for this demonstration, and the command line wizards can follow along using xattr as they wish. To do this, you need two simple test files, which might just as well be plain text containing a token amount of data. I like working with text using BBEdit, which also adds a minimum of xattrs; TextEdit gets a bit carried away at times.

metapersist1

Open your first test file using Metamer, set its Combo Box to the Comment item, type in a comment in the box below, and click Save. That’s added a standard comment xattr of type com.apple.metadata:kMDItemComment, which you should see displayed in the Finder.

Make a few copies of your test file in various places, and then inspect their xattrs. This is easiest using xattred, but you can still open them using Metamer and switch the Combo Box to read Comment: your comment should be shown below. By default, com.apple.metadata:kMDItemComment xattrs are preserved in most operations which can cope with xattrs. This makes them a good choice for storing metadata which you want to persist, and why they’re in Metamer’s menu.

…now you don’t

metapersist2

With your second test file, repeat the same process above, but this time instead of selecting the Comment item in Metamer, type com.apple.metadata:kMDItemComment#N into the Combo Box, type in your comment below, and click Save.

You can then inspect that xattr, either using xattred, or in Metamer. In the latter, instead of opening the test file and selecting the Comment item, you have to type com.apple.metadata:kMDItemComment#N into the Combo Box and press the Tab key. Notice how that xattr remains in the original file, but vanishes from most copies made of that. What’s unfortunate is that the Finder isn’t smart enough to recognise that a com.apple.metadata:kMDItemComment#N is actually still a comment.

So appending #N to a xattr’s name makes it ephemeral, and that xattr isn’t copied when the rest of the file is.

How macOS manages the persistence of metadata

For the last seven years, macOS has had an inbuilt mechanism for managing the persistence of extended attributes. It’s found still in the open source components distributed by Apple, in the copyfile component, where the relevant code is in xattr_flags.h and xattr_flags.c. For convenience I have summarised the relevant sections in the Appendix at the end of this article.

The persistence of any xattr is determined by appending flags to the name of the xattr type, a technique you may already have seen in use in the com.apple.lastuseddate#PS xattr, which is widely used. The hash sign # marks the start of what Apple refers to as a ‘property list’, but to avoid confusion I’ll call flags.

The N flag you used in the demonstration above ensures that xattr is never copied, so is ephemeral, and will remain attached to the original file, but not its copies. The PS flags used with com.apple.lastuseddate ensure that it’s normally preserved during copying, and even in syncing, making it sticky.

Two sets of defaults are defined, one for files which aren’t sandboxed, the other for those which are; these only differ in the handling of xattrs with types starting com.apple.security. The quarantine flag, com.apple.quarantine, is set to be maximally sticky, as we’ve all experienced. I use the same flag technique in my apps Fintch, Dintch and cintch to ensure that the xattrs those add to files to check their integrity are preserved when those files are copied or moved. It’s a simple but highly effective technique.

The snag is that few Apple engineers seem aware of this valuable feature, so these flags usually aren’t handled correctly. Just as with the Finder now, com.apple.metadata:kMDItemComment#N should be treated as a xattr of type com.apple.metadata:kMDItemComment with the flag N. Instead, it’s treated as a xattr with a different name. Presumably this is because Apple never got round to properly documenting this unique system for managing the persistence of metadata, and it has simply been forgotten. By Apple’s own engineers.

Appendix: macOS Metadata Persistence Flags

Flags consist of one or more characters following a hash # appended to the type name of the xattr. For example, the xattr named com.apple.lastuseddate#PS is properly of type com.apple.lastuseddate with the flags PS.

Flags can be upper or lower case letters C, N, P and S, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets (enables) that property, while lower case clears (disables) that property. The properties include:

  • C: XATTR_FLAG_CONTENT_DEPENDENT, which ties the flag and the file contents, so the xattr is rewritten when the file data changes. This can be used for checksums and hashes, text encoding, and position information. The xattr is preserved for copy and share, but not in a safe save.
  • P: XATTR_FLAG_NO_EXPORT, which doesn’t export the xattr, but normally preserves it during copying.
  • N: XATTR_FLAG_NEVER_PRESERVE, which ensures the xattr is never copied, even when copying the file.
  • S: XATTR_FLAG_SYNCABLE, which ensures the xattr is preserved even during syncing. Default behaviour is for xattrs to be stripped during syncing, to minimise the amount of data to be transferred, but this will override that default.

These must operate within another general restriction of xattrs: their name cannot exceed a maximum of 127 UTF-8 characters.

There is a set of system defaults which is baked into the xattr flag code, where the following default flags are set for different types of xattr (* here represents the wild card):

  • com.apple.quarantine – PCS
  • com.apple.TextEncoding – CS
  • com.apple.metadata:* – PS
  • com.apple.security.* – S for files which aren’t sandboxed, but N for those which are sandboxed
  • com.apple.ResourceFork – PCS
  • com.apple.FinderInfo – PCS

Full details are in xattr_flags.h and xattr_flags.c in the copyfile archive in Apple’s open source for macOS.

Share this:

  • Twitter
  • Facebook
  • Reddit
  • Pinterest
  • Email
  • Print

Like this:

Like Loading...

Related

Posted in Macs, Technology and tagged copy, extended attributes, file, Finder, macOS, metadata, Metamer, xattr, xattred. Bookmark the permalink.

Quick Links

  • Downloads
  • Mac Troubleshooting Summary
  • M1 & M2 Macs
  • Mac problem-solving
  • Painting topics
  • Painting
  • Long Reads

Search

Monthly archives

  • January 2023 (67)
  • December 2022 (74)
  • November 2022 (72)
  • October 2022 (76)
  • September 2022 (72)
  • August 2022 (75)
  • July 2022 (76)
  • June 2022 (73)
  • May 2022 (76)
  • April 2022 (71)
  • March 2022 (77)
  • February 2022 (68)
  • January 2022 (77)
  • December 2021 (75)
  • November 2021 (72)
  • October 2021 (75)
  • September 2021 (76)
  • August 2021 (75)
  • July 2021 (75)
  • June 2021 (71)
  • May 2021 (80)
  • April 2021 (79)
  • March 2021 (77)
  • February 2021 (75)
  • January 2021 (75)
  • December 2020 (77)
  • November 2020 (84)
  • October 2020 (81)
  • September 2020 (79)
  • August 2020 (103)
  • July 2020 (81)
  • June 2020 (78)
  • May 2020 (78)
  • April 2020 (81)
  • March 2020 (86)
  • February 2020 (77)
  • January 2020 (86)
  • December 2019 (82)
  • November 2019 (74)
  • October 2019 (89)
  • September 2019 (80)
  • August 2019 (91)
  • July 2019 (95)
  • June 2019 (88)
  • May 2019 (91)
  • April 2019 (79)
  • March 2019 (78)
  • February 2019 (71)
  • January 2019 (69)
  • December 2018 (79)
  • November 2018 (71)
  • October 2018 (78)
  • September 2018 (76)
  • August 2018 (78)
  • July 2018 (76)
  • June 2018 (77)
  • May 2018 (71)
  • April 2018 (67)
  • March 2018 (73)
  • February 2018 (67)
  • January 2018 (83)
  • December 2017 (94)
  • November 2017 (73)
  • October 2017 (86)
  • September 2017 (92)
  • August 2017 (69)
  • July 2017 (81)
  • June 2017 (76)
  • May 2017 (90)
  • April 2017 (76)
  • March 2017 (79)
  • February 2017 (65)
  • January 2017 (76)
  • December 2016 (75)
  • November 2016 (68)
  • October 2016 (76)
  • September 2016 (78)
  • August 2016 (70)
  • July 2016 (74)
  • June 2016 (66)
  • May 2016 (71)
  • April 2016 (67)
  • March 2016 (71)
  • February 2016 (68)
  • January 2016 (90)
  • December 2015 (96)
  • November 2015 (103)
  • October 2015 (119)
  • September 2015 (115)
  • August 2015 (117)
  • July 2015 (117)
  • June 2015 (105)
  • May 2015 (111)
  • April 2015 (119)
  • March 2015 (69)
  • February 2015 (54)
  • January 2015 (39)

Tags

APFS Apple AppleScript Apple silicon backup Big Sur Blake bug Catalina Consolation Console diagnosis Disk Utility Doré El Capitan extended attributes Finder firmware Gatekeeper Gérôme HFS+ High Sierra history of painting iCloud Impressionism iOS landscape LockRattler log logs M1 Mac Mac history macOS macOS 10.12 macOS 10.13 macOS 10.14 macOS 10.15 macOS 11 macOS 12 macOS 13 malware Mojave Monet Monterey Moreau MRT myth narrative OS X Ovid painting Pissarro Poussin privacy realism Renoir riddle Rubens Sargent scripting security Sierra SilentKnight SSD Swift symbolism Time Machine Turner update upgrade Ventura xattr Xcode XProtect

Statistics

  • 13,733,119 hits
Blog at WordPress.com.
Footer navigation
  • About & Contact
  • Macs
  • Painting
  • Language
  • Tech
  • Life
  • General
  • Downloads
  • Mac problem-solving
  • Extended attributes (xattrs)
  • Painting topics
  • Hieronymus Bosch
  • English language
  • LockRattler: 10.12 Sierra
  • LockRattler: 10.13 High Sierra
  • LockRattler: 10.11 El Capitan
  • Updates: El Capitan
  • Updates: Sierra, High Sierra, Mojave, Catalina, Big Sur
  • LockRattler: 10.14 Mojave
  • SilentKnight, silnite, LockRattler, SystHist & Scrub
  • DelightEd & Podofyllin
  • xattred, Metamer, Sandstrip & xattr tools
  • 32-bitCheck & ArchiChect
  • T2M2, Ulbow, Consolation and log utilities
  • Cirrus & Bailiff
  • Taccy, Signet, Precize, Alifix, UTIutility, Sparsity, alisma
  • Revisionist & DeepTools
  • Text Utilities: Nalaprop, Dystextia and others
  • PDF
  • Keychains & Permissions
  • LockRattler: 10.15 Catalina
  • Updates
  • Spundle, Cormorant, Stibium, Dintch, Fintch and cintch
  • Long Reads
  • Mac Troubleshooting Summary
  • LockRattler: 11.0 Big Sur
  • M1 & M2 Macs
  • Mints: a multifunction utility
  • LockRattler: 12.x Monterey
  • VisualLookUpTest
  • Virtualisation on Apple silicon
  • LockRattler: 13.x Ventura
Secondary navigation
  • Search

Post navigation

On the road: Itinerants and travellers in paintings 2
Solutions to Saturday Mac riddles 71

Begin typing your search above and press return to search. Press Esc to cancel.

  • Follow Following
    • The Eclectic Light Company
    • Join 3,125 other followers
    • Already have a WordPress.com account? Log in now.
    • The Eclectic Light Company
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...
 

    %d bloggers like this: