Skip to content

The Eclectic Light Company

Macs, painting, and more
Main navigation
  • Downloads
  • M1 & M2 Macs
  • Mac Problems
  • Mac articles
  • Art
  • Macs
  • Painting
hoakley July 23, 2019 Macs, Technology

How to save your file metadata – implementation isn’t simple

In the first article of this pair, I explained a previously little-known system of extended attribute (xattr) flags introduced in 2013, which allows macOS to determine whether to preserve or strip individual xattrs from files (or folders) when they are copied. This article looks at how this can be implemented in practice, and provides a demonstration app, SaveTheMetadata, which will preserve almost all xattrs when they’re passed through iCloud Drive.

In macOS, functions for working with xattr flags are provided as part of the Standard C Library libc, and documented in man xattr_name_with_flags, for instance. These are mainly concerned with discovering the copying intent for a given xattr, and aren’t suitable for manipulating xattrs from a language such as Swift. Given the essential simplicity of the xattr flag system, the manipulation of xattrs directly seems a better option.

Currently, I access xattrs based on a Swift wrapper originally posted by Martin R on StackOverflow. Neither that nor existing functions in macOS appear to give direct access to changing the name of an existing xattr: if you know better, please tell! Therefore the only means that I can see of achieving this is to create a new xattr named with the original name and xattr flag(s) appended, and the same contents as the original, then deleting that original:
// for a xattr named item
if item.count < 126 {
let theNewName = item + "#S"
let theData = try self.extendedAttribute(forName: item)
try self.setExtendedAttribute(data: theData, forName: theNewName)
try self.removeExtendedAttribute(forName: item) }

to add the xattr flag S to a xattr without any existing flags, for instance. This gets more complex when trying to deal with xattrs with existing flags, though, where removal of the original doesn’t appear to work.

It’s relatively complex to add xattr flags to existing xattrs using my free tool xattred, as there isn’t an easily-accessible feature yet to rename a xattr. The best way to do this is to select the xattr, then click on the Edit button. In the editor drop-down, append the # and flag(s) in the name at the top, then click Save. This leaves both the original xattr and the new one with its flagged name, so you then select the original and click on Cut to remove it.

I have built a demo app, SaveTheMetadata, which does this automatically to all xattrs of files which are dropped onto it. All xattrs which don’t currently have flags, apart from com.apple.quarantine and com.apple.security*, are rewritten with #S appended to their names, provided the name doesn’t then exceed the 127 character limit.

SaveTheMetadata version 1.0b2 is available from here: STM10b2
but I’m not putting it into other pages here just yet, as it needs more extensive testing, and has limited utility at present, as I’ll explain.

The snag with these methods is that, while they preserve xattrs robustly, the renamed xattrs aren’t recognised by most apps which use them. This is because the apps aren’t expecting the #S xattr flag, so don’t drop the flag from the name, thus don’t recognise the xattr. This is neatly demonstrated with one app which relies heavily on xattrs: Skim, the PDF reader and annotation tool.

xattrflags01

I started by adding annotations to a test PDF using Skim 1.5.2, as shown.

xattrflags02

The resulting xattrs, as shown here in xattred, consist of five added for the Skim annotations, two left from the document’s creation using PDF Expert, and four ‘standard’ xattrs from macOS. I then passed that file through iCloud Drive to another Mac running Mojave 10.14.5, which stripped all the xattrs apart from two of the standard ones.

xattrflags03

xattrflags04

This removed all the annotations which I had added in Skim, and is a well-known limitation of that app.

Returning to the original annotated file on the first Mac, I next ran it through SaveTheMetadata, which added the S xattr flag to all the xattrs apart from com.apple.lastuseddate, which already had the PS flags present.

xattrflags05

Passing that file through iCloud Drive to the other Mac resulted in preservation of all the custom xattrs, but com.apple.FinderInfo and com.apple.metadata:_kMDItemUserTags were still stripped, presumably because of overriding rules which ignored the S xattr flag.

xattrflags06

Thus the S flag preserved all the xattrs containing Skim’s annotations, but when that file was opened in Skim, it was unable to read them because it doesn’t strip the xattr flags from their names when reading xattrs. And that is the problem with this technique: although the xattrs are, in general, preserved, because most apps don’t expect to have to handle xattr flags appended to their names, the preserved xattrs aren’t used – after all that.

The solution therefore is for all apps which access xattrs by name to drop any xattr flags from the names before using them. This demonstrates the cost of this elegant kludge. As this isn’t handled transparently in calls such as getxattr() which access xattrs, every app is left to its own devices to handle xattr flags appended to xattr names, which is inefficient and encourages inconsistencies between apps.

One immediate solution would be to provide the inverse of SaveTheMetadata, to strip xattr flags after transit through iCloud Drive. However, this would be a kludge of a kludge, and possibly not workable in practice. For apps like Skim, though, there is at last a clear way ahead for them to preserve xattrs across iCloud, provided that they handle xattr flags transparently.

I look forward to your comments and discussion.

Share this:

  • Twitter
  • Facebook
  • Reddit
  • Pinterest
  • Email
  • Print

Like this:

Like Loading...

Related

Posted in Macs, Technology and tagged Apple, extended attributes, iCloud, iCloud Drive, macOS, macOS 10.14, metadata, Mojave, SaveTheMetadata, STM, xattr. Bookmark the permalink.

3Comments

Add yours
  1. 1
    Martin Wierschin on July 23, 2019 at 9:59 pm

    I’ve only worked with xattrs a little bit (and a long time ago), so I had no idea they potentially had associated flags appended to their names. That’s certainly a pitfall I’ll have to remember for the future.

    Thanks Howard for elucidating this subject and exploring some potential user-level consequences.

    I’m a little surprised that xattr functions like “getxattr” don’t handle flags transparently, or weren’t superseded by extended APIs when flags were introduced. Even worse, the Apple documentation and headers for the primitive xattr functions don’t even mention flags. Everything related to flags is sequestered in a separate header. That’s burying the potential problem for developers and might be worth a feedback report to Apple.

    LikeLiked by 1 person

    • 2
      hoakley on July 23, 2019 at 10:32 pm

      Thanks, Martin. I’ve been working with them in considerable detail now for a good couple of years, and was astonished to read these details, once I had been pointed in the right direction.
      Howard.

      LikeLike

  2. 3
    Michael Tsai - Blog - xattr Flags and iCloud Drive on July 24, 2019 at 8:30 pm

    […] Update (2019-07-24): Howard Oakley: […]

    LikeLike

·Comments are closed.

Quick Links

  • Downloads
  • Mac Troubleshooting Summary
  • M1 & M2 Macs
  • Mac problem-solving
  • Painting topics
  • Painting
  • Long Reads

Search

Monthly archives

  • February 2023 (7)
  • January 2023 (74)
  • December 2022 (74)
  • November 2022 (72)
  • October 2022 (76)
  • September 2022 (72)
  • August 2022 (75)
  • July 2022 (76)
  • June 2022 (73)
  • May 2022 (76)
  • April 2022 (71)
  • March 2022 (77)
  • February 2022 (68)
  • January 2022 (77)
  • December 2021 (75)
  • November 2021 (72)
  • October 2021 (75)
  • September 2021 (76)
  • August 2021 (75)
  • July 2021 (75)
  • June 2021 (71)
  • May 2021 (80)
  • April 2021 (79)
  • March 2021 (77)
  • February 2021 (75)
  • January 2021 (75)
  • December 2020 (77)
  • November 2020 (84)
  • October 2020 (81)
  • September 2020 (79)
  • August 2020 (103)
  • July 2020 (81)
  • June 2020 (78)
  • May 2020 (78)
  • April 2020 (81)
  • March 2020 (86)
  • February 2020 (77)
  • January 2020 (86)
  • December 2019 (82)
  • November 2019 (74)
  • October 2019 (89)
  • September 2019 (80)
  • August 2019 (91)
  • July 2019 (95)
  • June 2019 (88)
  • May 2019 (91)
  • April 2019 (79)
  • March 2019 (78)
  • February 2019 (71)
  • January 2019 (69)
  • December 2018 (79)
  • November 2018 (71)
  • October 2018 (78)
  • September 2018 (76)
  • August 2018 (78)
  • July 2018 (76)
  • June 2018 (77)
  • May 2018 (71)
  • April 2018 (67)
  • March 2018 (73)
  • February 2018 (67)
  • January 2018 (83)
  • December 2017 (94)
  • November 2017 (73)
  • October 2017 (86)
  • September 2017 (92)
  • August 2017 (69)
  • July 2017 (81)
  • June 2017 (76)
  • May 2017 (90)
  • April 2017 (76)
  • March 2017 (79)
  • February 2017 (65)
  • January 2017 (76)
  • December 2016 (75)
  • November 2016 (68)
  • October 2016 (76)
  • September 2016 (78)
  • August 2016 (70)
  • July 2016 (74)
  • June 2016 (66)
  • May 2016 (71)
  • April 2016 (67)
  • March 2016 (71)
  • February 2016 (68)
  • January 2016 (90)
  • December 2015 (96)
  • November 2015 (103)
  • October 2015 (119)
  • September 2015 (115)
  • August 2015 (117)
  • July 2015 (117)
  • June 2015 (105)
  • May 2015 (111)
  • April 2015 (119)
  • March 2015 (69)
  • February 2015 (54)
  • January 2015 (39)

Tags

APFS Apple AppleScript Apple silicon backup Big Sur Blake bug Catalina Consolation Console diagnosis Disk Utility Doré El Capitan extended attributes Finder firmware Gatekeeper Gérôme HFS+ High Sierra history of painting iCloud Impressionism iOS landscape LockRattler log logs M1 Mac Mac history macOS macOS 10.12 macOS 10.13 macOS 10.14 macOS 10.15 macOS 11 macOS 12 macOS 13 malware Mojave Monet Monterey Moreau MRT myth narrative OS X Ovid painting Pissarro Poussin privacy realism Renoir riddle Rubens Sargent scripting security Sierra SilentKnight SSD Swift symbolism Time Machine Turner update upgrade Ventura xattr Xcode XProtect

Statistics

  • 13,787,216 hits
Blog at WordPress.com.
Footer navigation
  • About & Contact
  • Macs
  • Painting
  • Language
  • Tech
  • Life
  • General
  • Downloads
  • Mac problem-solving
  • Extended attributes (xattrs)
  • Painting topics
  • Hieronymus Bosch
  • English language
  • LockRattler: 10.12 Sierra
  • LockRattler: 10.13 High Sierra
  • LockRattler: 10.11 El Capitan
  • Updates: El Capitan
  • Updates: Sierra, High Sierra, Mojave, Catalina, Big Sur
  • LockRattler: 10.14 Mojave
  • SilentKnight, silnite, LockRattler, SystHist & Scrub
  • DelightEd & Podofyllin
  • xattred, Metamer, Sandstrip & xattr tools
  • 32-bitCheck & ArchiChect
  • T2M2, Ulbow, Consolation and log utilities
  • Cirrus & Bailiff
  • Taccy, Signet, Precize, Alifix, UTIutility, Sparsity, alisma
  • Revisionist & DeepTools
  • Text Utilities: Nalaprop, Dystextia and others
  • PDF
  • Keychains & Permissions
  • LockRattler: 10.15 Catalina
  • Updates
  • Spundle, Cormorant, Stibium, Dintch, Fintch and cintch
  • Long Reads
  • Mac Troubleshooting Summary
  • LockRattler: 11.0 Big Sur
  • M1 & M2 Macs
  • Mints: a multifunction utility
  • LockRattler: 12.x Monterey
  • VisualLookUpTest
  • Virtualisation on Apple silicon
  • LockRattler: 13.x Ventura
Secondary navigation
  • Search

Post navigation

How to save file metadata in iCloud, and new info on extended attributes
The Nabis: A brief account

Begin typing your search above and press return to search. Press Esc to cancel.

  • Follow Following
    • The Eclectic Light Company
    • Join 3,133 other followers
    • Already have a WordPress.com account? Log in now.
    • The Eclectic Light Company
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...
 

    %d bloggers like this: