Skip to content

The Eclectic Light Company

Macs, painting, and more
Main navigation
  • Downloads
  • M1 & M2 Macs
  • Mac Problems
  • Mac articles
  • Art
  • Macs
  • Painting
hoakley July 23, 2019 Macs, Technology

How to save file metadata in iCloud, and new info on extended attributes

Early last year, I announced here that iCloud didn’t preserve file metadata in the form of extended attributes (xattrs) – an article which has proved one of the most popular of all those published here. I’m pleased at last to be able to report that this can be fixed, as rather than being a bug, it turns out to be a feature. This article takes an even deeper dive into xattrs to explain one of their most remarkable properties, one which seems to have escaped even the eagle eye of the great Jonathan Levin and his series of books on *OS internals.

When I looked at this issue 18 months ago, I found that passing files through iCloud Drive resulting in the stripping of some extended attributes, including (at that time):

  • com.apple.FinderInfo
  • com.apple.metadata:_kMDItemUserTags
  • com.apple.ResourceFork
  • com.apple.serverdocs.markup
  • net_sourceforge_skim-app_ series
  • net_sourceforge_skim-app_notes
  • net_sourceforge_skim-app_rtf_notes
  • net_sourceforge_skim-app_text_notes
  • co.eclecticlight.[any]

Xattrs which were preserved included:

  • com.apple.TextEncoding
  • com.apple.metadata:kMDItemDownloadedDate
  • com.apple.metadata:kMDItemWhereFroms
  • com.apple.metadata:kMDLabel_ series
  • com.apple.quarantine

This suggested some form of whitelist was being used, and we speculated that this may have been for security purposes. With no clue as to the location of that whitelist, I didn’t see a way to alter this behaviour. Then at the end of June this year, a comment added to the original article offered a way ahead. This clue was given by John Rodo (to whom I am eternally grateful), who referred me to xattr_flags.h, one of the less well-known header files for macOS. Coupled with the even more informative open source file xattr_flags.c, this article explains what I have pieced together.

In 2013, as part of its enhancements for iCloud in particular, Apple added support for flags on xattrs to indicate how those xattrs should be treated when the file is copied in various ways. Rather than change the file system, Apple opted for what is perhaps best seen as an elegant kludge: appending characters to the end of the xattr’s name.

If you work with xattrs, you’ve probably already seen this in xattrs whose name ends with a hash # then one or more characters: that’s actually the flags, not part of the name, what Apple refers to as a ‘property list’. To avoid confusion I won’t use that term here, but refer to them as xattr flags. A common example of this is com.apple.lastuseddate#PS, which is seen quite widely.

Flags can be upper or lower case letters C, N, P and S, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets (enables) that property, whilst lower case clears (disables) that property. The properties are currently:

  • C: XATTR_FLAG_CONTENT_DEPENDENT, which ties the flag and the file contents, so the xattr is rewritten when the file data changes. This is normally used for checksums and hashes, text encoding, and position information. The xattr is preserved for copy and share, but not in a safe save.
  • P: XATTR_FLAG_NO_EXPORT, which doesn’t export the xattr, but normally preserves it during copying.
  • N: XATTR_FLAG_NEVER_PRESERVE, which ensures the xattr is never copied, even when copying the file.
  • S: XATTR_FLAG_SYNCABLE, which ensures the xattr is preserved even during syncing. Default behaviour is for xattrs to be stripped during syncing, to minimise the amount of data to be transferred, but this will override that default.

These must operate within another general restriction of xattrs: their name cannot exceed a maximum of 127 UTF-8 characters.

The sought-for ‘whitelist’ is actually baked into the xattr flag code, where as of 2013 the following default flags are set for different types of xattr (* here represents the wild card):

  • com.apple.quarantine – PCS
  • com.apple.TextEncoding – CS
  • com.apple.metadata:* – PS
  • com.apple.security.* – S
  • com.apple.ResourceFork – PCS
  • com.apple.FinderInfo – PCS

Which explains some of my original observations from 18 months ago, although it would appear that some default flags have changed since 2013, or the C flag has special meaning in iCloud.

If you want a xattr preserved when it passes through iCloud, you therefore need to give it a name which ends in the xattr flag S, such as co.eclecticlight.MyTest#S. Sure enough, when xattrs with that flag are passed through iCloud Drive, those xattrs are preserved even if the default rule would treat them differently. Similarly, to have a xattr which is stripped even when you just make a local copy of that file, append #N to it.

In the next article, to appear very shortly, I’ll show how these xattr flags work in practice with iCloud Drive in Mojave, and provide a test app which can preserve all the xattrs on files which are passed through iCloud Drive.

Share this:

  • Twitter
  • Facebook
  • Reddit
  • Pinterest
  • Email
  • Print

Like this:

Like Loading...

Related

Posted in Macs, Technology and tagged Apple, extended attributes, iCloud, iCloud Drive, macOS, macOS 10.14, metadata, Mojave, SaveTheMetadata, STM, xattr. Bookmark the permalink.

1Comment

Add yours
  1. 1
    Michael Tsai - Blog - xattr Flags and iCloud Drive on July 23, 2019 at 8:42 pm

    […] Howard Oakley: […]

    LikeLike

·Comments are closed.

Quick Links

  • Downloads
  • Mac Troubleshooting Summary
  • M1 & M2 Macs
  • Mac problem-solving
  • Painting topics
  • Painting
  • Long Reads

Search

Monthly archives

  • February 2023 (7)
  • January 2023 (74)
  • December 2022 (74)
  • November 2022 (72)
  • October 2022 (76)
  • September 2022 (72)
  • August 2022 (75)
  • July 2022 (76)
  • June 2022 (73)
  • May 2022 (76)
  • April 2022 (71)
  • March 2022 (77)
  • February 2022 (68)
  • January 2022 (77)
  • December 2021 (75)
  • November 2021 (72)
  • October 2021 (75)
  • September 2021 (76)
  • August 2021 (75)
  • July 2021 (75)
  • June 2021 (71)
  • May 2021 (80)
  • April 2021 (79)
  • March 2021 (77)
  • February 2021 (75)
  • January 2021 (75)
  • December 2020 (77)
  • November 2020 (84)
  • October 2020 (81)
  • September 2020 (79)
  • August 2020 (103)
  • July 2020 (81)
  • June 2020 (78)
  • May 2020 (78)
  • April 2020 (81)
  • March 2020 (86)
  • February 2020 (77)
  • January 2020 (86)
  • December 2019 (82)
  • November 2019 (74)
  • October 2019 (89)
  • September 2019 (80)
  • August 2019 (91)
  • July 2019 (95)
  • June 2019 (88)
  • May 2019 (91)
  • April 2019 (79)
  • March 2019 (78)
  • February 2019 (71)
  • January 2019 (69)
  • December 2018 (79)
  • November 2018 (71)
  • October 2018 (78)
  • September 2018 (76)
  • August 2018 (78)
  • July 2018 (76)
  • June 2018 (77)
  • May 2018 (71)
  • April 2018 (67)
  • March 2018 (73)
  • February 2018 (67)
  • January 2018 (83)
  • December 2017 (94)
  • November 2017 (73)
  • October 2017 (86)
  • September 2017 (92)
  • August 2017 (69)
  • July 2017 (81)
  • June 2017 (76)
  • May 2017 (90)
  • April 2017 (76)
  • March 2017 (79)
  • February 2017 (65)
  • January 2017 (76)
  • December 2016 (75)
  • November 2016 (68)
  • October 2016 (76)
  • September 2016 (78)
  • August 2016 (70)
  • July 2016 (74)
  • June 2016 (66)
  • May 2016 (71)
  • April 2016 (67)
  • March 2016 (71)
  • February 2016 (68)
  • January 2016 (90)
  • December 2015 (96)
  • November 2015 (103)
  • October 2015 (119)
  • September 2015 (115)
  • August 2015 (117)
  • July 2015 (117)
  • June 2015 (105)
  • May 2015 (111)
  • April 2015 (119)
  • March 2015 (69)
  • February 2015 (54)
  • January 2015 (39)

Tags

APFS Apple AppleScript Apple silicon backup Big Sur Blake bug Catalina Consolation Console diagnosis Disk Utility Doré El Capitan extended attributes Finder firmware Gatekeeper Gérôme HFS+ High Sierra history of painting iCloud Impressionism iOS landscape LockRattler log logs M1 Mac Mac history macOS macOS 10.12 macOS 10.13 macOS 10.14 macOS 10.15 macOS 11 macOS 12 macOS 13 malware Mojave Monet Monterey Moreau MRT myth narrative OS X Ovid painting Pissarro Poussin privacy realism Renoir riddle Rubens Sargent scripting security Sierra SilentKnight SSD Swift symbolism Time Machine Turner update upgrade Ventura xattr Xcode XProtect

Statistics

  • 13,786,150 hits
Blog at WordPress.com.
Footer navigation
  • About & Contact
  • Macs
  • Painting
  • Language
  • Tech
  • Life
  • General
  • Downloads
  • Mac problem-solving
  • Extended attributes (xattrs)
  • Painting topics
  • Hieronymus Bosch
  • English language
  • LockRattler: 10.12 Sierra
  • LockRattler: 10.13 High Sierra
  • LockRattler: 10.11 El Capitan
  • Updates: El Capitan
  • Updates: Sierra, High Sierra, Mojave, Catalina, Big Sur
  • LockRattler: 10.14 Mojave
  • SilentKnight, silnite, LockRattler, SystHist & Scrub
  • DelightEd & Podofyllin
  • xattred, Metamer, Sandstrip & xattr tools
  • 32-bitCheck & ArchiChect
  • T2M2, Ulbow, Consolation and log utilities
  • Cirrus & Bailiff
  • Taccy, Signet, Precize, Alifix, UTIutility, Sparsity, alisma
  • Revisionist & DeepTools
  • Text Utilities: Nalaprop, Dystextia and others
  • PDF
  • Keychains & Permissions
  • LockRattler: 10.15 Catalina
  • Updates
  • Spundle, Cormorant, Stibium, Dintch, Fintch and cintch
  • Long Reads
  • Mac Troubleshooting Summary
  • LockRattler: 11.0 Big Sur
  • M1 & M2 Macs
  • Mints: a multifunction utility
  • LockRattler: 12.x Monterey
  • VisualLookUpTest
  • Virtualisation on Apple silicon
  • LockRattler: 13.x Ventura
Secondary navigation
  • Search

Post navigation

Mojave 10.14.6 is ready for the new Mac Pro – what has changed in the update
How to save your file metadata – implementation isn’t simple

Begin typing your search above and press return to search. Press Esc to cancel.

  • Follow Following
    • The Eclectic Light Company
    • Join 3,133 other followers
    • Already have a WordPress.com account? Log in now.
    • The Eclectic Light Company
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
%d bloggers like this: