Skip to content

The Eclectic Light Company

Macs, painting, and more
Main navigation
  • Downloads
  • M1 & M2 Macs
  • Mac Problems
  • Mac articles
  • Art
  • Macs
  • Painting
hoakley December 7, 2017 Macs, Technology, xattr

Extended attributes: surprisingly common, and information-rich

It has been bugging me for a while that I know so little about extended attributes, xattrs. Even a simple question such as how common they are in macOS files seem impossible to answer. In my earlier article and short summary about them, I only found seven different types. Does anyone have any idea as to how many types are in current use?

I thought that the best way to tackle this was to write a little app to explore folders on my Macs, calculating how many files have xattrs, and what types there are. Originally, I thought the app might steadily crawl through the whole of my 1 TB of macOS, apps, documents, and accumulated junk, but for the moment that may be a little ambitious.

Yesterday, I got my app working, and obtained some of its first results. I thought that you might like to be equally surprised by these initial figures.

xattrs are used very widely by files in macOS

Roughly a quarter of all the files I have checked so far, that’s a total of 4.6 million to date, have xattrs. That’s a little over 1.1 million files with xattrs on my Mac.

Fewest files have xattrs in /System/Library, where only 313 of 322575 files that I was able to check have them, and only a handful have more than one xattr. That’s to be expected, as xattrs are most widely used to store small amounts of transient information about a file, mutable metadata, and all the files in that folder are locked down by SIP.

/Library, with its much higher proportion of writable files, has a higher proportion using xattrs, and many with two or more of them.

As you might expect, the highest proportion of files with xattrs is in ~/Documents, where nearly 60% of files have xattrs, and the average xattr count for those files which use them is 1.5.

There are very many types of xattr in use

I thought that I had done quite well to identify seven different types when I first looked at xattrs back in August. I hadn’t even scratched the surface! In ~/Documents alone, I found 120 different types of xattr. I haven’t measured their frequencies, but many appear to be one-offs used by a single vendor for a single purpose.

I don’t propose going through the huge list of different types here, and my original idea of publishing a listing with examples here seems not to be a particularly good way ahead.

There is also huge variation in the size and content of xattrs. A few are traditional resource forks, containing large slabs of binary data. Apple’s sysdiagnose output uses one xattr type SampleWeight which often contains just the single digit ‘1’. Common types like com.apple.FinderInfo are inevitably widespread. The type com.apple.cs.CodeSignature can contain complete security certificates, although I’m not sure when it is used instead of a separate code signature file.

Apple’s xattr types

One generic approach which Apple uses to name many of its xattr types is to make them qualified sub-types of its generic com.apple.metadata. So com.apple.metadata:kMDItemCopyright is the kMDItemCopyright metadata. Apple has documented some of these in its File Metadata Attributes Reference, but additionally uses many more which do not apparently conform to this scheme, and don’t appear to be documented anywhere.

Apple has further extended this scheme to allow unique types, such as com.apple.metadata:kMDLabel_7u5sdyunqij73ecjzmbzivu4zq, which is the kMDLabel subtype of com.apple.metadata, rendered unique by appending 7u5sdyunqij73ecjzmbzivu4zq. It’s not clear how these are intended to be used, or whether they are for internal use only.

xattrs are also used extensively by third-party developers. You may come across the type com.dropbox.attributes, for example, which contains binary data used for items in a Dropbox, I presume.

You may well encounter a rival metadata naming scheme in xattrs too: that of openmetainfo.org, which uses a similar sub-typing system to Apple. For example, org.openmetainfo:kMDItemKeywords contains keywords which might be attached to an image. Apple and openmetainfo.org have common metadata types, such as kMDItemKeywords and kMDItemStarRating, and some files attach both variants for good measure.

So what? They’re no trouble, are they?

Problems with xattrs do appear very rare, although sometimes they could be the cause of the metadata compiler daemon mdworker to crash repeatedly.

xattrs are, though, storing a lot of information, some of it very relevant in security and forensics situations. As users almost never directly modify xattrs themselves, information contained in xattrs is remarkably persistent, so long as the file in question is not cleaned of its xattrs by passage through a file system which cannot support them.

The metadata contained in xattrs is also of great significance to Spotlight and related searches. Many users complain of poor results from Spotlight searches, which might be attributed to patchy metadata coverage.

Finally, I have so far only looked at xattrs in Sierra, using HFS+. APFS is claimed to offer full support for xattrs, so my next step is to check that out in High Sierra.

The app which I have been using to look at xattrs on this larger scale is XattrXverser. I don’t currently have any plans to offer it in Downloads here, although if you are interested in exploring xattrs it could save you a lot of work in the command shell.

Share this:

  • Twitter
  • Facebook
  • Reddit
  • Pinterest
  • Email
  • Print

Like this:

Like Loading...

Related

Posted in Macs, Technology, xattr and tagged extended attributes, forensics, HFS+, macOS, macOS 10.12, metadata, Sierra, xattr, xattred. Bookmark the permalink.

7Comments

Add yours
  1. 1
    Victor Maurice Faubert on December 7, 2017 at 8:50 pm

    Any idea why com.apple.quarantine shows up on photos exported from Aperture? This happens a lot to me; sometimes the xattr prevents me from viewing the photo until I remove it in Terminal and other times I have no problem. Any insights you can shed would be most helpful. Thank you.

    LikeLike

    • 2
      hoakley on December 7, 2017 at 8:54 pm

      Have these images been imported into Aperture from an online source?
      The quarantine xattr is remarkably sticky: once attached it remains until it is stripped manually.
      If not, then it sounds like a bug in Aperture, I’m afraid.
      Howard.

      LikeLike

      • 3
        Victor Maurice Faubert on December 9, 2017 at 8:56 pm

        No, they were imported into Aperture from a folder on my hard drive which had been populated by Image Capture importing the photos from either an iPhone7 Plus or from a Nikon-formatted memory chip. I then imported them into Aperture (I keep two copies of my photos, one in Aperture and one in Finder). I too would think it’s an Aperture bug, but it doesn’t always occur—some exports have no extended attributes—and I can find no reason for the difference.

        LikeLiked by 1 person

        • 4
          hoakley on December 9, 2017 at 8:59 pm

          How odd.
          You can always use my app xattred, from Downloads, to strip the quarantine xattr, rather than using the command line.
          Howard.

          LikeLike

  2. 5
    Extended attributes in High Sierra 10.13.2 – The Eclectic Light Company on December 8, 2017 at 7:31 am

    […] I revealed how common and rich in information are extended attributes, xattrs, on a desktop system with years […]

    LikeLike

  3. 6
    Victor Maurice Faubert on December 9, 2017 at 9:07 pm

    Thanks. I’ll give xattred a look; the command line works well for me once the problem is brought to my attention.

    LikeLiked by 1 person

  4. 7
    Last Week on My Mac: Documenting the hidden – The Eclectic Light Company on December 10, 2017 at 8:00 am

    […] this week, I had no idea of their prevalence in Sierra and High Sierra. I had assumed that traditional […]

    LikeLike

·Comments are closed.

Quick Links

  • Downloads
  • Mac Troubleshooting Summary
  • M1 & M2 Macs
  • Mac problem-solving
  • Painting topics
  • Painting
  • Long Reads

Search

Monthly archives

  • January 2023 (67)
  • December 2022 (74)
  • November 2022 (72)
  • October 2022 (76)
  • September 2022 (72)
  • August 2022 (75)
  • July 2022 (76)
  • June 2022 (73)
  • May 2022 (76)
  • April 2022 (71)
  • March 2022 (77)
  • February 2022 (68)
  • January 2022 (77)
  • December 2021 (75)
  • November 2021 (72)
  • October 2021 (75)
  • September 2021 (76)
  • August 2021 (75)
  • July 2021 (75)
  • June 2021 (71)
  • May 2021 (80)
  • April 2021 (79)
  • March 2021 (77)
  • February 2021 (75)
  • January 2021 (75)
  • December 2020 (77)
  • November 2020 (84)
  • October 2020 (81)
  • September 2020 (79)
  • August 2020 (103)
  • July 2020 (81)
  • June 2020 (78)
  • May 2020 (78)
  • April 2020 (81)
  • March 2020 (86)
  • February 2020 (77)
  • January 2020 (86)
  • December 2019 (82)
  • November 2019 (74)
  • October 2019 (89)
  • September 2019 (80)
  • August 2019 (91)
  • July 2019 (95)
  • June 2019 (88)
  • May 2019 (91)
  • April 2019 (79)
  • March 2019 (78)
  • February 2019 (71)
  • January 2019 (69)
  • December 2018 (79)
  • November 2018 (71)
  • October 2018 (78)
  • September 2018 (76)
  • August 2018 (78)
  • July 2018 (76)
  • June 2018 (77)
  • May 2018 (71)
  • April 2018 (67)
  • March 2018 (73)
  • February 2018 (67)
  • January 2018 (83)
  • December 2017 (94)
  • November 2017 (73)
  • October 2017 (86)
  • September 2017 (92)
  • August 2017 (69)
  • July 2017 (81)
  • June 2017 (76)
  • May 2017 (90)
  • April 2017 (76)
  • March 2017 (79)
  • February 2017 (65)
  • January 2017 (76)
  • December 2016 (75)
  • November 2016 (68)
  • October 2016 (76)
  • September 2016 (78)
  • August 2016 (70)
  • July 2016 (74)
  • June 2016 (66)
  • May 2016 (71)
  • April 2016 (67)
  • March 2016 (71)
  • February 2016 (68)
  • January 2016 (90)
  • December 2015 (96)
  • November 2015 (103)
  • October 2015 (119)
  • September 2015 (115)
  • August 2015 (117)
  • July 2015 (117)
  • June 2015 (105)
  • May 2015 (111)
  • April 2015 (119)
  • March 2015 (69)
  • February 2015 (54)
  • January 2015 (39)

Tags

APFS Apple AppleScript Apple silicon backup Big Sur Blake bug Catalina Consolation Console diagnosis Disk Utility Doré El Capitan extended attributes Finder firmware Gatekeeper Gérôme HFS+ High Sierra history of painting iCloud Impressionism iOS landscape LockRattler log logs M1 Mac Mac history macOS macOS 10.12 macOS 10.13 macOS 10.14 macOS 10.15 macOS 11 macOS 12 macOS 13 malware Mojave Monet Monterey Moreau MRT myth narrative OS X Ovid painting Pissarro Poussin privacy realism Renoir riddle Rubens Sargent scripting security Sierra SilentKnight SSD Swift symbolism Time Machine Turner update upgrade Ventura xattr Xcode XProtect

Statistics

  • 13,733,119 hits
Blog at WordPress.com.
Footer navigation
  • About & Contact
  • Macs
  • Painting
  • Language
  • Tech
  • Life
  • General
  • Downloads
  • Mac problem-solving
  • Extended attributes (xattrs)
  • Painting topics
  • Hieronymus Bosch
  • English language
  • LockRattler: 10.12 Sierra
  • LockRattler: 10.13 High Sierra
  • LockRattler: 10.11 El Capitan
  • Updates: El Capitan
  • Updates: Sierra, High Sierra, Mojave, Catalina, Big Sur
  • LockRattler: 10.14 Mojave
  • SilentKnight, silnite, LockRattler, SystHist & Scrub
  • DelightEd & Podofyllin
  • xattred, Metamer, Sandstrip & xattr tools
  • 32-bitCheck & ArchiChect
  • T2M2, Ulbow, Consolation and log utilities
  • Cirrus & Bailiff
  • Taccy, Signet, Precize, Alifix, UTIutility, Sparsity, alisma
  • Revisionist & DeepTools
  • Text Utilities: Nalaprop, Dystextia and others
  • PDF
  • Keychains & Permissions
  • LockRattler: 10.15 Catalina
  • Updates
  • Spundle, Cormorant, Stibium, Dintch, Fintch and cintch
  • Long Reads
  • Mac Troubleshooting Summary
  • LockRattler: 11.0 Big Sur
  • M1 & M2 Macs
  • Mints: a multifunction utility
  • LockRattler: 12.x Monterey
  • VisualLookUpTest
  • Virtualisation on Apple silicon
  • LockRattler: 13.x Ventura
Secondary navigation
  • Search

Post navigation

What’s really in the High Sierra 10.13.2 update
In Between: the paintings of Anita Rée, 1

Begin typing your search above and press return to search. Press Esc to cancel.

  • Follow Following
    • The Eclectic Light Company
    • Join 3,125 other followers
    • Already have a WordPress.com account? Log in now.
    • The Eclectic Light Company
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
%d bloggers like this: