Skip to content

The Eclectic Light Company

Macs, painting, and more
Main navigation
  • Downloads
  • M1 & M2 Macs
  • Mac Problems
  • Mac articles
  • Art
  • Macs
  • Painting
hoakley October 24, 2020 Macs, Technology

There’s more to files than data: Extended Attributes

The attributes of a file include metadata which is common to many or all files, such as their name and date of creation. Extended attributes, or xattrs, are metadata which apply only to specific files, and aren’t used by the great majority: they’re options used for particular purposes.

In Classic Mac OS, many files have resource forks to contain structured metadata: a classic app, for example, stores definitions of windows, menus, dialogs, etc., in its resource fork. In macOS 10 and 11, files and folders can also have other named forks, which are implemented as extended attributes; a resource fork becomes resource metadata in a xattr of type com.apple.ResourceFork. Although sometimes claimed to be a quirk of Macs, extended attributes are now quite widely used in other file systems, in Linux and BSD, for example.

macOS and applications use xattrs for various purposes. One of the most prominent is implementing the quarantine flag which indicates that a file has been downloaded from the internet and requires full security checks. Other xattrs might attach details of the website from which a file was downloaded, copyright information, and more. They’re not normally used to store content-specific metadata such as EXIF for images, or those associated with other media files, which are normally incorporated within the file data to ensure their preservation on all file systems.

Storage

In Mac native file systems such as APFS, extended attributes aren’t stored with the main data for files, but in the Attributes area of the volume metadata. As such they’re out of reach of normal file tools, and can only be accessed using those specifically intended to work with xattrs. Each file and folder can have an effectively unlimited number of xattrs, each of which can be more than 100 KB.

Because they’re part of the volume metadata, some versions of macOS may not include the space occupied by them when calculating free and used disk space. It’s possible to fill a volume with extended attributes and run it out of free space, and that space may also be ignored by services which manage storage use, for example space allocations for clients in a server system. These should be pathological problems, but could also result from malicious activity.

Most file systems to which macOS can write either handle xattrs natively (HFS+, APFS), or macOS uses a scheme to preserve them. NFS is an important exception, and files copied to NFS will have all their xattrs stripped. Other file systems, including those popular with Linux, may impose stricter limits than APFS on the number and size of xattrs.

Types

Xattrs are almost universally named using a type signature similar to an app ID, which reads like a URL backwards, although that isn’t required and you may still encounter older names such as AppCrashCount and os_version.

The following types are examples of those most commonly encountered:

  • com.apple.FinderInfo is found very widely, and contains (a little) Finder information, sometimes including the old file type and creator strings.
  • com.apple.ResourceFork is a resource fork.
  • com.apple.rootless marks items which are protected by SIP.
  • com.apple.quarantine marks files which have been downloaded from the internet, and contains their Gatekeeper status, indicating whether they still require full checking, have passed a full check, and have been run on that Mac.
  • com.apple.metadata is for metadata generally, and is usually qualified by a subtype, such as those below.

Metadata subtypes (confusingly termed attributes by Apple) commonly encountered include:

  • _kMDItemUserTags contains Finder tag information
  • kMDLabel_ is normally serialised with a string of apparently random letters
  • kMDItemDownloadedDate gives the datestamp for when a downloaded item was obtained
  • kMDItemWhereFroms gives the URL from which a downloaded item was obtained
  • kMDItemIsScreenCapture, kMDItemScreenCaptureGlobalRect, and kMDItemScreenCaptureType for screenshots.

Examples of compound types include com.apple.metadata:kMDItemWhereFroms and com.apple.metadata:kMDLabel_jwzqfxqstyro4udfe2psoz7kyi, which has a serialised subtype.

Detection

Metadata stored in xattrs is usually hidden from the user, although Finder’s Get Info may show the content of some xattrs such as com.apple.metadata:kMDItemDownloadedDate and kMDItemWhereFroms. Some third-party apps may expose relevant metadata. For example, better text editors show and allow the setting of the information stored in com.apple.TextEncoding, which records the encoding scheme used by many text files.

In Terminal, the most reliable way of discovering whether one or more xattrs are associated with a file or folder is using the command ls -la. Resulting listings append the at sign @ at the end of the permissions flags to indicate that item has associated xattrs:
drwxr-xr-x@ 254 hoakley staff 8636 24 Jul 18:39 miscDocs

Tools

The standard command tool for working with xattrs is the command xattr. xattrs are listed with
xattr -l itempath
where itempath is the path to the file or folder to be examined.

Copying xattrs using this command is complex, and requires writing the xattr which is printed in hex form, e.g.
xattr -wx com.apple.FinderInfo "`xattr -px com.apple.FinderInfo thisitem`" thatitem
copies the com.apple.FinderInfo xattr from thisitem to thatitem.

If you’d prefer to use GUI apps to work with xattrs, I have a collection here, including:

  • xattred, a full-featured xattr editor which can do almost anything you want with xattrs, and can also attach quarantine flags if you wish to test that apps can clear Gatekeeper’s checks;
  • Metamer, a simple editor for text-based xattrs, including 16 standard types which are indexed by Spotlight;
  • Pratique, to mark document quarantine flags as clear;
  • Sandstrip, to remove spurious quarantine flags;
  • SearchKey and SearchKeyLite, which edit five major xattr types generally useful in document metadata;
  • Precize from this page gives detailed size information and a listing of xattrs for individual files;
  • cmpxat, a command tool to compare the xattrs of two files and report all differences between them.

xattred126
Xattred is a highly capable xattr editor.

metamer01
Metamer is a lightweight editor for text-based xattrs which can be used in Spotlight searches.

Common Problems

Problems with xattrs have been relatively uncommon, but are usually difficult to identify and diagnose because of their poor visibility. Perhaps their most common problem is that most measures of a file’s size forget to include the size of any xattrs. Although usually small in comparison to the size of file data, increasing use of xattrs can result in some puzzling discrepancies. To investigate xattr size more accurately, use Precize.

Over the last few years, Apple has increasingly used xattrs, both the quarantine flag com.apple.quarantine and a xattr new with Catalina, com.apple.macl, for security and privacy protection. These can lead to strange and apparently intractable problems; as they’re essentially undocumented by Apple, even discovering their cause can be difficult.

Other problems arise when xattrs whose contents are important are inadvertently stripped when a file is moved to a different volume or file system, such as iCloud. The latter now preserves more xattrs than it has in the past, but some may still need special protection. Xattrs have a longstanding system for specifying when they should be preserved, but this was only documented in obscure source code. It also appends flags to the end of the xattr type name, which isn’t generally respected and can cause problems of its own.

In the command line and shell scripts, xattrs can behave unexpectedly. Some old commands, including cpio, zip, and pax, may omit xattrs unless they’re specifically included using an option: if xattrs are to be preserved when using such commands, you should check with the man page and preferably perform a small-scale test before proceeding.

Occasionally, bugs in specific apps can attach incorrect or very large xattrs to files quite inappropriately. These can in turn cause some services to fail, most commonly the mdworker daemon responsible for indexing metadata for Spotlight. These usually present with bizarre symptoms, and until they are recognised as being caused by xattrs, they remain baffling.

Forthcoming articles will look in more detail at these different topics.

References

One of the few lists of xattrs is given for com.apple.metadata subtypes in Apple’s File Metadata Attributes Reference (last substantially updated in 2011).
There are many other articles on this blog, in the Category xattr, with its contents page (also listed to the right, at the top of the widgets there).

Share this:

  • Twitter
  • Facebook
  • Reddit
  • Pinterest
  • Email
  • Print

Like this:

Like Loading...

Related

Posted in Macs, Technology and tagged APFS, com.apple.macl, extended attributes, forks, HFS+, metadata, NFS, quarantine, resources, xattr. Bookmark the permalink.

3Comments

Add yours
  1. 1
    Week 43 – 2020 – This Week In 4n6 on October 25, 2020 at 9:20 am

    […] Howard Oakley at ‘The Eclectic Light Company’There’s more to files than data: Extended Attributes […]

    LikeLike

  2. 2
    Dan King on October 26, 2020 at 8:01 pm

    I was going to leave a long story with a request.
    I decided to just leave the request:
    Can you write about com.apple.quarantine?
    Does it have flags, what are their values. I doubt is it going to help me (never know) but it is on all the files I look at even ones that are over 10 years old (mp3s). And I would like to understand what might be going on.

    LikeLiked by 1 person

    • 3
      hoakley on October 26, 2020 at 8:32 pm

      Thank you.
      Funnily enough, that’s just what I have planned.
      Howard.

      LikeLike

·Comments are closed.

Quick Links

  • Downloads
  • Mac Troubleshooting Summary
  • M1 & M2 Macs
  • Mac problem-solving
  • Painting topics
  • Painting
  • Long Reads

Search

Monthly archives

  • February 2023 (9)
  • January 2023 (74)
  • December 2022 (74)
  • November 2022 (72)
  • October 2022 (76)
  • September 2022 (72)
  • August 2022 (75)
  • July 2022 (76)
  • June 2022 (73)
  • May 2022 (76)
  • April 2022 (71)
  • March 2022 (77)
  • February 2022 (68)
  • January 2022 (77)
  • December 2021 (75)
  • November 2021 (72)
  • October 2021 (75)
  • September 2021 (76)
  • August 2021 (75)
  • July 2021 (75)
  • June 2021 (71)
  • May 2021 (80)
  • April 2021 (79)
  • March 2021 (77)
  • February 2021 (75)
  • January 2021 (75)
  • December 2020 (77)
  • November 2020 (84)
  • October 2020 (81)
  • September 2020 (79)
  • August 2020 (103)
  • July 2020 (81)
  • June 2020 (78)
  • May 2020 (78)
  • April 2020 (81)
  • March 2020 (86)
  • February 2020 (77)
  • January 2020 (86)
  • December 2019 (82)
  • November 2019 (74)
  • October 2019 (89)
  • September 2019 (80)
  • August 2019 (91)
  • July 2019 (95)
  • June 2019 (88)
  • May 2019 (91)
  • April 2019 (79)
  • March 2019 (78)
  • February 2019 (71)
  • January 2019 (69)
  • December 2018 (79)
  • November 2018 (71)
  • October 2018 (78)
  • September 2018 (76)
  • August 2018 (78)
  • July 2018 (76)
  • June 2018 (77)
  • May 2018 (71)
  • April 2018 (67)
  • March 2018 (73)
  • February 2018 (67)
  • January 2018 (83)
  • December 2017 (94)
  • November 2017 (73)
  • October 2017 (86)
  • September 2017 (92)
  • August 2017 (69)
  • July 2017 (81)
  • June 2017 (76)
  • May 2017 (90)
  • April 2017 (76)
  • March 2017 (79)
  • February 2017 (65)
  • January 2017 (76)
  • December 2016 (75)
  • November 2016 (68)
  • October 2016 (76)
  • September 2016 (78)
  • August 2016 (70)
  • July 2016 (74)
  • June 2016 (66)
  • May 2016 (71)
  • April 2016 (67)
  • March 2016 (71)
  • February 2016 (68)
  • January 2016 (90)
  • December 2015 (96)
  • November 2015 (103)
  • October 2015 (119)
  • September 2015 (115)
  • August 2015 (117)
  • July 2015 (117)
  • June 2015 (105)
  • May 2015 (111)
  • April 2015 (119)
  • March 2015 (69)
  • February 2015 (54)
  • January 2015 (39)

Tags

APFS Apple AppleScript Apple silicon backup Big Sur Blake bug Catalina Consolation Console diagnosis Disk Utility Doré El Capitan extended attributes Finder firmware Gatekeeper Gérôme HFS+ High Sierra history of painting iCloud Impressionism iOS landscape LockRattler log logs M1 Mac Mac history macOS macOS 10.12 macOS 10.13 macOS 10.14 macOS 10.15 macOS 11 macOS 12 macOS 13 malware Mojave Monet Monterey Moreau MRT myth narrative OS X Ovid painting Pissarro Poussin privacy realism Renoir riddle Rubens Sargent scripting security Sierra SilentKnight SSD Swift symbolism Time Machine Turner update upgrade Ventura xattr Xcode XProtect

Statistics

  • 13,788,274 hits
Blog at WordPress.com.
Footer navigation
  • About & Contact
  • Macs
  • Painting
  • Language
  • Tech
  • Life
  • General
  • Downloads
  • Mac problem-solving
  • Extended attributes (xattrs)
  • Painting topics
  • Hieronymus Bosch
  • English language
  • LockRattler: 10.12 Sierra
  • LockRattler: 10.13 High Sierra
  • LockRattler: 10.11 El Capitan
  • Updates: El Capitan
  • Updates: Sierra, High Sierra, Mojave, Catalina, Big Sur
  • LockRattler: 10.14 Mojave
  • SilentKnight, silnite, LockRattler, SystHist & Scrub
  • DelightEd & Podofyllin
  • xattred, Metamer, Sandstrip & xattr tools
  • 32-bitCheck & ArchiChect
  • T2M2, Ulbow, Consolation and log utilities
  • Cirrus & Bailiff
  • Taccy, Signet, Precize, Alifix, UTIutility, Sparsity, alisma
  • Revisionist & DeepTools
  • Text Utilities: Nalaprop, Dystextia and others
  • PDF
  • Keychains & Permissions
  • LockRattler: 10.15 Catalina
  • Updates
  • Spundle, Cormorant, Stibium, Dintch, Fintch and cintch
  • Long Reads
  • Mac Troubleshooting Summary
  • LockRattler: 11.0 Big Sur
  • M1 & M2 Macs
  • Mints: a multifunction utility
  • LockRattler: 12.x Monterey
  • VisualLookUpTest
  • Virtualisation on Apple silicon
  • LockRattler: 13.x Ventura
Secondary navigation
  • Search

Post navigation

Why have my HP printers stopped working? How to check their software signature
Saturday Mac riddles 70

Begin typing your search above and press return to search. Press Esc to cancel.

  • Follow Following
    • The Eclectic Light Company
    • Join 3,133 other followers
    • Already have a WordPress.com account? Log in now.
    • The Eclectic Light Company
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...
 

    %d bloggers like this: