hoakley Macs, Technology

An introduction to extended attributes, xattrs

Origin and Purposes

Extended attributes associate metadata with individual files and folders. In Classic MacOS, many files have resource forks to contain structured metadata: a classic app, for example, stores definitions of windows, menus, dialogs, etc., in its resource fork. In macOS, files and folders can also have forks, which are implemented as extended attributes: a resource fork becomes resource metadata in a xattr. Extended attributes are quite widely used in other file systems, in Linux and BSD, for example.

macOS and applications use xattrs for various purposes. The most prominent is implementing the quarantine flag which indicates that an app has been downloaded from the internet and requires full Gatekeeper checks. Other xattrs attach details of the website from which a file was downloaded, and they are used by macOS Server. They are not normally used to store metadata such as EXIF, and those associated with other media files, which are normally part of the file data.

🍌 xattrs are not hidden files with the extension .xattr, but are stored centrally on each volume.

Storage

Extended attributes are not stored in the main data for files, but in the Attributes area of the volume metadata. As such they are out of reach of normal file tools, and can only be accessed using tools specifically intended to work with xattrs. Each file and folder can have an effectively unlimited number of xattrs, each of which can be more than 100 KB.

⚠️ Because they are part of the volume metadata, some versions of macOS may not include the space occupied by them when calculating free and used disk space. It is possible to fill a volume with extended attributes and run it out of free space, and that space may also be ignored by services which manage storage use, for example space allocations for clients in a server system. These should be pathological problems, but could also result from malicious activity.

⚠️ Most file systems to which macOS can write either handle xattrs natively (HFS+, APFS), or macOS uses a scheme to preserve them. NFS is an important exception, and files copied to NFS will have all their xattrs stripped.

Types

xattrs are often named using a type signature, much like preference files. The following types are commonly encountered:

  • com.apple.FinderInfo is found very widely, and contains (a little) Finder information, sometimes including the old file type and creator strings.
  • com.apple.progress.fractionCompleted is sometimes associated with a user’s Home folder.
  • com.apple.ResourceFork is a resource fork.
  • com.apple.rootless marks items which are protected by SIP.
  • com.apple.serverdocs.markup appears to be left on some folders which have been handled by macOS Server.
  • com.apple.quarantine marks files which have been downloaded from the internet, and contains their Gatekeeper status, indicating whether they still require full checking, have passed a full check, and have been run on that Mac. This is detailed in the Gatekeeper topic.
  • com.apple.metadata is for metadata generally, and is usually qualified by a subtype, such as those below.

Metadata subtypes (termed attributes by Apple) commonly encountered include:

  • _kMDItemUserTags contains Finder tag information
  • kMDLabel_ is normally serialised with a string of apparently random letters
  • _kTimeMachineNewestSnapshot gives the latest Time Machine backup timestamp for a major folder such as your Home folder
  • _kTimeMachineOldestSnapshot gives the oldest Time Machine backup timestamp for a major folder such as your Home folder
  • kMDItemDownloadedDate gives the datestamp for when a downloaded item was obtained
  • kMDItemWhereFroms gives the URL from which a downloaded item was obtained
  • kMDItemIsScreenCapture, kMDItemScreenCaptureGlobalRect, and kMDItemScreenCaptureType for screenshots.

Examples of compound types include com.apple.metadata:kMDItemWhereFroms and com.apple.metadata:kMDLabel_jwzqfxqstyro4udfe2psoz7kyi, which has a serialised subtype.

⚠️ However, using a type signature is not a requirement. Other xattr types you may encounter include AppCrashCount and os_version.

Detection

Metadata stored in xattrs is usually hidden from the user, although Finder’s Get Info may show the content of some xattrs such as com.apple.metadata:kMDItemDownloadedDate and kMDItemWhereFroms. Some third-party apps may expose relevant metadata. For example, better text editors show and allow the setting of the information stored in com.apple.TextEncoding, which records the encoding scheme used by many text files.

The most reliable way of discovering whether one or more xattrs are associated with a file or folder is using the command ls -la. Resulting listings append the at sign @ at the end of the permissions flags to indicate that item has associated xattrs:
drwxr-xr-x@ 254 hoakley staff 8636 24 Jul 18:39 miscDocs

Tools

The standard tool for working with xattrs is the command xattr. xattrs are listed with
xattr -l itempath where itempath is the path to the file or folder to be examined.

Copying xattrs using this command is complex, and requires writing the xattr which is printed in hex form, e.g.
xattr -wx com.apple.FinderInfo "`xattr -px com.apple.FinderInfo thisitem`" thatitem
copies the com.apple.FinderInfo xattr from thisitem to thatitem.

xattred from Downloads above is a free GUI tool which displays and edits xattrs, and can be used to set the quarantine xattr for test purposes.

Common Problems

Problems with xattrs are uncommon, but can be very difficult to diagnose.

🍌 Some old commands, including cpio, zip, and pax, may omit xattrs unless they are specifically included using an option: if xattrs are to be preserved when using such commands, you should check with the man page and preferably perform a small-scale test before proceeding.

🍌 Occasionally, bugs in specific apps can attach incorrect or very large xattrs to files quite inappropriately. These can in turn cause some services to fail, most commonly the mdworker daemon responsible for indexing metadata for Spotlight. These usually present with bizarre symptoms, and until they are recognised as being caused by xattrs, they remain baffling.

References

One of the few lists of xattrs is given for com.apple.metadata subtypes in Apple’s File Metadata Attributes Reference.
There are many other articles on this blog, in the Category xattr, with its contents page (also listed to the right, at the top of the widgets there).