It’s a simple question: we know that Mac files can have data and extended attributes (xattrs), and we know that the Finder, the ls
command, and other regular tools for discovering a file’s size essentially ignore those xattrs, but does macOS know the full size of its files?
I thought that I had found the answer when I happened across a URLResourceKey named totalFileSizeKey. As with most other things, macOS provides multiple methods for finding out the size of a file. You can use a FileManager call, for example, which only gives the size of the data fork of a file without the xattrs.
The macOS programming class which provides most information about files is URL. It has quite an elaborate interface which involves telling a file URL object which ‘keys’ you want it to reveal, then accessing those that you want. In this case, the URLResourceKey in question is totalFileSize, which Apple’s developer documentation describes as:
Key for the total displayable size of the file in bytes, returned as an NSNumber object (read-only). This includes the size of any file metadata.
Note that this doesn’t refer specifically to extended attributes, but to “file metadata”. Off the top of my head, I cannot think of any other metadata which wouldn’t be included in the data fork size, so I assumed that this actually means the total size of all extended attributes.
I don’t know of any app or tool which displays the value for that URLResourceKey, so I had to write my own – Precize, which you can download from here: precize10b1
and from Downloads above. It should work in El Capitan but I have not tested it there, and it works well in both Sierra and High Sierra, with both HFS+ and APFS.
Precize displays three measures of file size, for the data fork alone, the xattrs alone, and the total of both, using two different methods. The first row URL Keys gives the values derived from Apple’s URLResourceKeys, and the second from FileManager and examining each of the xattrs. The ‘correct’ total is that in the lower right corner, in bytes.
Armed with that tool, I discovered that no single figure obtained from macOS gives the total size of a file’s xattrs, nor the total size of those together with its data fork. In other words, you cannot obtain from macOS the full size of any of its files, unless they don’t contain any xattrs, in which case their total size is the same as that of their data fork.
What does totalFileSize return? It is a number between the data fork size and the true total file size, but is very seldom the same as the latter. As far as I can tell, the only xattr which it includes as “file metadata” is the classic Resource fork, a xattr of type com.apple.ResourceFork. This means that it is functioning as in Classic Mac OS, in only considering one data and one resource fork per file.
URL’s totalFileSize does not take into account the following widely used xattrs:
- com.apple.TextEncoding,
- com.apple.FinderInfo,
- com.apple.metadata:kMDItemDownloadedDate,
- com.apple.metadata:kMDItemWhereFroms,
- com.apple.quarantine,
- com.apple.serverdocs.markup,
- com.apple.metadata:_kMDItemUserTags,
- com.apple.metadata:kMDLabel_[*]
- any third-party xattr like co.eclecticlight.[*] or net_sourceforge_skim-app[*].
Thus there seems very little point in ever using totalFileSize.
The evidence from Precize is that the only accurate way to measure the full size of a Mac file is to total the sizes of each of its xattrs, and add those to the size of its data fork. That doesn’t appear to be a function performed by macOS, or at least it is not exposed anywhere to developers or users. So, as far as I can tell, macOS itself doesn’t have any direct access to the total size of any of its files – which seems a startling omission.
The situation on macOS High Sierra, with its shiny new file system APFS, is exactly the same as in Sierra running on HFS+.
Where it does become more complex, if not downright weird, is with iCloud Drive, and its xattr filtering. To test this, I Zipped some test files with abundant xattrs and moved them onto my iCloud Drive. I then unZipped them on iCloud Drive using one Mac, and checked their sizes using Precize from that Mac and a different one, simultaneously.
The effect of iCloud Drive’s xattr filter is to show that almost all files with xattrs, viewed from two Macs at the same time, have different sizes, as shown below. Although this follows logically from the effect of the filter, it seems to defy good sense and confirms how iCloud Drive doesn’t respect your file content.
But most xattrs are small, and this doesn’t really have any significance, does it?
Like all current features of macOS which are not deprecated, xattrs are there to be used; Apple and third-party developers do store important information in xattrs, and that does build into significant quantities. I have 1.7 million files in the Home folder on my iMac, of which over 750,000 have xattrs. If their average size is 1 KB, then those xattrs alone occupy nearly 0.8 GB, which as far as macOS is concerned is essentially unaccounted for.
Xattrs are handled correctly in many functions of macOS, such as moving, copying, and duplicating files. It seems more than remiss that macOS doesn’t seem to know their total size for each file, without laboriously enumerating each of the xattrs and totting up their sizes. To not provide that information to users also seems careless.