Can macOS tell how large a file really is?

It’s a simple question: we know that Mac files can have data and extended attributes (xattrs), and we know that the Finder, the ls command, and other regular tools for discovering a file’s size essentially ignore those xattrs, but does macOS know the full size of its files?

I thought that I had found the answer when I happened across a URLResourceKey named totalFileSizeKey. As with most other things, macOS provides multiple methods for finding out the size of a file. You can use a FileManager call, for example, which only gives the size of the data fork of a file without the xattrs.

The macOS programming class which provides most information about files is URL. It has quite an elaborate interface which involves telling a file URL object which ‘keys’ you want it to reveal, then accessing those that you want. In this case, the URLResourceKey in question is totalFileSize, which Apple’s developer documentation describes as:
Key for the total displayable size of the file in bytes, returned as an NSNumber object (read-only). This includes the size of any file metadata.

Note that this doesn’t refer specifically to extended attributes, but to “file metadata”. Off the top of my head, I cannot think of any other metadata which wouldn’t be included in the data fork size, so I assumed that this actually means the total size of all extended attributes.

I don’t know of any app or tool which displays the value for that URLResourceKey, so I had to write my own – Precize, which you can download from here: precize10b1
and from Downloads above. It should work in El Capitan but I have not tested it there, and it works well in both Sierra and High Sierra, with both HFS+ and APFS.

Precize displays three measures of file size, for the data fork alone, the xattrs alone, and the total of both, using two different methods. The first row URL Keys gives the values derived from Apple’s URLResourceKeys, and the second from FileManager and examining each of the xattrs. The ‘correct’ total is that in the lower right corner, in bytes.

Armed with that tool, I discovered that no single figure obtained from macOS gives the total size of a file’s xattrs, nor the total size of those together with its data fork. In other words, you cannot obtain from macOS the full size of any of its files, unless they don’t contain any xattrs, in which case their total size is the same as that of their data fork.

What does totalFileSize return? It is a number between the data fork size and the true total file size, but is very seldom the same as the latter. As far as I can tell, the only xattr which it includes as “file metadata” is the classic Resource fork, a xattr of type com.apple.ResourceFork. This means that it is functioning as in Classic Mac OS, in only considering one data and one resource fork per file.

URL’s totalFileSize does not take into account the following widely used xattrs:

com.apple.TextEncoding,
com.apple.FinderInfo,
com.apple.metadata:kMDItemDownloadedDate,
com.apple.metadata:kMDItemWhereFroms,
com.apple.quarantine,
com.apple.serverdocs.markup,
com.apple.metadata:_kMDItemUserTags,
com.apple.metadata:kMDLabel_[*]
any third-party xattr like co.eclecticlight.[*] or net_sourceforge_skim-app[*].

Thus there seems very little point in ever using totalFileSize.

The evidence from Precize is that the only accurate way to measure the full size of a Mac file is to total the sizes of each of its xattrs, and add those to the size of its data fork. That doesn’t appear to be a function performed by macOS, or at least it is not exposed anywhere to developers or users. So, as far as I can tell, macOS itself doesn’t have any direct access to the total size of any of its files – which seems a startling omission.

The situation on macOS High Sierra, with its shiny new file system APFS, is exactly the same as in Sierra running on HFS+.

Where it does become more complex, if not downright weird, is with iCloud Drive, and its xattr filtering. To test this, I Zipped some test files with abundant xattrs and moved them onto my iCloud Drive. I then unZipped them on iCloud Drive using one Mac, and checked their sizes using Precize from that Mac and a different one, simultaneously.

The effect of iCloud Drive’s xattr filter is to show that almost all files with xattrs, viewed from two Macs at the same time, have different sizes, as shown below. Although this follows logically from the effect of the filter, it seems to defy good sense and confirms how iCloud Drive doesn’t respect your file content.

But most xattrs are small, and this doesn’t really have any significance, does it?

Like all current features of macOS which are not deprecated, xattrs are there to be used; Apple and third-party developers do store important information in xattrs, and that does build into significant quantities. I have 1.7 million files in the Home folder on my iMac, of which over 750,000 have xattrs. If their average size is 1 KB, then those xattrs alone occupy nearly 0.8 GB, which as far as macOS is concerned is essentially unaccounted for.

Xattrs are handled correctly in many functions of macOS, such as moving, copying, and duplicating files. It seems more than remiss that macOS doesn’t seem to know their total size for each file, without laboriously enumerating each of the xattrs and totting up their sizes. To not provide that information to users also seems careless.

6Comments

Add yours

1

Michael Tsai - Blog - Can macOS Tell How Large a File Really Is? on February 15, 2018 at 9:32 pm

[…] Howard Oakley: […]

LikeLike
2

Joss on February 16, 2018 at 10:03 am

This is a copy of my comment at Tsai’s place:

macOS and file sizes are a botch, including the naming schemes. You should stick to Unix and the command line, if you want to find out the actual size (correctly: sizes) of a file, or a bundle; as an example, see Quinn.app, which has resource forks and xattrs – https://i.imgur.com/GyLM1Tz.jpg –, and TextEdit, which has HFS+ compression: https://i.imgur.com/8AQJtNK.jpg

You can run the du command to get the actual disk usage (incl. slack space), but (in case of directories) you can also run stat on all file contents and add their sizes (each increased to the next allocation block size). Both are what we know as “size on disk”, or physical size. However, with HFS compression, macOS (e.g. via mdls) will output a wrong physical size, namely the size on disk that the file or directory *would* have, if there were no HFS compression. So macOS tells you that TextEdit is 8 MB on disk (physical size), whereas in reality it’s only about half of that (!), i.e. what macOS prints as physical size is actually just a “virtual size” (as it’s called here). This already tells you that you should stay far away from macOS tools (like mdls), if you want to know something about the size of an object. (Question: does APFS have compression? I think not, at least not yet.)

Then there’s the data size part, which is just the stat output, or the combined stat outputs of all files in a directory, disregarding allocation block size, i.e. not counting slack space. The totalFileSize in macOS is what’s called “Apparent Size” here, i.e. data size plus resource fork size. But the resource fork is *not* part of the file it’s associated with. It’s separate, otherwise it would be counted in the size output of the stat command. (Side note: macOS mdls command also calls this “logical size”, and there’s also a key called “file system size”, which seems to be the same; don’t know if that’s always the case.)

Then there’s the xattrs, and they are separate from the original file too, like the resource fork. So there’s what’s called “data size on volume” here, i.e. the data size of the file(s) plus its (their) resource fork(s) plus its (their) xattr(s).

In case of a bundle or directory, the root directory (*.app) should actually be added to the size count, which is 102 bytes, but it can have xattrs too. (Naturally, for regular files this is not relevant, because there is no root size, only a single file.) So in this case the data size on volume would even increase further (called “total data size on volume” here).

Yep, stick to Unix, if you want the truth. Many weird things in macOS. Leftovers from earlier times? totalFileSize = Apparent Size = data size + resource fork size seems to be a good candidate.

LikeLike
- 3
  
  hoakley on February 16, 2018 at 10:18 am
  
  Thank you.
  There isn’t, AFAIK, a command tool which actually tells you total xattr size, is there?
  ls with the @ option gives individual sizes, but not the total.
  Howard.
  
  LikeLike
4

Mats Weber on February 16, 2018 at 10:18 am

The command
ls -l@
lists xattrs and their sizes.

LikeLike
- 5
  
  hoakley on February 16, 2018 at 10:19 am
  
  I haven’t found a way of getting it to report the total size. Have you?
  It seems silly to have to sit and add them up using a calculator…
  Howard.
  
  LikeLike
  - 6
    
    Joss on February 16, 2018 at 10:23 am
    
    I haven’t found such a command, but it would be nice to have. Currently I list all the xattrs (excluding resource fork), then awk & add their data sizes.
    
    LikeLiked by 1 person

Share this:

Like this:

Related