Skip to content

The Eclectic Light Company

Macs, painting, and more
Main navigation
  • Downloads
  • M1 & M2 Macs
  • Mac Problems
  • Mac articles
  • Art
  • Macs
  • Painting
hoakley February 12, 2018 Macs, Technology

Can macOS tell how large a file really is?

It’s a simple question: we know that Mac files can have data and extended attributes (xattrs), and we know that the Finder, the ls command, and other regular tools for discovering a file’s size essentially ignore those xattrs, but does macOS know the full size of its files?

I thought that I had found the answer when I happened across a URLResourceKey named totalFileSizeKey. As with most other things, macOS provides multiple methods for finding out the size of a file. You can use a FileManager call, for example, which only gives the size of the data fork of a file without the xattrs.

The macOS programming class which provides most information about files is URL. It has quite an elaborate interface which involves telling a file URL object which ‘keys’ you want it to reveal, then accessing those that you want. In this case, the URLResourceKey in question is totalFileSize, which Apple’s developer documentation describes as:
Key for the total displayable size of the file in bytes, returned as an NSNumber object (read-only). This includes the size of any file metadata.

Note that this doesn’t refer specifically to extended attributes, but to “file metadata”. Off the top of my head, I cannot think of any other metadata which wouldn’t be included in the data fork size, so I assumed that this actually means the total size of all extended attributes.

precize00

I don’t know of any app or tool which displays the value for that URLResourceKey, so I had to write my own – Precize, which you can download from here: precize10b1
and from Downloads above. It should work in El Capitan but I have not tested it there, and it works well in both Sierra and High Sierra, with both HFS+ and APFS.

precize01

Precize displays three measures of file size, for the data fork alone, the xattrs alone, and the total of both, using two different methods. The first row URL Keys gives the values derived from Apple’s URLResourceKeys, and the second from FileManager and examining each of the xattrs. The ‘correct’ total is that in the lower right corner, in bytes.

precize03

Armed with that tool, I discovered that no single figure obtained from macOS gives the total size of a file’s xattrs, nor the total size of those together with its data fork. In other words, you cannot obtain from macOS the full size of any of its files, unless they don’t contain any xattrs, in which case their total size is the same as that of their data fork.

What does totalFileSize return? It is a number between the data fork size and the true total file size, but is very seldom the same as the latter. As far as I can tell, the only xattr which it includes as “file metadata” is the classic Resource fork, a xattr of type com.apple.ResourceFork. This means that it is functioning as in Classic Mac OS, in only considering one data and one resource fork per file.

URL’s totalFileSize does not take into account the following widely used xattrs:

  • com.apple.TextEncoding,
  • com.apple.FinderInfo,
  • com.apple.metadata:kMDItemDownloadedDate,
  • com.apple.metadata:kMDItemWhereFroms,
  • com.apple.quarantine,
  • com.apple.serverdocs.markup,
  • com.apple.metadata:_kMDItemUserTags,
  • com.apple.metadata:kMDLabel_[*]
  • any third-party xattr like co.eclecticlight.[*] or net_sourceforge_skim-app[*].

Thus there seems very little point in ever using totalFileSize.

The evidence from Precize is that the only accurate way to measure the full size of a Mac file is to total the sizes of each of its xattrs, and add those to the size of its data fork. That doesn’t appear to be a function performed by macOS, or at least it is not exposed anywhere to developers or users. So, as far as I can tell, macOS itself doesn’t have any direct access to the total size of any of its files – which seems a startling omission.

The situation on macOS High Sierra, with its shiny new file system APFS, is exactly the same as in Sierra running on HFS+.

Where it does become more complex, if not downright weird, is with iCloud Drive, and its xattr filtering. To test this, I Zipped some test files with abundant xattrs and moved them onto my iCloud Drive. I then unZipped them on iCloud Drive using one Mac, and checked their sizes using Precize from that Mac and a different one, simultaneously.

The effect of iCloud Drive’s xattr filter is to show that almost all files with xattrs, viewed from two Macs at the same time, have different sizes, as shown below. Although this follows logically from the effect of the filter, it seems to defy good sense and confirms how iCloud Drive doesn’t respect your file content.

 

precize01precize04

 

precize03precize05

But most xattrs are small, and this doesn’t really have any significance, does it?

precize02

Like all current features of macOS which are not deprecated, xattrs are there to be used; Apple and third-party developers do store important information in xattrs, and that does build into significant quantities. I have 1.7 million files in the Home folder on my iMac, of which over 750,000 have xattrs. If their average size is 1 KB, then those xattrs alone occupy nearly 0.8 GB, which as far as macOS is concerned is essentially unaccounted for.

Xattrs are handled correctly in many functions of macOS, such as moving, copying, and duplicating files. It seems more than remiss that macOS doesn’t seem to know their total size for each file, without laboriously enumerating each of the xattrs and totting up their sizes. To not provide that information to users also seems careless.

Share this:

  • Twitter
  • Facebook
  • Reddit
  • Pinterest
  • Email
  • Print

Like this:

Like Loading...

Related

Posted in Macs, Technology and tagged APFS, extended attributes, file system, files, Finder, forks, HFS+, macOS, metadata, resource, Sierra, xattr, xattred. Bookmark the permalink.

6Comments

Add yours
  1. 1
    Michael Tsai - Blog - Can macOS Tell How Large a File Really Is? on February 15, 2018 at 9:32 pm

    […] Howard Oakley: […]

    LikeLike

  2. 2
    Joss on February 16, 2018 at 10:03 am

    This is a copy of my comment at Tsai’s place:

    macOS and file sizes are a botch, including the naming schemes. You should stick to Unix and the command line, if you want to find out the actual size (correctly: sizes) of a file, or a bundle; as an example, see Quinn.app, which has resource forks and xattrs – https://i.imgur.com/GyLM1Tz.jpg –, and TextEdit, which has HFS+ compression: https://i.imgur.com/8AQJtNK.jpg

    You can run the du command to get the actual disk usage (incl. slack space), but (in case of directories) you can also run stat on all file contents and add their sizes (each increased to the next allocation block size). Both are what we know as “size on disk”, or physical size. However, with HFS compression, macOS (e.g. via mdls) will output a wrong physical size, namely the size on disk that the file or directory *would* have, if there were no HFS compression. So macOS tells you that TextEdit is 8 MB on disk (physical size), whereas in reality it’s only about half of that (!), i.e. what macOS prints as physical size is actually just a “virtual size” (as it’s called here). This already tells you that you should stay far away from macOS tools (like mdls), if you want to know something about the size of an object. (Question: does APFS have compression? I think not, at least not yet.)

    Then there’s the data size part, which is just the stat output, or the combined stat outputs of all files in a directory, disregarding allocation block size, i.e. not counting slack space. The totalFileSize in macOS is what’s called “Apparent Size” here, i.e. data size plus resource fork size. But the resource fork is *not* part of the file it’s associated with. It’s separate, otherwise it would be counted in the size output of the stat command. (Side note: macOS mdls command also calls this “logical size”, and there’s also a key called “file system size”, which seems to be the same; don’t know if that’s always the case.)

    Then there’s the xattrs, and they are separate from the original file too, like the resource fork. So there’s what’s called “data size on volume” here, i.e. the data size of the file(s) plus its (their) resource fork(s) plus its (their) xattr(s).

    In case of a bundle or directory, the root directory (*.app) should actually be added to the size count, which is 102 bytes, but it can have xattrs too. (Naturally, for regular files this is not relevant, because there is no root size, only a single file.) So in this case the data size on volume would even increase further (called “total data size on volume” here).

    Yep, stick to Unix, if you want the truth. Many weird things in macOS. Leftovers from earlier times? totalFileSize = Apparent Size = data size + resource fork size seems to be a good candidate.

    LikeLike

    • 3
      hoakley on February 16, 2018 at 10:18 am

      Thank you.
      There isn’t, AFAIK, a command tool which actually tells you total xattr size, is there?
      ls with the @ option gives individual sizes, but not the total.
      Howard.

      LikeLike

  3. 4
    Mats Weber on February 16, 2018 at 10:18 am

    The command
    ls -l@
    lists xattrs and their sizes.

    LikeLike

    • 5
      hoakley on February 16, 2018 at 10:19 am

      I haven’t found a way of getting it to report the total size. Have you?
      It seems silly to have to sit and add them up using a calculator…
      Howard.

      LikeLike

      • 6
        Joss on February 16, 2018 at 10:23 am

        I haven’t found such a command, but it would be nice to have. Currently I list all the xattrs (excluding resource fork), then awk & add their data sizes.

        LikeLiked by 1 person

·Comments are closed.

Quick Links

  • Downloads
  • Mac Troubleshooting Summary
  • M1 & M2 Macs
  • Mac problem-solving
  • Painting topics
  • Painting
  • Long Reads

Search

Monthly archives

  • January 2023 (74)
  • December 2022 (74)
  • November 2022 (72)
  • October 2022 (76)
  • September 2022 (72)
  • August 2022 (75)
  • July 2022 (76)
  • June 2022 (73)
  • May 2022 (76)
  • April 2022 (71)
  • March 2022 (77)
  • February 2022 (68)
  • January 2022 (77)
  • December 2021 (75)
  • November 2021 (72)
  • October 2021 (75)
  • September 2021 (76)
  • August 2021 (75)
  • July 2021 (75)
  • June 2021 (71)
  • May 2021 (80)
  • April 2021 (79)
  • March 2021 (77)
  • February 2021 (75)
  • January 2021 (75)
  • December 2020 (77)
  • November 2020 (84)
  • October 2020 (81)
  • September 2020 (79)
  • August 2020 (103)
  • July 2020 (81)
  • June 2020 (78)
  • May 2020 (78)
  • April 2020 (81)
  • March 2020 (86)
  • February 2020 (77)
  • January 2020 (86)
  • December 2019 (82)
  • November 2019 (74)
  • October 2019 (89)
  • September 2019 (80)
  • August 2019 (91)
  • July 2019 (95)
  • June 2019 (88)
  • May 2019 (91)
  • April 2019 (79)
  • March 2019 (78)
  • February 2019 (71)
  • January 2019 (69)
  • December 2018 (79)
  • November 2018 (71)
  • October 2018 (78)
  • September 2018 (76)
  • August 2018 (78)
  • July 2018 (76)
  • June 2018 (77)
  • May 2018 (71)
  • April 2018 (67)
  • March 2018 (73)
  • February 2018 (67)
  • January 2018 (83)
  • December 2017 (94)
  • November 2017 (73)
  • October 2017 (86)
  • September 2017 (92)
  • August 2017 (69)
  • July 2017 (81)
  • June 2017 (76)
  • May 2017 (90)
  • April 2017 (76)
  • March 2017 (79)
  • February 2017 (65)
  • January 2017 (76)
  • December 2016 (75)
  • November 2016 (68)
  • October 2016 (76)
  • September 2016 (78)
  • August 2016 (70)
  • July 2016 (74)
  • June 2016 (66)
  • May 2016 (71)
  • April 2016 (67)
  • March 2016 (71)
  • February 2016 (68)
  • January 2016 (90)
  • December 2015 (96)
  • November 2015 (103)
  • October 2015 (119)
  • September 2015 (115)
  • August 2015 (117)
  • July 2015 (117)
  • June 2015 (105)
  • May 2015 (111)
  • April 2015 (119)
  • March 2015 (69)
  • February 2015 (54)
  • January 2015 (39)

Tags

APFS Apple AppleScript Apple silicon backup Big Sur Blake bug Catalina Consolation Console diagnosis Disk Utility Doré El Capitan extended attributes Finder firmware Gatekeeper Gérôme HFS+ High Sierra history of painting iCloud Impressionism iOS landscape LockRattler log logs M1 Mac Mac history macOS macOS 10.12 macOS 10.13 macOS 10.14 macOS 10.15 macOS 11 macOS 12 macOS 13 malware Mojave Monet Monterey Moreau MRT myth narrative OS X Ovid painting Pissarro Poussin privacy realism Renoir riddle Rubens Sargent scripting security Sierra SilentKnight SSD Swift symbolism Time Machine Turner update upgrade Ventura xattr Xcode XProtect

Statistics

  • 13,758,187 hits
Blog at WordPress.com.
Footer navigation
  • About & Contact
  • Macs
  • Painting
  • Language
  • Tech
  • Life
  • General
  • Downloads
  • Mac problem-solving
  • Extended attributes (xattrs)
  • Painting topics
  • Hieronymus Bosch
  • English language
  • LockRattler: 10.12 Sierra
  • LockRattler: 10.13 High Sierra
  • LockRattler: 10.11 El Capitan
  • Updates: El Capitan
  • Updates: Sierra, High Sierra, Mojave, Catalina, Big Sur
  • LockRattler: 10.14 Mojave
  • SilentKnight, silnite, LockRattler, SystHist & Scrub
  • DelightEd & Podofyllin
  • xattred, Metamer, Sandstrip & xattr tools
  • 32-bitCheck & ArchiChect
  • T2M2, Ulbow, Consolation and log utilities
  • Cirrus & Bailiff
  • Taccy, Signet, Precize, Alifix, UTIutility, Sparsity, alisma
  • Revisionist & DeepTools
  • Text Utilities: Nalaprop, Dystextia and others
  • PDF
  • Keychains & Permissions
  • LockRattler: 10.15 Catalina
  • Updates
  • Spundle, Cormorant, Stibium, Dintch, Fintch and cintch
  • Long Reads
  • Mac Troubleshooting Summary
  • LockRattler: 11.0 Big Sur
  • M1 & M2 Macs
  • Mints: a multifunction utility
  • LockRattler: 12.x Monterey
  • VisualLookUpTest
  • Virtualisation on Apple silicon
  • LockRattler: 13.x Ventura
Secondary navigation
  • Search

Post navigation

The Rise of the Clinic: 2, hospitals
Changing Stories: Ovid’s Metamorphoses on canvas, 74 – The Shrinking Sibyl

Begin typing your search above and press return to search. Press Esc to cancel.

  • Follow Following
    • The Eclectic Light Company
    • Join 3,127 other followers
    • Already have a WordPress.com account? Log in now.
    • The Eclectic Light Company
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...
 

    %d bloggers like this: