In case you haven’t realised it yet, the greatest skill in memory is not the remembering, but knowing what to forget. And that’s something that macOS is useless at.

When I announced details of my new app Scrub, there was quite an intense discussion on Twitter over the dangers of removing extended attributes and log files. Although I accept that there are risks, and that it is easy to destroy information which you might come to regret later, few users appreciate just how much invisible data is stored on their Macs – and never removed.

For me, the most obvious of the invisible data are extended attributes (xattr). They do some important things: the quarantine xattr tells macOS which files have been downloaded from the internet, so that if they contain apps those can be passed through Gatekeeper’s signature checks. They’re amazingly sticky: when attached to a compressed archive, they stay with each file extracted from it, and never seem to go away.

As macOS doesn’t seem to do anything useful with quarantine xattrs attached to ordinary files, the great majority of them have been sat there doing nothing since they were first written. And nothing dares to do anything about them. I’ve got com.apple.metadata:kMDItemWhereFroms, com.apple.serverdocs.markup and com.apple.quarantine xattrs still from downloads of ten years ago, when I was using Mozilla’s Camino browser.

Individually, most xattrs are quite small, but they quickly add up. Just over half of the more than 480,000 files in my ~/Documents folder have xattrs, taking up a total of 2.6 GB of metadata of my internal storage. As xattrs are kept inside the file system metadata, it’s even difficult to learn how much space they’re taking up.

They’re also tricky to weed selectively. There’s no record of when a xattr was added to a file, as adding a xattr doesn’t alter the file’s modification date – a strange omission in macOS. Add a new xattr to a file which was last modified on 7 June 2008, and its modification date remains unchanged. So when I look at any xattr, I have no way of knowing whether it has been there for the decade since that file was created, or it was added recently for an active purpose.

Xattrs are also sticky in the sense that they get copied whenever that file is copied or moved, except when it is passed through some non-Mac file systems, or through iCloud Drive. Old versions stored in the macOS versions database are more fragile, but can silently grow to consume a significant amount of storage.

I’ve recently had a bit of a purge on old versions, going through some of my old Keynote presentations and removing their previous versions. But as fast as I can do that, the database has grown back to 250 MB in size.

One very recent file, part of the source code for my Scrub app, has already stored no less than 97 old versions, each running to a little less than 30 KB. That’s almost 3 MB of old versions for a file which is currently only 31 KB in size, and was created less than a week ago.

It was when I started to develop Aquiline Check, though, that I came across the obfuscated user defaults folder, referred to in Swift as NSTemporaryDirectory or in Terminal’s command line as $TMPDIR. Although it holds material which should more properly have belonged in my ~/Library folder, it is partially concealed in the innocently-named and little-known /var/folders.

I was surprised to discover that each of my apps has its own folder there, containing files which my code had never intentionally accessed. So, it appears, do almost all the apps that I have ever run. In all, the files and folders there account for another 500 MB of storage space. As everything in there is undocumented, I have no way of knowing what is redundant, what is vital, nor of sorting the wheat from the chaff.

So if macOS isn’t prepared to help me clear old crud, what is?

Every so often, an editor manages to persuade me to review an app which claims to tidy up my Mac’s storage. Most, if not all, concentrate on files and folders, eliminating duplicates, working with the visible rather than hidden data such as xattrs, versions, and allegedly temporary caches. Maybe none of us outside Apple understands enough about all that old crud to try doing anything about it. So it continues to accumulate, and when I migrate to my next Mac, a great deal will undoubtedly move over to that system too.