Disk images are one of those wonderful ideas which we take for granted, and can be powerful tools. They can do much more than provide a convenient means to distribute software, particularly when you use their variable-size variety, the sparse bundle. Although, as I explained just over two weeks ago, there are numerous issues with supporting utilities for APFS-format sparse bundles, they work fine in HFS+, and are even better-supported by C-Command’s wonderful DropDMG.

DropDMG is primarily aimed at those working with disk images for the delivery of apps, and has extensive features for software licence agreements, custom volume icons, background images, and icon layouts. As far as I can see it’s the only utility which does pretty well everything else you might want to do with regular fixed-size disk images and sparse bundles. Most importantly, it’s able to compress them.

DropDMG’s developer Michael Tsai commented to my previous article that “I don’t see much reason to use APFS disk images, except for testing how an app works with APFS. HFS+ sparse bundles are dependable and can be compacted with DropDMG.” This article looks at how well macOS manages space in sparse bundles in HFS+ and APFS: the short answer is badly.

Sparse bundles are one of two varieties of disk images which have the great advantage that they don’t occupy a fixed size, and can grow and shrink as they need. Make an ordinary disk image of 1 GB, and that’s the space (plus a little overhead) that it requires on disk. The two sparse image formats are small when empty, and grow up to the pre-set maximum as they need. They differ in how they store their data: sparse disk images are single files, whereas sparse bundles are really folders, within which the data for the disk image is divided into separate files, or bands.

This should make it easier for a sparse bundle’s size to track that of its contents. When freshly created, an empty APFS sparse bundle which can grow up to 50 GB in size only occupies around 25 MB, using a band size of up to 8.4 MB. In some special circumstances, for Time Machine backups to networked disks, for example, the band size can rise to 268.4 MB to minimise the number of bands require for large backups. (Thanks to Cyclo Pontivy for telling me that.)

Although HFS+ and APFS sparse bundles use the same maximum band size, empty sparse bundles differ substantially in size: for HFS+ this is around 165.3 MB for a sparse bundle which can grow to 50 GB; for APFS this is only 24.8 MB. For an empty sparse bundle, the overhead imposed by APFS is significantly smaller: just six bands, none of which comes close to the 8.4 MB maximum, while the same empty sparse bundle in HFS+ requires 23 bands, most of which are 8.4 MB. Interestingly, DropDMG creates slightly smaller empty sparse bundles than Disk Utility does.

What I then did to my two 50 GB sparse bundles was copy in seven hunky PDFs, totalling a little over 41 MB. Inevitably, both grew in size correspondingly. The APFS version reached 66.8 MB, and HFS+ 206.6 MB, with little change in overhead. Then I turned mean: I trashed all bar one of those files, then compacted the sparse bundle, and repeatedly copied across those six files, trashed them, and compacted again.

What happened was that both sparse bundles grew in size despite being frequently compacted.

Using APFS, the compacted size grew from 26.1 MB (start) to 60.9, 101.7, 107.7, 106.9, and 107 MB. All that disk space is needed to contain a single 817 KB file. Before compacting, the largest bundle size reached 145.9 MB. When no compacting was performed at all, the sparse bundle grew quickly to 234 MB and then more slowly after that, but didn’t appear to reach a stable size.

With HFS+, it grew from 166.2 MB (start) to 172, 172, 204.7, and then fell back to 178.4 MB, again for the same 817 KB file. Before compaction, the largest bundle was 245.2 MB. When no compacting was performed, the sparse bundle remained at a fixed size of 206.7 MB, irrespective of whether it contained the single 817 KB file or 41 MB.

Although APFS grew greater, the end result with each file system was similar, a sparse bundle which was almost entirely filled with the detritus of the file system, and couldn’t be compacted any further. DropDMG’s documentation warns of this, and suggests trying to defragment the file system, which isn’t yet possible with APFS, of course, because of its longstanding lack of documentation. Alternatively, it suggests creating a fresh sparse bundle and copying its contents across, which works excellently.

I’d just like to pause for a moment there and consider that workaround. Imagine any other form of storage which accumulated fragments and other dross to the point where its contents periodically needed to be copied to a freshly-formatted volume before you continued to use it. Because that’s what happens with sparse bundles at present.

Once I had completed this copy-delete-compact cycling, I looked again at the APFS sparse bundle. It had (intentionally) two volumes on it, so I erased them both in Disk Utility, compacted the sparse volume, and returned some of its storage space: the size of the sparse bundle fell from 107 MB to 66.9 MB. So around 40 MB of the lost space had been in the volume and its file system. It was only when I erased the whole container that compaction was able to shrink the size of the sparse bundle back down to its starting point. Indeed, it actually became smaller then, at 21.2 MB, than its original 26.1 MB.

Compacting sparse bundles doesn’t clean up fragments of deleted files, nor does it remove detritus accumulated in the file system, or the APFS container. Sparse bundles can grow up to their maximum size, but are poor at shrinking again when their contents are removed. Although this is more spectacular in APFS, it affects HFS+ too, and overall size required may differ little in practice. The only reliable workaround if you use sparse bundles much is to periodically copy their contents to a fresh bundle.

Whether this, or similar issues in sparse bundle housekeeping, may result in problems using them for networked Time Machine backups remains an open question.

Finally, if you do work with any type of disk image, buy DropDMG. It will save you a great deal of frustration trying to get the tools provided with macOS to work properly.