Snug sparse bundles work better, but…

Following my look yesterday at wasted space in sparse bundles, NSSynapse kindly commented that I should have looked at “less extreme scenarios”, and proposed that rather than examining the performance of largely empty 50 GB sparse bundles, I should look instead at 3 GB of files in a 5 GB sparse bundle. The good news is that does work better, and in that case APFS appears to outperform HFS+. There is an important caveat, though, which I’ll explain after giving the performance figures.

Using the HFS+ file system, a freshly created and empty 5 GB sparse bundle is 165.4 MB. As before I then added a placeholder file, a PDF of just under 1 MB, which didn’t alter the reported bundle size. I then cycled through adding 3.1 GB of files, bringing the bundle up to 3.24 GB in size, then trashing those, which left the bundle at the same 3.24 GB, compacting it down to 170.4 MB, a small loss of 5 MB, and repeating.

There was no sign of any persistent leak, although compacting the bundle was required to recover its size. Functionally, this may not be important, as the size of a sparse bundle isn’t an indicator of how much free space it has. The only way to discover how much of it remains free is to mount the sparse bundle and then check its free space. This 5 GB sparse bundle could be filled almost to capacity, occupying nearly 5 GB on disk, but so long as it had sufficient free space (by trashing the files in it), more files could be copied to it without error.

Using the APFS file system, results were even better: the fresh empty 5 GB sparse bundle was only 20.5 MB in size, and remained at that after copying the 1 MB placeholder file. When 3.1 GB of files were added, it occupied just 3.1 GB on disk, and remained at that size after all bar one of those was trashed. Compacting that reduced the file size to smaller than the original, at 16 MB, although in subsequent cycles that rose to 19 MB. Like its HFS+ version, it coped perfectly well with additional files so long as it had sufficient free space when mounted.

It therefore looks as if the best way to use sparse bundles is to make them ‘snug’, with a maximum size close to the intended maximum total size of files to be stored on them. Of the two formats, APFS appears to be the more efficient in terms of storage overhead.

There is a catch, though, with adopting that strategy. If you get the initial sizing wrong and want to increase the maximum size of your sparse bundle, you will have to use Terminal to do so, as this feature in Disk Utility remains broken, as it has been since Mojave (at least). If you decide that you need to increase the maximum size of your 5 GB sparse bundle to 10 GB, Disk Utility will simply return an error, and it is possibly the only feature which isn’t supported in DropDMG.

sparsebundle06

Before resizing, ensure the sparse bundle is unmounted, then check the limits imposed on it using the command
hdiutil resize -limits myDocuments.sparsebundle
which should return something like
min cur max
16368 16368 18014398509481904

Then enter a command like
hdiutil resize -size 10g myDocuments.sparsebundle
giving the new size in MB (m), GB (g), etc.

When using APFS in Mojave, that may not be sufficient, as resizing might not expand the container within the sparse bundle, so a more elaborate procedure also has to be performed in Terminal to rectify that. If you hit that problem, I’ve described how to do this, but thankfully it’s no longer necessary in Catalina.

My strategy in creating sparse bundles has been to set a maximum size which comfortably exceeds the greatest I’m likely to need, so as to avoid the pain of having to resize the sparse bundle. Playing safe like that may result in the issues which I described here yesterday, and finding the best compromise doesn’t seem straightforward. For a more flexible variety of disk image, that is frustratingly restrictive.