Disk images may be far from novel, but they can do things that normal storage can’t. A disk image is just a file which can be loaded into memory and function as a whole disk, with volumes, folders and more. They’re commonly used to distribute software, to protect sensitive documents with encryption, and to provide a virtual file system for Time Machine backups on other host systems such as networked storage.

The snag with conventional disk images is that they’re not particularly efficient. Make a 1 GB disk image, and it requires a little more than 1 GB on disk. Far more efficient are sparse versions, the sparse image (a single file) and sparse bundle (a structured folder pretending to be one file). This article looks at sparse bundles in Catalina 10.15.4: their structure, and how to work around the many bugs and incomplete features which might at first deter you from using them at all.

A sparse bundle consists of a folder, the bundle, with the extension .sparsebundle. Inside that is an Info.plist property list, a duplicate copy named Info.bckup, an empty file named token, and a folder named bands which contains the file system and files forming the virtual disk. As a minimum, there are two numbered bands, but these can grow into as many as needed to contain the files stored in the sparse bundle. Each has a maximum size of 8.4 MB; when the sparse bundle needs more storage than can be accommodated within its existing bands, another will be added, up to the maximum capacity of the sparse bundle.

Creation in Disk Utility

There are two methods to create and maintain sparse bundles: using Disk Utility (where its features work), and the command tool hdiutil in Terminal.

Creating a new sparse bundle is well-supported by Disk Utility using the New Image… command in its File menu. The minimum size depends on the file system selected in the Format section of the drop-down sheet: APFS has a minimum size of 8.4 MB, and HFS+ 9.4 MB. Encryption options are set here. Ensure that you select the Image Format as sparse bundle disk image before clicking on the Save button.

Although sparse bundles grow and can shrink in size on disk, the Size chosen here determines their maximum storage capacity in use. This can be changed later by resizing, but it’s best if you ensure that you’ll have ample room at this stage. You should also allow a few MB of free space, as I’ll explain later.

When you save your empty sparse bundle, it has a minimum size of 8.4 MB, which increases slightly with higher maximum size. For example, an empty 1 GB sparse bundle requires 18.8 MB of disk space, which is still a vast improvement on an ordinary disk image.

To use your empty sparse bundle, double-click it to mount, then use it as a normal storage volume. You may experience a strange interface bug in Finder when copying files to it, which can leave unfilled progress circles for some of the files being copied, even though their copying is complete. This has been present since Mojave, I believe, and might mislead you into thinking that the files are still being copied. If you eject the sparse bundle and mount it again, they should vanish.

Don’t trust sizes or free space

One of the strangest and most irritating bugs with sparse bundles in APFS format is the unreliability of their estimates of size and free space. Whether you look at free space shown in the Finder or in Disk Utility, there is no correlation between what is shown and what is actually available for use in the sparse bundle.

For example, the smallest APFS sparse bundle allowed is 8.4 MB, which is stated as having 8 MB of free space. Yet you’ll find it impossible to copy a 7 MB file to that sparse bundle, as you’ll be told that it’s full even when it’s completely empty. macOS simply refuses to let you use the claimed free space on APFS sparse bundles, although HFS+ sparse bundles aren’t as unreliable.

Catalina shows two different types of error when you try to copy what it feels is too much to your sparse bundle. If the sparse bundle should be able to accommodate the file(s) but won’t, a small dialog will appear simply informing you that it’s full although it’s not. Sometimes, the new file is also created in the sparse bundle but abandoned there with zero size.

If you try to copy across files which do exceed the capacity of the sparse volume, a different dialog appears, reporting that there isn’t enough disk space to perform the copy operation.

If you click on the Manage Disk Space button in the hope that this offers a way of resizing the sparse bundle, you’re in for a shock, as it instead opens the storage management options available in the Storage tab of About This Mac, which is both irrelevant and purposeless.

In fact, the only way to address this issue is by resizing the sparse bundle, which can’t even be performed in Disk Utility. I’ll describe later how to do this using Terminal.

Sparse bundles grow but don’t shrink

A sparse bundle is sparse in the sense that its disk storage grows (up to its set limit) to accommodate what you store in it. Common sense would suggest that sparse bundles also shrink as contents are removed from them, but APFS sparse bundles don’t work according to common sense. Even after you have put the entire contents of your sparse bundle in the Trash and emptied it, its size remains unchanged. Neither is there any feature in Disk Utility to shrink a sparse bundle to the size that’s required for its current contents.

The only way to perform this basic housekeeping task is to eject the sparse bundle and resort to Terminal, using the command
hdiutil compact myDocuments.sparsebundle
for example. Output following that should resemble
Starting to compact…
Reclaiming free space…
............................................
Finishing compaction…
Reclaimed 41.5 MB out of 950.3 MB possible.

Compaction can be so successful that it reduces the size of the sparse bundle to smaller than when it was originally created. That sounds like magic, or inefficient use of space in the first place.

Don’t try resizing them in Disk Utility

The maximum size of any sparse bundle is determined when it is created, but can be changed either up or down later when you wish. Apple’s Help for Disk Utility in Catalina claims that this can be performed using Disk Utility, which is true for sparse bundles in HFS+ format, but not for APFS, which consistently returns an error alert claiming that the sparse bundle can’t be resized.

Thankfully, that claim is completely untrue, but has to be performed in Terminal. Why this feature continues to be omitted from Disk Utility I don’t know. Before resizing, ensure the sparse bundle is unmounted then check the limits imposed on it using the command
hdiutil resize -limits myDocuments.sparsebundle
which should return something like
min cur max
16368 16368 18014398509481904

Then enter a command like
hdiutil resize -size 20m myDocuments.sparsebundle
giving the new size in MB (m), GB (g), etc.

When I last looked at this resizing process a year ago using Mojave, this wasn’t sufficient, as resizing failed to expand the container, so a more elaborate procedure also had to be performed in Terminal to rectify that. If you hit that problem, I’ve described how to do this, but it’s no longer necessary in Catalina. At least one bug has been fixed in working with sparse bundles over the past year.

Sparse bundles don’t have ECC

I bring one disappointment, though. In this article, I wondered whether the sparse bundle format used error-correcting code to improve resilience against corruption. Sadly, I can find no evidence of that from looking through the storage bands inside sparse bundles. Instead, they contain a lot of unused space, frequent repetition of similar data structures which might even amount to redundancy, even a hidden EFI volume, which enable them to withstand substantial amounts of corruption across much of their length. But if any of the file system structures within the bands are damaged, the sparse bundle will be irrevocably broken. They’re not smart, just lucky.

The future

Despite their widespread use and advantages, macOS features supporting the use of APFS-format sparse bundles are at present riddled with bugs (I count 6 above) and have serious shortcomings. Their inability to self-compact is a major failing which makes their use clumsy at best, and Disk Utility’s lack of support for basic maintenance functions through the last 2.5 years reflects badly on Apple’s engineering priorities. Disk Utility remains of early beta-test quality and far from complete.