How plain disk images went sparse in Monterey

Conventional wisdom, aided by the lengthy documentation for hdiutil and Apple’s help page for Disk Utility, is that regular read/write disk images (UDRW format) are of fixed size, and neither grow nor shrink. For instance, Apple states: “You can create a disk image that includes the data and free space on a physical disk or connected device, such as a USB device. For example, if a USB device or volume is 80GB with 10GB of data, the disk image will be 80GB in size and include data and free space.”

This is summarised well in C-Command’s documentation for DropDMG: “.dmg — constant file size. The disk image is stored as a single file. This is less efficient than a sparse disk image because the file size is determined by the capacity of the disk image rather than by the amount of data currently stored on it. This can be desirable, for example if you want to reserve a certain amount of space for the disk image’s use.”

What if I were to tell you that a UDRW disk image is really sparse storage that works more efficiently than regular sparse disk images (UDSP format) or sparse bundles (UDSB format)? All you need to know is how to unlock that potential.

Creation and use

Open Disk Utility, and with its File/New Image/Blank Image… menu command create yourself an empty read/write disk image in APFS format of substantial size, anything between 5-100 GB.

diskimage90

Disk Utility takes a few moments to create that, then automatically mounts it. Once complete, unmount that disk image, select it in the Finder and Get Info.

diskimage91

As expected, your disk image is a regular flat file, in this case of 100 GB and takes up 100 GB on disk. Now double-click the disk image to mount it again. Once it has mounted, verify that it appears empty (apart from its hidden .fseventsd folder). Now unmount it again and Get Info.

diskimage92

Although the Size given for that disk image remains at 100 GB, it now takes much less space on disk, here a mere 334 MB. It has become an APFS sparse file, as you can verify using either of my free utilities Precize or Sparsity. When you write files to that disk image, it will grow in size to accommodate them, and when they’re deleted it will shrink again.

diskimage93

Here it is with over 53 GB of files from a disk performance test. Delete those and the space taken on disk will drop back to 334 MB without your having to compact it or do anything else by way of housekeeping.

This also works for some other variants of UDRW format. You can opt for the disk image to be encrypted, and for its internal file system to be APFS or HFS+J, but not ExFAT, which doesn’t result in conversion to an APFS sparse file.

There is disappointment, though, for those who prefer using the command line. While unmounting and mounting the disk image in the Finder brings about its conversion to an APFS sparse file, the hdiutil attach command doesn’t appear to have the same effect, and leaves the disk image as a flat file.

Performance

One longstanding problem with most disk image formats is their poor performance when writing to them. Interestingly, when I last looked at that, there were anomalies depending on whether the image had been mounted before, which probably correlate with their conversion from flat to sparse file.

When measured on a Thunderbolt 3 external SSD normally delivering around 3 GB/s read and write speeds to a regular APFS volume, read speeds for these sparse UDRW disk images are close to native at 2.8 GB/s, and write speeds are about a third of native at 1.0 GB/s. Those are essentially the same for APFS on both unencrypted and encrypted (128-bit) disk images, and for HFS+J as the file system in the image.

The most significant difference here is performance with an encrypted disk image, which in previous testing could fall below 0.1 GB/s even on the faster internal SSD of a Mac Studio. Whether or not you see this sparse version of UDRW disk images as advantageous, its beneficial effect on the write performance of encrypted disk images makes it important to unmount and remount encrypted disk images before writing to them for the first time.

Disadvantages

Provided they remain in APFS sparse file format, plain UDRW read-write disk images now work better and more efficiently than either UDSP sparse images or UDSB sparse bundles, on macOS Monterey and Ventura. They don’t need to be compacted to recover storage space, and macOS does all the work for you.

However, APFS sparse files have to be handled carefully to ensure that they remain in sparse format, and don’t explode to full size. I’ve previously looked at the precautions needed, summarised as follows:

  • access them only in macOS 12 or later;
  • store them only on APFS volumes;
  • transfer them over a network using macOS file sharing;
  • never use AirDrop, and avoid iCloud;
  • compress them inside a folder using Apple Archive, or another protection such as tar;
  • back them up using Time Machine to APFS.

This specifically rules them out as a container for backups on NAS.

Why the change?

It’s mystifying that Apple doesn’t appear to have recorded this change in man hdiutil or other documentation, as far as I can see. I wonder, though, whether the primary reason for this change was to support disk images used in lightweight virtualisation, which I’ll explore in an article here tomorrow, where I’ll also examine how they’re transformed from flat into sparse files.

I’m very grateful to Maurizio for being the first person to draw attention to this change in macOS, and to Tim R for reporting his observations on HFS+ and ExFAT.