Last Week on My Mac: How APFS trims a disk image to size

I’m a fan of sparse files in APFS, and was excited to discover how they’re now used in virtual machines (VMs) for macOS lightweight virtualisation, but hadn’t expected to find them in regular user disk images. Although those are both variants of the same type of file, and ideal candidates for sparse format, they’re created quite differently, as I’ll explain.

The rules for creating sparse files in APFS are strict: if an app writes 10 GB of blank data to a file, followed by 1 KB of ‘real’ data, then that file can’t be sparse. Instead, the app has to create the file, then skip forward or ‘seek’ between its blocks of real data. That skipped space isn’t written to storage, only the real data. Thus the total file size might be 10 GB, but that required to store the file’s data could be the minimum of 4 KB. Apple puts it succinctly as “on-disk blocks are allocated only when those blocks are actually written to”.

When creating the boot disk image for a VM, the rule for sparse files is followed, as an empty file is created by skipping to its end. macOS later writes the file system and boot disk contents into that empty sparse file, causing it to grow from nothing to occupy just the storage required for those contents.

Creating a UDIF read/write (UDRW) disk image is quite different. Make an empty one, unmount it, and it really does occupy the whole of that empty space in storage. Neither is it marked as a sparse file by macOS. This matches the description usually given of this type of disk image, that it contains the files transferred to it, and any free space up to the total of its fixed size. That’s just what you get when you first unmount a UDRW image: even if it’s empty, its size and the space it occupies on disk are the same. You may also notice that, the larger the size of UDRW image you create in Disk Utility, the longer it takes to create it on disk.

It’s only when you mount that UDRW disk image for a second time that it’s transformed into a sparse file, as a result of the mounting of its file system. With that, the unused blocks in the mounted image are recognised, and converted to skipped space. This doesn’t happen during its initial mount, because this ‘trim’ feature is only triggered automatically when its file system is mounted.

If Maurizio, who first reported this in UDRW images, is correct in his observation that this conversion first appeared in beta-releases of Monterey, it seems to be a feature introduced in APFS version 1933.x. As a behaviour of APFS rather than UDRW disk images as such, that may explain why the normally reliable and detailed account given in man hdiutil doesn’t mention this change, nor its significant consequences to UDRW images.

The evidence for this appears in the mount sequence in the log, which begins with its preliminaries (times given in decimal seconds from an arbitrary start):
2.015645 dev_init:299: disk6 device_handle block size 4096 block count 1169499 features 16 external
2.015826 nx_mount:1224: disk6 initializing cache w/hash_size 8192 and cache size 32768
2.023228 nx_checkpoint_find_valid_checkpoint:565: disk6 sanity checking all recently-changed container state... please be patient.
2.023567 nx_mount:1553: disk6 checkpoint search: largest xid 1, best xid 1 @ 1
2.023574 nx_mount:1580: disk6 stable checkpoint indices: desc 0 data 0

That’s followed by Spaceman, the APFS Space Manager, scanning for free blocks, and trimming them:
2.023580 spaceman_metazone_init:110: disk6 no metazone for device 0, of size 4790267904 bytes, block_size 4096
2.023783 spaceman_scan_free_blocks:3311: disk6 scan took 0.000179 s (no trims)
2.023813 apfs_newfs:30416: disk6s1 FS will NOT be encrypted.
2.024045 spaceman_scan_free_blocks:3293: disk6 scan took 0.000256 s, trims took 0.000035 s
2.024048 spaceman_scan_free_blocks:3295: disk6 1164953 blocks free in 1 extents
2.024050 spaceman_scan_free_blocks:3303: disk6 1164953 blocks trimmed in 1 extents (35 us/trim, 28571 trims/s)
2.024052 spaceman_scan_free_blocks:3306: disk6 trim distribution 1:0 2+:0 4+:0 16+:0 64+:0 256+:1

In this case, with an empty UDRW disk image, a single extent (contiguous area of storage) of 1164953 blocks of 4 KB each was trimmed, shrinking the storage required by the disk image by 4.77 GB. There’s only one instance of Spaceman for each APFS container; disk6 is here the container for disk6s1, the APFS volume in the disk image.

This feature is limited to disk images containing APFS and HFS+ file systems, as they have trim support, unlike FAT or ExFAT. That in itself can appear confusing, because HFS+ as a file system doesn’t support sparse images, although here a disk image containing an HFS+ file system enjoys the same preferential treatment, and is converted into a sparse file when hosted on APFS. Try this with a UDRW image stored on HFS+, and no change in file format can take place.

Sadly, as few other types of file contain mountable file systems, this isn’t going to magically convert other files into sparse format. Apple warns us that “you can’t use FileHandle to create a sparse file from an existing file that has blank data already stored on disk,” but this demonstrates that APFS can convert some flat files into sparse format after all.

I’m very grateful to Maurizio and Thomas Tempelmann for their observations and discussion.