APFS and macOS 10.13: many apps and tools will need to be revised

Apple hasn’t yet stated that the next major release of macOS, due to be announced in less than two months at WWDC, and to ship this autumn, will require you to convert to using its new file system, APFS, but that is expected – just as iOS 10.3 requires its own variant of APFS.

APFS now exists in two variants: one which, like the current HFS+, is case-insensitive, and the other which is case-sensitive. There is a variant of HFS+ which features case-sensitivity, but it remains (after several years) so incompatible with many macOS apps that very few ever consider using it.

I have already posted my experiences with using the case-sensitive variant of APFS here. Suffice it to say that, even before you worry about how your apps would cope with it, it is chaotic. As you will see from comments to that, others are confident that Apple will not make that variant the standard for macOS 10.13, as it has already with iOS 10.3. But iOS is a very different situation, and its case-sensitivity is nothing new, as it has always used case-sensitive HFS+ anyway.

The question this article tries to answer is whether using case-insensitive APFS for macOS 10.13 solves the problems with the case-sensitive variant, or whether there will still be significant problems in migrating from the current HFS+. The TL;DR is that both variants of APFS will cause problems – they are just different problems requiring different solutions. Either way, many current apps, tools, and scripts will perform strangely when run on APFS, and many will therefore need to be revised and updated to cope with it.

What is different?

To understand the differences between the two variants of APFS, it is important to understand not case-sensitivity, but how APFS handles the Unicode strings which make up file and folder names.

In HFS+, as with several other file systems, all file and folder names are converted to a normalised form before use. This is one solution to dealing with Unicode encodings which look identical but which use different Unicode characters. If you’re still not clear about this, please read one of my previous postings and explore this using Apfelstrudel, available from the Downloads item above.

This is not the only solution, though. Another major modern file system, ZFS, which was once considered for Macs, does not perform normalisation except where it really has to. This has been termed normalisation-insensitive, as if it were like case-insensitivity, but it is actually quite different. ZFS does still normalise, but only where it is required, and not all the time.

Although ZFS is generally recognised as being an excellent modern file system, it is not deployed across tens of millions of user systems, and is designed primarily for server and similar systems. Current experience with ZFS is thus very different from that with, say, HFS+.

Both variants of APFS perform no normalisation of file or folder names. However, Apple has decided to make the case-insensitive variant ‘normalisation-insensitive’ too.

Case-insensitive APFS

So in the case-insensitive variant, whatever Unicode characters are supplied as a file or folder name will be used. Where it differs from the other variant is that it does not permit the presence in a single folder of two files or folders which differ only in the normalisation of their characters. So just as it won’t allow Cafe and cafe (case-insensitivity), it also won’t allow café and café (normalisation-insensitivity).

Currently, case and normalisation in APFS are determined when a file is created. Furthermore, once created, you cannot change the case or normalisation of any of the characters in a file or folder name, in the Finder or at the command line.

This creates some bizarre effects: on HFS+, you can happily change Cafe to cafe in either, although as it normalises, changing between the accented forms makes no difference to the name. Once you have created a file named café.txt, in the current case-insensitive variant of APFS, you cannot change its name to Café.txt or to café.txt, but you can change its name to cofé.txt.

apfsvol08

If you try changing the case or normalisation in the Finder, you are told that the name cannot be used, although (in the case of normalisation) there is no visible difference between the old and new names. In Terminal, you are told instead
mv: rename scafé copy.txt to Scafé copy.txt: Invalid argument
Apple may decide to change this behaviour, and certainly needs to make the error messages more informative and less puzzling.

apfsvol07

One way that you can change the normalisation of a file or folder name is to copy it to and back from HFS+, if the original name is non-normalised. As HFS+ will normalise the name, the name of the file which you then copy back will be different, and APFS then claims the file is different, although the two names are of course identical to the user, and their contents are unchanged. That is further bizarre behaviour.

Because HFS+ normalises to Form D, and Linux to Form C, case-insensitive APFS is actually more compatible when working with Linux than with HFS+. This is because most file and folder names with problem characters are entered from the keyboard, which almost universally generates Form C characters and not Form D ones. For those maintaining HFS+ external drives alongside case-insensitive APFS, there are likely to be many bizarre phenomena, most of which users will not immediately associate with differences in normalisation.

String operations

Generally speaking, though, the difficulties in using case-insensitive APFS are significantly less than with its case-sensitive variant, at least until you come to operations involving file and folder names as strings – particularly compare, search, and sort (CSS).

I have just made a new beta version of Apfelstrudel which displays the results of a range of CSS operations on the Form C and Form D versions of entered strings, and the results are very worrying. For examples in which the Form C and Form D encodings differ, different CSS operations behave very differently. For example, using Swift 3.1 on macOS 10.12.4:

  • Swift String == comparison returns that Form C and Form D are equal;
  • NSString isEqual() returns that Form C and Form D are not equal;
  • NSString localizedStandardCompare(), which is Apple’s recommended call when the strings are visible to the user, returns that Form C and Form D are not equal;
  • NSString compare(), which Apple discourages for strings visible to the user, returns that Form C and Form D are equal when compared on a caseInsensitive basis;
  • NSString contains() reports that Form C contains Form D;
  • NSString localizedStandardContains() also reports that Form C contains Form D.

apfsvol09

Oddly, there is as yet no option to any of these to make comparisons on a normalisation-insensitive basis. Furthermore, Apple does not disclose whether, when localisation is taken into account, that is affected by normalisation forms, changes according to localisation, or what.

So any apps which use CSS operations on file or folder names may exhibit behaviour which the user does not expect them to, depending on how they handle and perform operations on strings with mixed normalisation. For the many apps which delegate the handling of files, etc., to high-level calls, as Apple advises, this will have no effect. Once an app starts handling filenames and paths as strings, when it may perform CSS operations, the results on case-insensitive APFS may become unpredictable, according to which calls they use. If they use their own code to perform CSS, then that could well break altogether.

Testing out existing features in macOS Sierra 10.12.4 shows that it does not yet cope with mixed normalisation as occurs in either variant of APFS: Apple has still to revise and correct its own tools such as Spotlight and the Finder’s Find command.

Problems with CSS features are likely to be worst in command tools, particularly those which are ported from C source code and that intended to be relatively platform-independent, as they are least likely to use high-level calls to macOS, and most likely to use their own CSS routines, which may well break on APFS. This is likely to result in some command scripts breaking, and may affect some scripting languages particularly badly. As APFS cannot yet used for macOS startup volumes, a lot of these issues will not yet have come to light.

Users will also discover some strange consequences. For example, piping a folder listing to a text file and using that in a text editor has serious consequences on the effectiveness of CSS operations. The text editor does not known whether it is here dealing with filenames, nor that they have mixed normalisation. Some of the better text editors might need to incorporate normalisation-insensitive options for their CSS operations to help users cope.

Conclusions

Using the case-insensitive and normalisation-insensitive variant of APFS is not as chaotic and dysfunctional as the case-sensitive variant. For most English-based Mac users, the differences from HFS+ may be minimal, assuming that Apple fixes the problem preventing users from changing the case of characters in file and folder names.

For those using accented characters and non-Roman scripts, there are significant effects which will trip users and their apps up. The greater the use of Unicode characters whose Form C and D normalisations differ, the more severe these effects will be. They will not, in the main, cause crashes or severely limit functionality, except perhaps in a small minority of apps. However, their effects will be troublesome, pervasive, and hard to diagnose and fix.

Many Mac apps will need to be revised and updated to work better with case-insensitive APFS, as will many command-line tools, and all Mac developers should start checking their source code now to address these issues. The initial developer release of macOS 10.13 is likely to be less than two months away, and final release may be only six months into the future.

When macOS 10.13 is released (and during any public beta phase), users of languages which include accented characters or non-Roman scripts would be well-advised to delay upgrading until they can be confident that there are no significant problems with their language(s) and key apps. Some apps and tools could continue to suffer local dysfunctionality for many months or years after the release of 10.13.