When we were told that High Sierra was going to fix bugs and introduce the new APFS file system, none of us could have expected what has happened since its release on 25 September. Yes, there have been some significant improvements in its bundled apps like Photos, and some bugs – probably including the scheduling of Time Machine backups – appear to have been fixed.

What has been most prominent about High Sierra, even reaching the general news media, have been some glaringly obvious bugs in its security. Last week’s crisis over the root user vulnerability was the most serious, as it affected every single Mac running any version of High Sierra. In the end, when Apple was shamed in a tweet, its engineers responded magnificently, and all was patched in less than twenty-four hours.

Commentators are right to question Apple’s quality management, as I have done previously, but the buck shouldn’t stop there. There’s more to High Sierra’s problems than just inadequate checking of code and flimsy pre-release testing. We need to drill down beyond these proximal causes if we are to understand what is going wrong.

For me, the most striking observation about last week’s vulnerability, and the previous password hint bug which made some encryption near-useless, is that these bugs occurred in code which had previously been working well in Sierra and earlier. The same is true of the bugs in High Sierra’s new version of Disk Utility, which I and many others have been complaining about.

These are indicators that Apple is in the process of rewriting a lot of macOS. In the case of the root user vulnerability, this was in Open Directory, whose source code may well date back ten years or more. It has been my contention that macOS-only code, such as Time Machine, gets considerably less resources than that which is shared with iOS, but that doesn’t explain why, if such macOS development is being poorly resourced, Apple’s precious engineers are busy rewriting systems such as Open Directory, when there are so many other demands.

It also makes sense that, when there is a lot of code development taking place, code checking and testing tends to be rushed, and serious bugs like these can slip through into release. Apple must surely have proven procedures in place, so here the pace of development seems to have exceeded the capacity for checking and testing.

One simple reflection of what is happening in Apple’s software engineering is the change in version number for Open Directory, which leaped from 417.50.4 in 10.12.6 (released 19 July) to the current 483.20.7 in 10.13.1 updated (on 29 November). If Apple has numbered versions consecutively, that is 66 major version increments in four months; even if High Sierra started at 450.0.0 it still appears a prodigious pace.

Other clues come from what is happening in Apple’s use of Swift. Alexandre Colucci, in his Timac blog, tracks the use of Swift in iOS and macOS, and has found its use has grown from a mere 10 binaries in macOS 10.12.1 (released 24 October 2016) to 23 in macOS 10.13.1 (31 October 2017). Many of these are components which don’t appear to have undergone major external change.

We have been complaining about Apple’s apparent under-investment in macOS (and Mac hardware), when perhaps it has been investing more heavily than ever before. When we were promised that High Sierra would ‘fix bugs’, that may have been code for ‘our engineers are progressively re-writing much of macOS’, although not necessarily entirely in Swift, just yet.

Thus far, I think that we have evidence of what is happening, but from here we can only speculate as to why Apple is doing this now. Rewriting much of such a huge and protean operating system is not something that companies undertake lightly: there must be a compelling reason why this is occurring now.

Two good reasons for this sort of effort are maintainability, and hardware change.

Many years ago, when I was assessing its suitability for use by my team, I persuaded the author of a large and very sophisticated mathematical model to provide me with its source code. It took but a few minutes to discover that it was written in spaghetti Fortran, completely unstructured, and that it would actually have been quicker and cheaper to have re-written it from scratch than trying to use its source.

It is quite feasible that many of the older nooks and crannies in macOS have become similarly unmaintainable, although they would normally be revamped as part of a long-term rolling programme, and not all at once in a rush.

There may be one clue here in Apple’s sudden interest in its own EFI firmware, another new engineering task in High Sierra. Maybe the deadline which is driving all this activity has already been set in the introduction of new hardware next year.

I’d hazard a guess that those new systems include custom systems-on-a-chip (SoC), as are already used in the latest MacBook Pro models, and are rumoured for other imminent products. Once you need to move your code onto auxiliary processor systems, it’s a great help when it is modern, structured, and written in a language like Swift.

Apple would clearly never have wanted the embarrassment of last week’s root user vulnerability. Maybe in the long run it will prove a small price to pay for its ultimate goal in 2018.