Big Sur, Software Update, Content Caching Server and M1 Macs

Go back a few major versions of macOS, and all that had to happen to install a macOS update could be accomplished in scripts for the Installer app: it was basically a matter of copying a load of files to the System folder, rebuilding some key items such as the pre-linked kernel, containing both the kernel and all extensions to be loaded during startup, and that was it in a nutshell. This was made more complex with the arrival of SIP, when Apple necessarily gave itself a special free pass for its installers.

This became more complex when Apple split the startup volume into System and Data volumes in Catalina. For Big Sur’s new Sealed System Volume (SSV), there’s too much for Installer scripts to handle: building the bootable system now requires the whole System volume to have a Merkle Tree of cryptographic hashes culminating in the hash to rule them all, the Seal, which is compared against Apple’s requirement; then a snapshot is made of that, and the Mac started up from that snapshot.

Full installers are different

These differences are reflected in the full installers for macOS. Look inside one for Catalina 10.15.7, for example, and in the app’s SharedSupport folder you’ll see three Disk Images, each containing various parts of macOS to be installed.

A Big Sur installer comes in only one form, that’s a full installer, as there’s no such beast as a standalone Big Sur updater. What’s inside is also completely different: most of the payload is in a single Zip archive, which contains a whole load of files which appear quite alien. Some are in a format already used in iOS installations, but most are just large blobs of binary.

I have a suspicion that this new arrangement is convenient to Apple for several reasons, among them maintaining the security of the update process, in particular the construction of the Merkle Tree of hashes and sealing the System volume. Currently, once a user has broken the seal on Big Sur’s SSV, there’s no way to reseal it, even if its Seal were to match that prescribed by Apple, which seems highly improbable anyway. Were any user, whether well-intentioned or malicious, able to reseal the system, much of the value of the SSV would be lost. Extracting the tools necessary to perform resealing is currently extremely difficult; standalone delta updates could only increase the chances of that happening.

Big Sur updates

The simple process of downloading a few Installer packages and installing them has been replaced with a series of phases, best seen when updating an M1 Mac. A minimal update, such as that for macOS 11.2.1, is very large, and, if you run a local Content Caching Server and are updating an M1 Mac, requires a combination of fresh downloads and data from your local server’s cache.

The first gigabyte or so of the update has to be downloaded direct from Apple’s servers at [https] mesu.apple.com/assets/macos/ This includes a long list, such as

Embedded Speech support,
Ad Blocker,
Home Kit,
Software Authorization Header,
Mac Update Brain, responsible for running the update,
SFR Software Asset,
Device Check,
Time Zone Update,
Context Kit,
Kext Deny List,
Core Suggestions.

For M1 Macs these are marked out so that they can’t be obtained from the Content Caching Server. The size of this additional directly downloaded part of the update is essentially the same as the difference in total size of macOS updates between Intel and M1 Macs, around 1 GB.

The bulk of the update, over 2 GB of it, can be obtained from the local server, and in the second phase of updating,that’s delivered very quickly indeed, as the progress bar races towards the end.

The third and final phase of obtaining the software update is its preparation, which normally takes 15 minutes. This is run by the Update Brain Service (com.apple.MobileSoftwareUpdate.UpdateBrainService), and includes checking and verification of the entire update to be installed, much in the way this occurs on iOS devices. This has to be performed locally, and isn’t affected by using a local Content Caching Server.

Once that phase has been completed successfully, the update proceeds.

Intel Macs follow a similar course, but when updating from a local Content Caching Server, any direct download from Apple’s servers is very small and not noticeable. They are thus able to update (almost?) entirely from that local cache, and significantly faster than M1 models.

Should you use a Content Caching Server?

If you have two or more Macs which you keep up to date with similar macOS updates, then a local Content Caching Server will save you time updating your Macs. As an example, here are comparable figures for my own situation.

I have four active Macs: two T2 Intel models, and two M1 Macs. Without the Content Caching Server, updating them all from macOS 11.2 to 11.2.1 would have required over 11 GB of direct download from Apple’s servers. Using the Content Caching Server, the total required to be downloaded falls to just over 4 GB: 2.3 GB for one Intel Mac, and 1 GB extra for each of the two M1 Macs. The cost of each additional Mac beyond those four is 0 GB for an Intel Mac, and 1 GB for an M1 model.

You can read more about enabling the Content Caching Server in this article.

Over the coming week or two, I will be looking in more detail at what happens during Big Sur updates.

9Comments

Add yours

1

Simon on February 11, 2021 at 5:50 pm

IMHO the update process has started taking far too much time. Just a simple 11.2.1 update on my high-end 2020 Intel MBP took at least 30 min start to finish.

Our Macs have never been so powerful, yet even minor updates these days seem to be taking longer and longer. Add to the fact that Apple now releases .1 updates to updates only a week after the initial update, and you realize that the time you have to set aside for updates is becoming significant.

Now obviously you could just let your Mac do this while you sleep (assuming it has nothing else it should be doing during that time), but that is tinkering and definitely not something I’d call consumer grade or consider user friendly. Updates need to be better vetted before they leave the door and small delta updates need to become possible again in one way or another. Setting aside one night a week to do updates and download many GBs for that is just not sustainable IMHO.

LikeLiked by 1 person
- 2
  
  hoakley on February 11, 2021 at 6:53 pm
  
  Thank you.
  Are you using a local Content Caching Server? Although the first GB or so does still have to be downloaded direct, the rest comes from your local server, and takes no time at all once you’ve updated the first Mac.
  Although I agree about recent updates, I’d far sooner that Apple provided the updates as soon as it can, rather than waiting a few weeks to make a more substantial update. The sudo fix clearly couldn’t have been ready for 11.2, and waiting until 11.3 would have been unacceptable.
  I don’t think that small delta updates will ever return, because of the SSV and the rigmarole required to maintain its integrity.
  Howard.
  
  LikeLike
3

Phil on February 12, 2021 at 12:16 am

Thank you Howard for this informative article! I look forward to the followup articles you’ve planned.

LikeLiked by 1 person
- 4
  
  hoakley on February 12, 2021 at 12:54 pm
  
  Thank you.
  Howard.
  
  LikeLike
5

Chris Eldred on February 12, 2021 at 12:45 am

Is it possible that Apple is still working on their workflow to be able to produce incremental updates? Or does the crypto involved actually make increments impossible?

Also, I understand that that the sealed system is important for security, but how secure relatively speaking? What kind of exploits are eliminated? I’m seeing info here and elsewhere about what a pain it is, but I don’t have a sense of the benefits.

LikeLiked by 1 person
- 6
  
  hoakley on February 12, 2021 at 1:00 pm
  
  Thank you.
  My understanding of the situation is that Apple hadn’t made any decision about whether to product standalone updaters. As no decision had been made, and that is a task requiring considerable engineering effort, they simply aren’t being provided. If sufficient users were to make a strong enough case, that could persuade Apple to make a positive decision. Otherwise I suspect it will just be left and never done.
  Catalina’s system protection is good. But all (sic) and attacker has to do is find a way to mount the System volume read-write and bypass SIP, and they can do whatever they like with the files on that volume. It’s only a matter of time, I fear, before that happens.
  Big Sur’s protection is definitely a big step forward – not least because, even if it didn’t make attacking the System far harder, the whole system now has integrity checking. About time too, IMHO. But that’s only worth doing if malware can’t alter the hashes, hence all the fiddling around.
  Howard.
  
  LikeLike
7

Michael on February 13, 2021 at 4:42 pm

I’m really curious on Apple’s decision process on building the SSV Merkle Tree on device, it seems like a massive waste of compute resources.

On modern Android devices using dm-verify, the hash trees of system partitions are pre-generated at build time, updates that modify mountable filesystems are block-based which means they only contain the exact differences between the two OS versions. When an Android device installs a software update, it doesn’t have to concern about rebuilding the filesystems hash trees since the updated hashes are already included in the delta images.

https://source.android.com/security/verifiedboot/dm-verity#implementation
https://source.android.com/devices/tech/ota/nonab/block#File%20vs.%20Block%20OTAs

LikeLiked by 1 person
- 8
  
  hoakley on February 13, 2021 at 10:19 pm
  
  Thank you.
  I don’t know the Merkle Tree is built: that is something that Apple keeps close to its chest.
  However, I can tell you from my own integrity-checking software that checking an existing hash and computing a new one take almost identical times. So the only way that you could save time would be not to check the hashes at all, which then means that you don’t check integrity during installation, which is asking for trouble.
  Not many users run Android on computers with the power of Intel i7/i9 or the M1, so clearly compromises have to be made for mobile devices which you don’t need to accept on a modern Mac.
  Howard.
  
  LikeLike
9

Michael Tsai - Blog - Larger and Slower Updates With Big Sur on March 15, 2021 at 8:24 pm

[…] Howard Oakley: […]

LikeLike

Share this:

Related