hoakley May 29, 2019 Macs, Technology

Installing kernel extensions in Mojave 10.14.5

As the dust settles on the recent Mojave 10.14.5 update, I’ve been looking at its undocumented change in the way that it handles kernel extensions. This article examines how this could trip you up, as it already has done for many users who tried to install Oracle’s VirtualBox 6.0.8.

At last year’s WWDC, almost exactly a year ago, Apple told developers that it was introducing a new system in which they could submit apps to Apple’s Notary Service, which would check and approve them, certifying them as ‘notarized’. For the time being, this was to be an option, but in a future release of macOS, it would become mandatory for all Mac software distributed over the Internet but outside the App Store. Everyone assumed that meant macOS 10.15, due to be detailed in a few days at this year’s WWDC, and expected to ship this autumn/fall.

Although notarization is free, take-up by developers seems to have been quite slow. Then early in the beta-releases of macOS 10.14.5, back in April, Apple suddenly warned developers that notarization would be compulsory in 10.14.5 for two classes of software:

for all new and updated kernel extensions signed from 7 April;
for all software signed by new developers, i.e. developers who hadn’t signed products before that same date.

When Apple released the 10.14.5 update, there was no mention of any of this in its release notes, nor can I find any advice it has provided to users about these changes. But the day after that update was released, Oracle released a new version of VirtualBox, its popular virtualisation environment. Those trying to install VirtualBox 6.0.8 quickly ran into trouble, reporting that it failed when the Mac had been updated to 10.14.5.

A workaround was quickly posted to user forums, to restart in Recovery mode and enter the following code in Terminal there:
spctl kext-consent add VB5E2TV963
The last group of characters is Oracle’s Developer ID.

Following restarting normally, most users found that VirtualBox 6.0.8 then installed correctly, and could be used.

My assumption was that the kernel extensions shipped in VirtualBox 6.0.8 had not been notarized, and it was the first casualty of this change in Mojave’s security policy. On closer investigation, though, it isn’t as simple as that. I have successfully installed VirtualBox 6.0.8 on macOS 10.14.5 without resorting to Recovery mode, but its kernel extensions don’t appear to have been notarized at all, despite being signed after Apple’s deadline.

This calls into question whether 10.14.5 does enforce notarization on new and updated kernel extensions at all. Perhaps Apple hasn’t informed us as this change has been delayed until 10.14.6 or even 10.15.

Examining the VirtualBox Installer package, it does contain kernel extensions which it requires in order to run. These are easy to find: when you first open the package in Installer, list its contents, and you should here see four.

Extracting one of them using Suspicious Package, it’s clear from the contents that there is no notarization ticket stapled to (inside) the .kext folder. Running the ultimate check of
xcrun stapler validate kextname.kext
confirms that they aren’t notarized, although each was signed on 13 May 2019, more than a month after Apple’s original deadline.

When I ran the Installer package here, on my iMac Pro and macOS 10.14.5, it failed the first time, generating the normal dialog which directed me to open the Security & Privacy pane to authorise installation of the extensions.

This I did, then ran the package again, which was successful in completing this time.

If you want to install a package which might contain kernel extensions in macOS 10.14.5, I recommend that you:

Check at the start of the installation whether its list of contents does include kernel extensions.
If it does, proceed with caution.
If the installation fails or reports an error, look for the dialog inviting you to authorise installation of the kernel extensions, and use it to open the Security & Privacy pane. There, give approval for those extensions to be installed.
After that, install the package a second time. If that’s successful, the job’s done.
If that second installation fails too, discover the developer’s Developer ID, using WhatsYourSign or a similar utility.
Restart in Recovery mode, and there substitute that ID in the Terminal command given above, to add KEXT consent.
Finally restart in normal mode and install the package again.
If you still can’t complete the installation correctly, contact the developer.

I hope that prevents you from suffering the problems experienced by those who updated to 10.14.5, then found that they couldn’t install the shiny new VirtualBox 6.0.8.

13Comments

Add yours

1

KextMonster on May 29, 2019 at 7:40 am

Something must have changed recently to make this work without rebooting in recovery mode, because it definitively required recovery mode and “spctl kext-consent” previously. On the virtualbox bug tracker, an announcement on May 23 indicates “something” has been done to fix this problem, but it does not say what or how. Was the virtualbox .dmg installer silently updated, or did something change on Apple’s end (whitelist update, online notarization without having to include a ticket in the .dmg installer?). In case the .dmg has not been updated, I’m really curious to hear about how this was “fixed effective immediately”…? https://www.virtualbox.org/ticket/18645#comment:17

LikeLiked by 1 person
- 2
  
  hoakley on May 29, 2019 at 7:57 am
  
  The simple answer is that we don’t know.
  There’s no evidence that the current 6.0.8 is any different to that supplied originally. The signatures on the KEXTs, and the KEXTs themselves, date from 13 May, and the KEXTs aren’t notarized at all, according to the standard test method, and their structure.
  Apple hasn’t pushed any update to its whitelist since the 10.14.5 update – it’s not one which is pushed as a standalone update either, but comes with a macOS update. Indeed, since the 10.14.5 update there have been no changes in macOS system files, apart from a few MacBook Pro 2018 models requiring an additional EFI firmware update.
  So as far as I can tell, nothing has changed since the 6.0.8 update was first released. And there’s no evidence here that 10.14.5 does require new KEXTs to be notarized, although there is a developer release note for 10.14.5 which refers to a bug in such checking. There’s another twist to this in that, if this was a problem over the KEXTs not being notarized, then the user should surely not have been able to bypass that requirement using a single command in Recovery mode. Worse, if what I did was to bypass the KEXT notarization requirement, all I had to do was the normal consent in the Security & Privacy pane. So no, I don’t believe that macOS was or is enforcing any notarization requirement on those KEXTs.
  I suspect that only Apple knows what has actually happened, and isn’t telling anyone, as usual.
  Howard.
  
  LikeLike
  - 3
    
    KextMonster on May 29, 2019 at 8:07 am
    
    The same “effective immediately (2019-05-23 18:00 UTC)” comment was also posted by an Oracle representative on https://forums.virtualbox.org/viewtopic.php?f=8&t=93151&start=30#p449315 and I think this strongly indicates that the Oracle Virtualbox team must have been in contact with Apple to somehow “fix” this problem through some undisclosed online macOS system mechanism, since the .dmg apparently wasn’t updated? How else can the same .dmg behave differently before and after May23?
    
    LikeLiked by 1 person
    - 4
      
      hoakley on May 29, 2019 at 8:18 am
      
      Maybe the certificate used to sign the KEXT hadn’t been registered by Apple as being good for KEXTs, and now Apple has enabled that?
      I just checked to see whether any of the installation scripts might have been altered, but the datestamp on the Installer package is 13 May, and there’s no suggestion in the scripts themselves.
      This looks like a change made at Apple’s end to enable online checks during the KEXT installation process.
      Howard.
      
      LikeLike
5

Randy Saldinger (Mothers Ruin Software) on May 29, 2019 at 4:55 pm

I don’t have any insight into the kext rules or this particular case, but I think you might be conflating notarization and “stapling.”

As I understand it, notarization means that Apple checked the app/bundle/code-signed-thing, decided it was benign, and registered its cryptographic digest as approved. (The digest is usually the “CDHash” as reported by “codesign -d -v -v -v” on the app/bundle/whatever.) When macOS decides to check that thing for notarization (subject to quarantine or whatever the rules might be), its canonical test is to query an Apple server to see if that CDHash is registered as such.

Stapling seems to be only an optimization, which allows the network query to be avoided, by attaching a cryptographically-signed version of the same information which macOS would otherwise get from an Apple server. It’s possible for a thing to be notarized even if it isn’t stapled, and macOS can indeed recognize this.

This means that “stapler validate” is not really a reliable test for notarization. Much better is to use spctl as:
spctl –assess –type execute -v path-to-app-bundle-or-whatever
This will return
source=Notarized Developer ID
if the thing is notarized (even if it was never stapled!), but if not notarized (but is otherwise validly Developer ID-signed), will be simply:
source=Developer ID
This is the closest thing to an API I’ve found regarding notarization status.

Also note that the only thing that is typically stapled is the container that the software comes in. That is usually a dmg (which was the thing uploaded to Apple for notarization in the first place). Of course, Apple’s notarization service opened up that dmg, and checked everything inside. And if you examine the notarization log, you’ll see that it has all of the found bundles and frameworks and such, each with their CDHash, which presumably indicates that those are all considered notarized now. The dmg itself also has a CDHash, and is also registered as notarized (though I’ve no clue if macOS ever queries for notarization at that level).

Along the same lines, you can notarize an installer package (oh, that’s why I’ve learned too much about this :-). The package has the equivalent of a CDHash, which is the checksum of the xar TOC, retrievable using “xar –dump-toc-cksum -f the-package”. Apple apparently expands the package similarly to a dmg and checks/notarizes the contents (the notarization log bears this out), and then also notarizes the package by its digest. You can (but don’t have to!) staple the package after that. (I know; I notarized an old version of Suspicious Package that was delivered in a package, after which the original, not-touched-in-years package file was now recognized as notarized.) If and when the macOS Installer checks the notarization status, I don’t know; but given that the installer does not add quarantine attributes, I’m assuming that it must check notarization on the package.

Speaking of which, you can check the notarization status of a container (dmg or package), without regard to stapling, by using the spctl command above, but with a “–type install” instead of execute. I’ve tried this on notarized-but-unstapled packages and dmgs, and it does indeed report them as notarized.

(Disclaimer: this is all based on the experimenting I’ve done, since there is obviously no documentation. But I’ve spent enough time with it that I think it is pretty much correct.)

LikeLiked by 1 person
- 6
  
  hoakley on May 29, 2019 at 5:07 pm
  
  Thank you.
  I’m trying to follow Apple’s documentation as to how to validate a notarized app (it is silent on other types of code bundle), and recent discussions on Twitter in which it was proposed that the only way to check for notarization is using stapler.
  I have a follow-up article which will publish here on Friday morning in which I look specifically at notarized executable code which doesn’t have a ticket attached, something which Apple doesn’t document at all well.
  But, anticipating my conclusions there, stapler doesn’t actually check notarization at all, merely the presence of a stapled ticket, as you have seen. You may, though, be surprised at spctl, which is also not at all reliable!
  If you or anyone else has any bright ideas as to how to check whether a code bundle has been notarized, with or without a stapled ticket, please let me know.
  Howard.
  
  LikeLike
  - 7
    
    Randy Saldinger (Mothers Ruin Software) on May 29, 2019 at 5:43 pm
    
    I’m curious to see what you’ve found about spctl. So far, I haven’t seen any surprising results (except for the crappy errors one gets when choosing the “wrong” assessment type, such as “execute” for a dmg); but I haven’t tried a wide range of products, either.
    
    Are you seeing false positives or false negatives with regard to notarization of bundles?
    
    In any case, if spctl is not reliable w.r.t. to notarization, then I suspect Gatekeeper is not, either, since the latter is built on the same System Policy database to which spctl is a front-end. (The support for notarization in Security.framework is actually amongst Apple’s shrinking open-source code, and I’ve spent some time with the most recently available version, which is from 10.14.1.)
    
    LikeLiked by 1 person
    - 8
      
      hoakley on May 29, 2019 at 6:02 pm
      
      Spoiler alert!
      spctl doesn’t want to check executable code bundles like KEXTs which aren’t apps. I just can’t get it to give any useful answer. So I don’t think that reflects on trustd, but on the front end in the tool. However, Apple doesn’t document the notarization of code bundles other than apps. Maybe they’re only really supposed to be notarized inside an Installer package. But being unable to check whether any given bundle has been notarized seems a fairly serious omission.
      Howard.
      
      LikeLike
    - 9
      
      Randy Saldinger (Mothers Ruin Software) on May 29, 2019 at 6:25 pm
      
      Ah, interesting. I hadn’t experimented with kexts before now.
      
      Have you tried the (baffling!) “install” type for assessment? At least on my 10.14.5 system, this actually indicates notarization on one of the kexts from this package, e.g.:
      
      $ spctl -v –assess –type install ~/Desktop/VBoxDrv.kext
      /Users/randy/Desktop/VBoxDrv.kext: accepted
      source=Notarized Developer ID
      
      Why it should want “install” instead of “execute” here, I don’t know, but at least the response makes sense.
      
      It doesn’t, however, look like the containing VirtualBox.pkg is notarized, which is … puzzling? Unless they submitted the kexts for notarization in a different container (or a different version of the package)? Which is possible, I guess, although I’d think this would eventually be problematic…
      
      LikeLiked by 1 person
    - 10
      
      hoakley on May 29, 2019 at 7:00 pm
      
      Thank you for reminding me that with the state of documentation in macOS it’s better to ignore it completely and just randomly guess.
      Yes, pretending that a KEXT is an Installer package for the purpose of checking its notarization using spctl solves the problem.
      Given that spctl‘s man page was last updated in 2012, I suppose that’s only to be expected.
      I have given you due credit in the article – thank you for the solution.
      It then of course begs the question as to whether Oracle hadn’t notarized and got Apple to do this post hoc, or whether this was the first failure of the new notarization system at Apple’s end. I guess we’ll never know, but it doesn’t bode well for 10.15.
      Oh – and I don’t think there was any need to notarize the Installer package, only the KEXTs, from my understanding of Apple’s instructions to developers.
      Howard.
      
      LikeLike
    - 11
      
      Randy Saldinger (Mothers Ruin Software) on May 29, 2019 at 7:49 pm
      
      If I understand the timeline, I’d suspect that Oracle didn’t notarize before 10.14.5 was released, but then did so a week ago. They could (and probably did) submit the same exact kexts for notarization as they’d released previously; once those were approved, the previously-released package would just work, because in the absence of a stapled ticket, macOS would query Apple’s servers for the CDHash of the kext, and now it is there as approved. (The CDHash will change with every build of the kext, though, so it would need to be the absolute identical bundle as before.)
      
      There was probably nothing special or exceptional about this, and it probably did not require Apple to hack anything, either. Indeed, I can [and have] notarized years-old packages that are now recognized as such, unchanged. Likely, the only thing that Apple had to do here was to tell Oracle to just submit the same versions for notarization.
      
      The fact that the package does not show as notarized makes me suspect that, for whatever reason, they didn’t use that exact package as the container for their notarization request. But as long as the kexts were identical, it oughtn’t matter (and I guess it doesn’t).
      
      As to what is gained by notarizing an installer package, that’s a bit nebulous. If the package installs only bundled things (like kexts and apps), it might not matter. If the package installs any standalone executables (which are properly code-signed), I believe those CDHashes get registered and thus those tools are also considered notarized. (Maybe they’d also get notarized if tossed into a dmg, but that would be a strange way to deliver a standalone executable in any case.)
      
      Anyway, whether you need to notarize the package per se, you must choose some container (dmg, zip or installer package) to deliver your software to Apple’s notarization service in the first place. You might as well use the same one you use to deliver your software to users, and at least in the case of a dmg or an installer package, the container gets notarized along with the contents. (Not sure if a zip container is ever considered notarized, since I haven’t played with this, and I don’t know if it has a cryptographically secure hash.) But again, does it matter if the container is notarized? The standalone executable in a package is the only case I can concoct where it might (and I haven’t tried that).
      
      By the way, I thought of another way to directly query the notarization status of a code-signed-thing:
      
      $ codesign –test-requirement=’=notarized’ –verify –verbose ~/Applications/Suspicious\ Package.app/Contents/SharedSupport/spkg
      /Users/randy/Applications/Suspicious Package.app/Contents/SharedSupport/spkg: valid on disk
      /Users/randy/Applications/Suspicious Package.app/Contents/SharedSupport/spkg: satisfies its Designated Requirement
      /Users/randy/Applications/Suspicious Package.app/Contents/SharedSupport/spkg: explicit requirement satisfied
      
      That last line, “explicit requirement satisfied” says that our simple Code Signing Requirement (consisting of one term: notarized) is satisfied. As near as I can tell, this consults the syspolicyd daemon in the same way as spctl and Gatekeeper, and will consult Apple servers, if the information is not already cached. If it were not notarized, you’d get:
      
      test-requirement: code failed to satisfy specified code requirement(s)
      
      That said, spctl is a higher-level check and is probably preferable, I’m guessing.
      
      LikeLiked by 1 person
    - 12
      
      hoakley on May 29, 2019 at 9:21 pm
      
      Thank you again.
      Yes, I also suspect that is what happened, in which case it begs the question as to why Oracle’s engineers failed to heed the instruction on notarization of KEXTs. It also explains why the tickets aren’t stapled to the KEXT bundles.
      One disadvantage of this is that successful installation requires an Internet connection, or the notarization can’t be checked, which seems the purpose of stapling the ticket.
      Howard.
      
      LikeLike
13

Michael Tsai - Blog - macOS 10.14.5 Requires New Developers to Notarize on May 30, 2019 at 8:04 pm

[…] Update (2019-05-30): Howard Oakley: […]

LikeLike

·Comments are closed.

Share this:

Related