hoakley November 25, 2020 Macs, Technology

macOS has checked app signatures online for over 2 years

A week ago, largely as the result of a server problem on 12 November, there was a storm of concern over the use by macOS of Apple’s OCSP service to check certificates, and resulting leakage of private data. Apple responded rapidly to mounting concerns and made commitments to address these issues over the coming year. What has been puzzling me ever since is that these OCSP checks have been well-known for a couple of years, and only now have attracted attention. With the immediate aftermath of the release of Big Sur now subsiding, this article traces their history, and explains how they came about.

Although the origin of code signing in macOS has become lost in the mists of time, as far as I can see, it appeared in 2007, but wasn’t really taken seriously until Gatekeeper was introduced in 2012, and became even more important with notarization, which was new with Mojave in 2018.

Various vulnerabilities have been discovered in the processes involved in signing and their use in macOS over that period. Among the most important, and most relevant to this story, are those detailed by Josh Pitts in June 2018. These affected a lot of well-known security products including LittleSnitch, and more generally software from Facebook. What is particularly significant, with the wisdom of hindsight, is that these vulnerabilities exploited Universal binaries, which Apple internally knew would soon become widespread again, and of potentially great importance.

At the end of that year, I reported here that macOS Mojave 10.14.2 was happy to run apps whose developer certificates appeared to have been revoked. This provoked long discussions, in which a very experienced developer asserted:
“I disagree with the whole notion that there are ‘signature problems’. Code signatures are designed for Gatekeeper. Gatekeeper is designed for first launch. Gatekeeper has changed over the years. Old signatures on installed apps are irrelevant, not a problem.”

A security researcher expressed opposite opinions about the value of signature checks:
“Since macOS doesn’t check code signatures after the first run, malware could infect many of the apps on your system, without root, and you’d never know. All it would take is running the wrong app once. Plus, of course, when malware gets revoked, it’ll still run on infected Macs.”

I delved a bit deeper, and a couple of days later I described how macOS 10.14.2 was starting to check signatures more thoroughly after first launch. Among the log excerpts that I published in that article were the telling entries:
30.343884 SecTrustEvaluateIfNecessary 30.345255 com.apple.securityd asynchronously fetching CRL (http://crl.apple.com/root.crl) for client (lsd[355]/0#-1 LF=0) 30.345305 com.apple.securityd cert[2]: AnchorTrusted =(leaf)[force]> 0 30.346576 com.apple.securityd MacOS error: -67030 30.346629 com.apple.securityd MacOS error: -67030 30.361455 SecTrustEvaluateIfNecessary 30.362900 com.apple.securityd asynchronously fetching CRL (http://crl.apple.com/root.crl) for client (amfid[124]/0#-1 LF=0) 30.362964 com.apple.securityd cert[2]: AnchorTrusted =(leaf)[force]> 0 30.364183 com.apple.securityd MacOS error: -67030 30.378125 com.apple.securityd MacOS error: -67030 30.378189 com.apple.securityd MacOS error: -67030 30.378271 com.apple.securityd MacOS error: -67030 30.378316 com.apple.securityd MacOS error: -67030 30.378356 com.apple.MobileFileIntegrity Basic requirement validation failed, error: (null) 30.378463 /Applications/SignetTest.app/Contents/MacOS/Signet signature not valid: -67030 30.378478 AMFI: code signature validation failed. 30.380499 SecTrustEvaluateIfNecessary 30.381862 com.apple.securityd asynchronously fetching CRL (http://crl.apple.com/root.crl) for client (amfid[124]/0#-1 LF=0) 30.381904 com.apple.securityd cert[2]: AnchorTrusted =(leaf)[force]> 0 30.383124 com.apple.securityd MacOS error: -67030 30.383692 com.apple.MobileFileIntegrity <private>: Broken signature with Team ID fatal. 30.383781 mac_vnode_check_signature: /Applications/SignetTest.app/Contents/MacOS/Signet: code signature validation failed fatally: When validating /Applications/SignetTest.app/Contents/MacOS/Signet: The code contains a Team ID, but validating its signature failed. Please check your system log. 30.383800 proc 17245: load code signature error 4 for file "Signet" 30.403372 com.apple.launchservices RETURNING: { "ApplicationType"="Foreground", "CFBundleExecutablePath"="/Applications/SignetTest.app/Contents/MacOS/Signet", "CFBundleIdentifier"="co.eclecticlight.Signet", "DeathTime"=now-ish 2018/12/21 09:18:30, "LSBundlePath"="/Applications/SignetTest.app", "LSDisplayName"="SignetTest", "LSExitStatus"=9, "pid"=17245 } 30.648151 Saved crash report for Signet[17245] version ??? to Signet_2018-12-21-091830_Howards-iMac-Pro.crash

That was for a notarized app which didn’t have a quarantine flag set, and had never even passed through my local network, let alone been downloaded from the internet. Those entries show clearly three separate connections being made by com.apple.securityd to Apple’s Certificate Revocation List (CRL) service using OCSP and a plain HTTP connection (in bold). In this case, the validation failed on each occasion, and as a result the app was crashed and not allowed to open. At the time, no one raised any concerns about these connections or their use of plain HTTP.

In July 2019, I explained here how different types of signature checks worked, and how developers could add their own code integrity checks which include a CRL check with Apple’s OCSP service. This included log extracts which again showed clearly what happened.

Until 2018-19, it appears that macOS stored information about certificate revocations locally, in the ‘Gatekeeper’ database at /private/var/db/gkopaque.bundle, which at one time Apple updated every couple of weeks. But those Macs which have kept pace with the latest release of macOS stopped accessing that database in September 2019, with the release of macOS 10.15 Catalina. Apple hasn’t released an update to it since 26 August 2019, and anyone with a fresh installation of Big Sur will have a truly ancient version installed. As I pointed out here, that ‘Gatekeeper’ database is now disused.

Instead, Catalina and Big Sur now check all executable code on loading, and, when that code is signed with a developer certificate, perform an online check with Apple’s OCSP service, which has suddenly become so controversial.

Since the introduction of Gatekeeper in 2012, Apple has apparently revoked many compromised developer certificates. We see the tip of the iceberg of malicious software which is signed, detected by Apple, and quickly has its certificate revoked. That has already occurred with several malware products which were also notarized, including Shlayer and MacOffers. Without a rapid and effective means of checking the validity of a signing certificate with Apple’s OCSP server, there is little point in using signatures as a means of distinguishing benign from malicious code.

Those who consider that Apple’s current online certificate checks are unnecessary, invasive or controlling should familiarise themselves with how they have come about, and their importance to macOS security. They should also explain how, having enjoyed their benefits for a couple of years, they’ve suddenly decided they were such a bad idea after all, and what should replace them.

Postscript

This article has generated a lot of discussion, and I’m very grateful to Jeff Johnson in particular who has run more tests on older versions of macOS. I think there’s reasonable consensus that, when code signatures were first introduced, by “Perry the Cynic”, signing certificates passed unchecked, and if Apple did revoke certificates it seems to have had little if any effect until the introduction of Gatekeeper and the quarantine system from 2012.

As that system developed, well before High Sierra and probably before El Capitan too, Gatekeeper started to perform OCSP queries to check code signing certificate validity, but only for quarantined apps undergoing their first run. That would probably place the start of such limited checks to around 2014, but not that much earlier, as others have pointed out that in 2009 many apps still had broken signatures.

With the release of High Sierra in 2017, code signing certificate checks remained confined to Gatekeeper and quarantine, and that appears to have been the case with the first release of Mojave in 2018. However, recent certificate revocation incidents appear to have struck both Mojave and Catalina users as quickly, which suggests that by 10.14.6 in July 2019, code signing certificate checks had been extended to apps which had already cleared quarantine. Whether that was taking place around the time that I ran my tests late in 2018 (on 10.14.2) we’ll probably never know. One phenomenon which certainly confused me at that time was that moving an unquarantined app and launching it from a previously unknown path triggered more thorough signature checks, although I don’t know whether those might have included certificate checks using OCSP.

By the release of Catalina in October 2019, code signing certificates were being checked on loading all executable code when no quarantine flag was set. As far as we can tell those included the systematic checks of both signature and cdhashes which Jeff has described and I’ve summarised here.

So Apple only seems to have been performing such extensive checks over the last 16, and no more than 23, months, although they have been applied to quarantined apps for around six years. Perry the Cynic must be over the moon.

47Comments

Add yours

1

Peter on November 25, 2020 at 9:23 am

I remember the dark old Windows XP times when our office PCs were running mandatory virus scans around lunch time. *That* was invasive…

Big Sur introduced a new setting *System Preferences > Security & Privacy > Privacy > Developer Tools*. I noticed it after installing the Xcode Command Line Tools. Do you have any idea what it does? Does it disable the certificate checks?

LikeLiked by 1 person
- 2
  
  hoakley on November 25, 2020 at 9:25 am
  
  No it doesn’t disable those as far as I know.
  Howard
  
  LikeLike
- 3
  
  hoakley on November 25, 2020 at 9:44 am
  
  It has also been available intermittently through Catalina. I believe it enables developer tools to run and debug apps.
  Howard.
  
  LikeLike
- 4
  
  Milo on November 25, 2020 at 10:58 am
  
  The following blog post has some insights into what the Developer Tools section is about:
  
  https://lapcatsoftware.com/articles/catalina-executables.html
  
  Apparently since Catalina macOS will try to check the notarization status of shell scripts and programs that were not compiled with Xcode online. This can get quite annoying if you have a bad internet connection because the checks can take multiple seconds on every first run of your script or program.
  
  I’m not sure how one would use this in practice though. In case of shell scripts, do you add the shell executable to Developer Tools, or your Terminal app? I don’t know.
  
  LikeLiked by 1 person
  - 5
    
    hoakley on November 25, 2020 at 11:05 am
    
    It’s not actually checking notarization status, as I pointed out here last week.
    Howard
    
    LikeLike
  - 6
    
    Milo on November 25, 2020 at 11:45 am
    
    I’m sorry if I’m confusing the terminology here. My point was that using the special section of Developer tools it seems possible to prevent some of the first run checks and to thereby make the developer experience somewhat more pleasant (especially on slow internet connections).
    
    I just reread your article about code checks from last week again. Thank you for publishing this information as a reference for others to find and learn. Unfortunately I’m sure I will forget the details again very quickly.
    
    LikeLiked by 1 person
  - 7
    
    Peter on November 25, 2020 at 11:52 am
    
    Thank you Howard and Milo. I didn’t know the Developer Tools pane was available in Catalina. Had never seen it there.
    
    Following the links in Jeff Johnson’s addendum, we should be able to add Terminal or iTerm, and their child processes should inherit the privileges. If the pane disappears, we can use spctl to put it back.
    
    I’m wondering if it’s a good idea to allow Terminal or even a beast like VSCode and all their children to bypass the system’s security measures. It may come handy to investigate strange delays when running command-line tools, though.
    
    LikeLiked by 1 person
8

Jeff Johnson on November 25, 2020 at 1:03 pm

“the origin of code signing in macOS has become lost in the mists of time”

Code signing in macOS was created by an Apple engineer known as Perry the Cynic. I actually met him one year at the Macworld San Francisco trade show. Here’s a public comment by Perry: “I do work for Apple, and I designed and implemented Code Signing in Leopard. If you think it’s going to usher in a black wave of OS fascism, you have every right to blame me – it was, pretty much, my idea.”
https://weblog.rogueamoeba.com/2008/03/07/code-signing-and-you/#comment-318

gkopaque.db is not a list of Developer ID certificate revocations. It’s explicitly a whitelist, as can be seen by dumping the SQL: “CREATE TABLE whitelist”. Here’s an old article about it by Daniel Jalkut. (Note the comment at the end of the article: “Thanks to Jeff Johnson for talking through some of the discoveries that led to this post”)
https://indiestack.com/2014/10/gatekeepers-opaque-whitelist/

The request to http://crl.apple.com/root.crl is simply checking the revocation status of Apple’s own Developer ID Certification Authority intermediate signing certificate. If you examine the cert in the System Roots keychain, you can see that URL under CRL Distribution Points. This request contains no information specific to third-party developers or apps. In contrast, there’s no CRL for third-party Developer ID leaf certs signed by Apple’s intermediate cert. Their status is only available via OCSP.

The Microsoft Messenger case is puzzling. Although Developer ID OCSP has existed for many years, as can be verified by looking at https://www.apple.com/certificateauthority/ in the Wayback Machine, I’m not sure whether it was a first launch or every launch check. I just did a packet trace on mac 10.13.6, and I didn’t see any Developer ID OCSP requests. I’ll have to do more testing on this older system to investigate.

It’s not clear how the Josh Pitts vulnerability is relevant to this article.

LikeLiked by 1 person
- 9
  
  hoakley on November 25, 2020 at 1:36 pm
  
  Thank you, Jeff, for unlosing the history of code signing from those mists, and for your additional comments.
  I’m aware that the ‘Gatekeeper’ databases are known to contain whitelists. What I haven’t seen anywhere is an explanation of how code revocation worked before Mojave/Catalina. Apple certainly did revoke code-signing certificates on malware then – I can remember security researchers reporting signed malware which then had its certificate revoked by Apple. The assumption has been that this was managed by local databases which were updated in those frequent ‘Gatekeeper’ data updates rather than OCSP lookups. Maybe Apple has been using OCSP queries even longer than I suggest, but only for the more thorough certificate checks performed when clearing Gatekeeper with a quarantine flag?
  What puzzled me two years ago, and still puzzles me today, is that the security system was clearly checking that Messenger certificate somewhere and being told that it had been revoked, but was allowing it to open all the same.
  My point about the Josh Pitts vulnerability is that he reported that to Apple in February 2018, before these changes in certificate checks took place (if I’m right as to timescale). Although specific to Universal binaries, that would have been a time when Apple’s engineers were starting to focus their attention on exactly those, knowing that ARM systems were in the pipeline. I think that was the time that Apple was looking seriously at how those new systems would handle signatures, and the vulnerability was notified at a critical time in that process. Apple’s response to the vulnerability was a stern ignoring, because by then the future, with these frequent checks, was already being built in. So that’s part of how we got here. At the time, there was a bit of the “so who uses Universal binaries now?”, which with the wisdom of hindsight reads very differently.
  Howard.
  
  LikeLike
  - 10
    
    Jeff Johnson on November 25, 2020 at 2:47 pm
    
    I’m not sure about your timeline. After all, macOS allowed an app with a revoked cert to launch in December 2018, right? This would not happen in 2020, as we saw with several cases: Charlie Monroe, HP printer drivers. It looks like Gatekeeper has performed an OCSP check on first launch for a very long time, perhaps back to 2012. The question is, when did it start checking on every launch? It’s likely that Signet with kSecCSStrictValidate causes an OCSP check, but what about Gatekeeper in 2018? My initial test of High Sierra suggests no OCSP on non-quarantined apps. Also, I’m hearing some anecdotal data that High Sierra users were not affected by the November 17 “appocalypse”. So, did Apple change the implementation after macOS 10.14.2?
    
    LikeLiked by 1 person
    - 11
      
      hoakley on November 25, 2020 at 8:53 pm
      
      Thank you again, Jeff. I’ll try to summarise the main points arising in a short postscript later tonight.
      Howard.
      
      LikeLike
12

alex on November 25, 2020 at 5:07 pm

Just how big are these revocation lists? Is it not feasible to simply send the revocation lists to the clients, so they can check locally, without blocking on network access? Update them once a day with the diffs.

LikeLiked by 1 person
- 13
  
  alex on November 25, 2020 at 5:45 pm
  
  This also has a better failure mode than the current system. When the network is down, you’d still have yesterday’s revocation list, which will be (in all likelihood) 99.9% complete.
  
  AIUI, with the current system, when the network is down, every app gets a free pass.
  
  LikeLiked by 1 person
- 14
  
  hoakley on November 25, 2020 at 8:58 pm
  
  Apple doesn’t publish CRLs for code signing signatures, other than obtained by individual OCSP queries, for reasons which I would have thought were obvious. So only Apple knows how many certificates it has revoked.
  Apple PKI’s other revocation lists are vast, involving sometimes hundreds of revocations each day, but those for code signing certificates should be far smaller.
  It’s also worth bearing in mind that there are huge numbers of Mac apps being published, but very few users are likely to run more than a tiny fraction of them. That’s why OCSP queries are so efficient, as they are with TLS certificates.
  Howard.
  
  LikeLike
15

paws on November 25, 2020 at 7:31 pm

I recall around similar timeframe to when I first heard about Gatekeeper, there was this new ‘XProtect service’ that made some news. Seems like it doesn’t get talked about much these days.

I was curious about it and just checked both Catalina 10.15.7 and Big Sur 11.0.1.
Running
md5 “/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.plist”
on both OSes returns
MD5 (/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.plist) = 5f78d709cdae623977f969180be92144

LikeLiked by 1 person
- 16
  
  hoakley on November 25, 2020 at 9:01 pm
  
  Thank you. I have lots of articles here about XProtect, which is very much alive and kicking, and whose data files are updated every fortnight. XProtect doesn’t have anything to do with checking signatures, of course.
  Howard.
  
  LikeLike
17

NTN Labs on November 25, 2020 at 11:28 pm

I’m not mad at Apple for trying to control what code is running on their machines. I’m mad that they can control what I can run on my machine. Because Gatekeeper can be used both ways: to stop harmful software AND to stop harmless software.

LikeLiked by 1 person
- 18
  
  hoakley on November 25, 2020 at 11:42 pm
  
  Thank you.
  You seem to be forgetting that you can run whatever you want, outside Apple’s control. On a per app basis, strip the signature using codesign, a tool provided by Apple. For your whole Mac or network, simply block connections to Apple’s server, and your apps won’t have their certificates checked at all.
  It’s that simple.
  Howard
  
  LikeLiked by 1 person
  - 19
    
    NTN Labs on November 30, 2020 at 10:07 am
    
    I agree, but IMHO user should not have to circumvent the protection to be free. Maybe it could be the other way around…
    
    LikeLiked by 1 person
    - 20
      
      hoakley on November 30, 2020 at 10:10 am
      
      Ah. Opt-in security?
      I don’t think that’s a wise idea for the great majority of Mac users who will of course be swayed by the latest hysterical claims about Apple spying on them.
      Make the default free from security protection, and guess what most users will choose?
      Howard.
      
      LikeLike
21

Natan on November 26, 2020 at 1:27 am

Yes, it’s not a new thing. I became aware of this when I noticed some slow downs on my Macbook Pro.

I have two problems with what Apple is doing:

1. It can’t easily be disabled. We need to use terminal commands that disable more than just this.
2. There’s a performance impact, especially on slow or unreliable networks.

Regarding point 2, Apple’s implementation is terrible. We don’t notice much difference on a fast connection or when offline, but if you’re having DNS issues or are connected to an unstable network, apps can take a few seconds to open or your system can freeze or become sluggish.

A few weeks ago, when Big Sur was released, Apple had a network outage that also affected the OCSP service. The result was apps not opening or super slow computers for many macOS users. This could be “fixed” by disconnecting from the internet or by adding the OCSP server URL to the hosts file, but almost no one knew this.

Maybe someone at Apple forgot that users don’t always have reliable gigabit connections like they have in their offices?

I don’t mind that this “feature” exists, but it bothers me that it can’t be easily disabled and that they kept quiet about it. I ended up disabling Gatekeeper a while ago – which is bad – but the performance hit was just too big for with my usage.

LikeLiked by 1 person
- 22
  
  hoakley on November 26, 2020 at 6:58 am
  
  Thank you.
  On 1, I think this needs to be thought out very carefully, and is something that Apple has already said it will address. You don’t want a big encouraging sign for ordinary users to disable this. What’s currently available is already very effective, although few people have been bothered to discover just how easy it is to disable at present. But everyone seems to expect these features are presented on a plate, rather than spending a couple of minutes research.
  On 2, I disagree completely about the current implementation. What’s more, almost anything that Apple does (as it has promised) to address privacy concerns will make this slower and open to more delays. What happens currently is a simple HTTP exchange, done in the twinkling of an eye. If you don’t believe me, watch it happen in the log every time executable code is loaded in Big Sur, or Catalina for that matter. Using a secure connection would impose overhead on every such connection. The current implementation uses a cache to avoid repeated checks, now within 12 hours, which also seems a fair compromise. I’d be interested to hear of something as effective which performs better.
  I don’t have a reliable or high bandwidth connection, but it almost never imposes any significant overhead here. And that’s on 3.5 Mbps most of the time.
  Being easily able to disable this would mean that every time there was a scare story, millions (yes, millions) of Mac users who really don’t understand what they’re doing with it would turn these checks off, and never turn them back on. Then there’d be a malware problem. Give most users a loaded pistol like that and they will shoot themselves quite happily.
  Howard.
  
  LikeLike
23

luke on November 26, 2020 at 3:25 am

Interesting to check popularity of the crl.apple.com DNS query (via Cisco DNS data)
https://domain.glass/crl.apple.com

LikeLiked by 1 person
24

Johnno on November 27, 2020 at 4:47 pm

This problem did actually affect users who’d done a fresh install of 10.13.6 around the time as the cache at /var/db/crls/crlcache2.db could get seeded with incorrect revocations rendering many App Store apps unable to launch but instead giving a code signature error that the signing cert had been revoked.

So whilst this wasn’t an online check, data from an OCSP response could poison the machine and break offline checks.

In some respects this is worse than doing online checks since revocations of revocations did not appear to be an automated thing that ever happened to fix the cache, thus making some App Store signing certs be revoked forever on a user’s machine at least without manual intervention.

I’ve got one app from the App Store which was delivered signed by a cert from 2016 which did run OK, but this means it does not look as if apps get resigned and re-delivered signed with different certs either unless there is a new version of course.

This means some users on 10.13.6 or perhaps earlier may just be experiencing the inability to run App Store apps right now if their revocations cache is broken and they don’t know what to do.

Funnily enough, I found all this out by importing the incorrectly revoked certs into the system keychain where a red error message saying the certs were revoked proved to me I’d gotten the correct ones.
I then thought that trusting them would fix things but it didn’t… well not until by fluke I used a tool (RB App Checker Lite) to check if an App Store app was signed OK.

After doing this the app and all App Store apps would launch OK for several hours but later stop launching until one of them was checked with RB App Checker lite again.
For a few days I was using this process to make App Store apps run again (MAARA).

So my trusting of the cert was overriding the revocation until the revocation cache was refreshed, likely not including incorrect revocations arriving just re-asserting it’s dominance as the canonical source. Me checking an app would reassert my dominance.

The proper fix of course was just to delete the cache.

I didn’t find this cache though until learning how to make OCSP requests myself and see that at least today that Apple’s OCSP server was not saying certain certificates were revoked when I presume it was a couple of weeks ago.

What a nightmare, but I did learn a lot and this forum was the kick-start point of everything so thank you.

LikeLiked by 2 people
- 25
  
  hoakley on November 27, 2020 at 5:02 pm
  
  Thank you.
  I don’t understand how a short-life cache can seed itself with revocations. The only way that I know that a Mac can learn that a certificate has been revoked is to send a query on that specific certificate ID to Apple’s server. Sure, you can send a manual request, again on a specific ID. But there’s no system for the server to broadcast IDs of revoked certificates, is there?
  You refer to multiple incorrectly revoked certificates. We know of the case of Charlie Monroe’s apps, and the HP certificate(s), but I’ve not heard of any others being revoked, nor of such problems recently. Which developers were affected and when, please?
  Also, AFAIK revocation isn’t something which the user can override in any way, other than be stripping the certificate.
  Howard.
  
  LikeLike
- 26
  
  cavenewt on November 29, 2020 at 5:44 pm
  
  There is a lengthy thread of people experiencing this High Sierra problem at https://discussions.apple.com/thread/252037712; if you start reading around November 25 you’ll see that we’re narrowing it down.
  
  High Sierra installs prior to November 11 have a file /Library/Keychains/crls/valid.sqlite3 that is 6.4mb. Newer HS installs have one that’s more like 17mb and many apps don’t launch, among other App Store problems. It’s not the HS installer, though, because even an older HS install will re-create a broken database if the existing working one is deleted.
  
  Many workarounds are only temporary, as you point out. Replacing the new or larger broken database with a copy of the older smaller one seems to be a longer-term workaround. But are you saying that deleting /var/db/crls/crlcache2.db will be a solution?
  
  It’s very frustrating that Apple has not even acknowledged this, much less done something to fix it.
  
  LikeLiked by 1 person
27

Johnno on November 27, 2020 at 5:34 pm

I was seeing these 2 certifications being considered revoked on my machine which had a fresh install of High Sierra on it (both an update from Sierra and a fresh volume) but not on other machines:

Certificate “Apple Mac OS Application Signing”:
Will expire on 7 Feb 2023.
SHA1 fingerprint: “B93BDAAAF1A8846B34BA32332635CB2B84853DA8”.
Serial: 345231221407495159 (04CA81F77D5E33F7)

Certificate “Mac App Store and iTunes Store Receipt Signing”:
Will expire on 7 Feb 2023.
SHA1 fingerprint: “27E253E32897D677B9C9FFCBC2E48BCDC3FB1101”.
Serial: 1075049177276090765 (0EEB5787E79E098D)

The machine appeared to be polling in the order of hours and that would make all apps stop working after my workaround but I don’t think that anything to do with these certs was being updated in var/db/crls/crlcache2.db but were already recorded incorrectly. I have a copy of that db rather than truly deleting it which is a keychain file but I can’t appear to see anything in it when I import it, but revocations as such don’t appear as things even hidden ones.

I didn’t go fully scientific and remove the ethernet after my workaround and come back to check after going to be or anything like that, so my observations are a little loose.

I think what the machine is polling for in this case though is the signing certificate of those shown above as console logs when search for “ocsp” mention it’s name “Apple Worldwide Developer Relations Certification Authority>, <cert(0x7feaf8810200)"

Aside: I did also clear the rows of the ocsp table from a couple of databases as described in https://support.globalsign.com/ssl/general-ssl/delete-crl-and-ocsp-cache as this installation was an upgrade from Sierra but given these were in ~/Library/Keychains/ and not system wide, plus the problem had occurred with a fresh install of High Sierra too, I don't think that anything to do with anything.

So in summary, I think that High Sierra checks for revocations of the important Apple owned certs by polling the OCSP service and then caches these results. During the incident, High Sierra was interpreting responses as meaning that these 2 certs were revoked and caching that incorrect state.

I think that only recent installations (new or upgrades) were affected. My prior volume running Sierra does have this cache present too but how it was written to could of course be different. If High Sierra was the first version to introduce this particular style of cache in that particular location then that would make my theory more plausible.
If all existing High Sierra installations had been knocked out and given a nasty cache then I'd expect more people to have experience the problem.

I would imagine that only 2 or 3 people installed High Sierra over the last couple of weeks given how old it is.
There is a report of one user doing exactly this and not being able to run 1Password from the App Store over on the 1Password forums, so at least with all this I could see that somebody else was experiencing the same issue.

LikeLiked by 1 person
- 28
  
  hoakley on November 27, 2020 at 7:06 pm
  
  Thank you.
  Nothing you show above suggests that either certificate has been revoked. When the check is run by macOS, the error you’ll see in the log or from an appropriate tool such as spctl or codesign is -2147409652 CSSMERR_TP_CERT_REVOKED, which is quite unmistakeable.
  I don’t recognise the certificates which you quote either. They don’t look to be developer signing certificates at all, but Apple’s. If it were to revoke such certificates that would have enormous global impact, so I think that somewhere along the line there’s a misunderstanding.
  Do you have log extracts or packet traces demonstrating that Apple’s OCSP server was being polled? When was this? On 12 November 2020, there was a service problem on that service which caused many issues in checking certificates, but that was cleared by about 2359 UTC that night.
  The link which you provide is most bizarre, as it describes a Windows behaviour and then explains how to ‘fix’ that on a Mac. I think that’s exceedingly dangerous and could readily cause security vulnerabilities on your Mac. Don’t tamper with CRL or other security databases unless you really know what you’re doing, no matter how ‘authoritative’ the advice might appear.
  Funnily enough, there are a very large number – millions – of Macs around the world which are still running High Sierra, yet no one else appears to have reported similar problems. Or have I missed them?
  As I have written above, we understand that High Sierra performs certificate revocation checks on code which is being checked by Gatekeeper when the quarantine flag is set. I see no evidence to the contrary.
  Howard.
  
  LikeLike
  - 29
    
    Johnno on November 27, 2020 at 11:52 pm
    
    Hi Howard,
    
    On the build I was running a couple of Apple certificates were being incorrectly marked as revoked locally, periodically and persistently, and I thought that perhaps the fact that this installation was new but of an ancient build would somehow explain why there wasn’t a massive outage for all the millions of others running the same or similar OS builds that hadn’t been installed so recently.
    
    Caching related bugs can of course be subtle and combining them with more obscure scenarios can sometimes mean that they don’t trigger too often. They are also difficult to discus when referring to state as with caching there is usually more than one copy of a state (that’s the whole point of a cache) where one can be erroneous. It’s an issue of scope and whilst I was talking about a local scope, you may have thought that I was talking about a global scope and have come to the conclusion that I thought whatever happens on my machine happens on everybody else’s running the same OS build.
    
    I still suspect that 10.13.6 does make regular OCSP requests. I’ll run Wireshark to see if I can get some captures. Since OCSP requests are currently plaintext regular HTTP with the payloads in a der binary encoding of ASN.1 (I believe), they should be easy to spot. Certainly the URL of http://ocsp.apple.com/ocsp03-wwdr08 should be easy to see. The ocsptool (which I believe is the backend to the openssl frontend) can read a captured request and show it in more human readable form to see which serial is being queried against which issuer.
    
    The general gist of my post was to thank you for your investigations which helped to solve an issue and let me delve a little into everything you’ve been explaining here. I wanted to agree with you that older builds of the OS do appear to be downloading and caching crls the chunky way of getting the full list but may also be running what looks like early attempts at using OCSP though not tied closely/at all with an application launch as newer version appear to be. I’ll get traces for my own curiosity.
    
    Thanks once again for your tools and posts. Posts on this kind of subject matter (at least those indexed well) are few and far between but very interesting so please keep up the good work.
    
    LikeLiked by 1 person
    - 30
      
      hoakley on November 28, 2020 at 12:02 am
      
      Thank you.
      AFAIK there’s no such thing as local revocation: the only authority which can revoke a security certificate is the CA which issued it, in this case Apple PKI. It’s then revoked worldwide for everyone, and officially once revoked that can’t be undone (although Apple seems to have undone some of its recent revocations which were performed in error).
      You can locally mark a certificate as not trusted on your system, I believe, but that’s quite different from revocation.
      Just over a year ago, one of Apple’s installation certificates expired, and many of its existing installers ceased working because of that. There’s no way to work around that: with those certificates the rules are different and once they had expired none of those installers could be used. Apple had to re-assemble all its system and other installers and post versions with new certificates before they’d work.
      Howard.
      
      LikeLike
- 31
  
  cavenewt on November 29, 2020 at 5:53 pm
  
  “I would imagine that only 2 or 3 people installed High Sierra over the last couple of weeks given how old it is.”
  
  Considering that 2009-era Macs max out at High Sierra, more and more people are installing it in order to give their machine the highest OS can run. I can assure you that it’s affecting more than just a couple of people. Even the pandemic might have something to do with it; the reason I got into this was having to upgrade a client’s computer to High Sierra so that she could run the latest version of Slack or Zoom for online courses.
  
  LikeLiked by 1 person
  - 32
    
    hoakley on November 29, 2020 at 7:53 pm
    
    Thank you.
    There’s something quite perverse about waiting to upgrade to a version of macOS which has just become completely unsupported!
    Howard.
    
    LikeLike
  - 33
    
    cavenewt on November 29, 2020 at 8:19 pm
    
    @hoakley
    
    “There’s something quite perverse about waiting to upgrade to a version of macOS which has just become completely unsupported!”
    
    It’s not perversity. For someone technical like you it may seem so. But I work for a lot of people who are not technically proficient, and keep using whatever they have for as long as it works.
    
    Often they resist updates because they need to keep using some important third-party application like QuickBooks or a film scanner, and don’t have the money or the knowledge to upgrade everything. Upgrading an operating system can be daunting for a lot of people, despite the fact that Apple has made it really hard to not upgrade one’s operating system over the last few years!
    
    LikeLiked by 1 person
    - 34
      
      hoakley on November 29, 2020 at 8:35 pm
      
      No: the perversity is not in running High Sierra (although if you read of all the problems in High Sierra documented here, you might think so!). It’s delaying upgrading to it until it has just become unsupported. A couple of years ago, it might have made sense. But running an unsupported version of macOS on hardware for which the only spares are also secondhand (thus of similar age) doesn’t seem to me to be much of a plan for the future.
      I well understand the problem with film scanners – it’s something which I had to address recently – but accounting software? Are you saying that there’s no accounting software which runs on a supported version of macOS?
      Why do you think that Apple is so keen for users to be running supported versions of macOS? It’s not as if it even charges for them, is it?
      Howard.
      
      LikeLike
  - 35
    
    cavenewt on November 30, 2020 at 5:02 pm
    
    Howard, I have been trying to find out if High Sierra is indeed “completely unsupported”. Apple did release a security update for it just a couple of weeks ago. I could not find any information from Apple or my usual sources; the best I could find was this 2018 article that said Apple has no specifically stated policy on when it renders an operating system obsolete https://www.howtogeek.com/350901/which-releases-of-macos-are-supported-with-security-updates/
    
    Not arguing, just asking! I do appreciate the thoroughness of your blog, and of course I recognize you from MacInTouch. Ric actually pointed me to this thread.
    
    —Colleen
    
    LikeLiked by 1 person
    - 36
      
      hoakley on November 30, 2020 at 5:23 pm
      
      Thank you.
      I keep coming back to look for this, but as far as I can see, Apple has never been explicit about its support of old versions of macOS. Its licence actually makes it sound a miracle that it supports anything at all!
      That said, its standard practice for many years has been:
      – full support, with bug and security fixes, only for the current (latest) version of macOS; currently that’s macOS 11, of course
      – security fixes, and very occasionally a serious bug fix, for the two previous major versions of macOS, which are currently 10.14 and 10.15.
      Apple normally rolls forward with the release of its new macOS in the fall, when there may be one final security fix for the outgoing version.
      But, as I said, this is only practice, not written policy in any published document. Any further support now for 10.13 and earlier would frankly be a miracle.
      Howard.
      
      LikeLike
37

Michael Tsai - Blog - Apple Server Outage Makes Mac Apps Hang on Launch on November 27, 2020 at 6:27 pm

[…] (2020-11-27): Howard Oakley (Hacker […]

LikeLike
38

Anonymous on November 29, 2020 at 5:41 pm

Many people are affected, new installations of High Sierra since early November can’t launch new apps. Something to do with certificate revocation…
https://discussions.apple.com/thread/252037712

LikeLiked by 2 people
- 39
  
  hoakley on November 29, 2020 at 7:51 pm
  
  Thank you.
  What evidence is there that this has anything to do with certificate revocation?
  I suspect this stems from fundamental misunderstandings. Documented CRL checks before Mojave have only been reported when Gatekeeper performs first run checks on non-App Store apps which have a quarantine flag set. As App Store apps are never in that situation, certificate checks are quite different: they don’t go through the same checks at all.
  There are a large number of users running more recent versions of macOS who are currently reporting similar problems with the App Store – I have heard of these in Catalina, for instance. Those don’t appear to be related to certificates, but an internal problem with the App Store or its app.
  Has anyone reported their problems to Apple Support? What have they said? Has anyone reported the problem to the vendors of the apps themselves, who can in turn raise them with Apple?
  Howard.
  
  LikeLike
  - 40
    
    cavenewt on November 29, 2020 at 8:26 pm
    
    If you read that linked thread you will see that it definitely has something to do with CRL. They have found two certificates that show up as revoked that should not be. Not only are apps refusing to run with a codesigning error (Pages etc), but some apps can’t even be downloaded from the App Store, until those two certificates are deleted from a database or the entire database is replaced with a copy that predates November 11. Then all of that stuff works again. There are several workarounds, some of which are only temporary, and it would be nice if someone would help us address the basic problem — which would appear to be Apple.
    
    Everybody who has tried to contact Apple support has, of course, no luck with tier 1 techs, and have not been able to talk to somebody up to feed food chain. There is someone who’s waiting on a callback from tier 3 but keeps getting put off.
    
    LikeLiked by 1 person
    - 41
      
      hoakley on November 29, 2020 at 8:43 pm
      
      Thank you.
      I see absolutely no good evidence that this has anything to do with CRLs. Sadly, many of the comments come from people who are way out of their depth, and it appears that many have been blindly deleting undocumented security databases from their Macs. That’s not how you go about investigating this sort of problem.
      Can you point me to any packet traces or log extracts which confirm any of these claims?
      Howard.
      
      LikeLike
42

Johnno on November 29, 2020 at 11:17 pm

cavenewt – Thanks for the link to the Apple forum. I was thinking that I and the only other example of a person on the 1Password forum were the only person with this problem.

Hi Howard,

Here you are 100% correct that there was no real revocation.
There was also no regular OCSP polling either.
However there were false clues along the way:

1) CSSMERR_TP_CERT_REVOKED can be returned on 10.13.16 when the underlying problem is that an intermediate cert https://developer.apple.com/certificationauthority/AppleWWDRCA.cer is not cached (outside of the normal keychains) and an an app is evaluated with signet or what’s your sign.
I can show a video of this being demonstrated on a Parallels VM if necessary.

2) Even more freaky is that if a keychain holds the prior version of the AppleWWDRCA cert which expired in 2016 and also holds the final signing cert “Apple Mac OS Application Signing” in a keychain then Key Chain Access will say say that this last certificate is revoked too.
Somebody else saw that on their travels as they have a screen shot here: https://discussions.apple.com/thread/252037712?answerId=253975528022#253975528022
This is more of an edge cases as normally only developers would install the AppleWWDRCA cert. It was on my machine but I haven’t written a Mac or iPhone app in years.

So I don’t think it’s unreasonable to take two signals from the OS saying that a cert is considered to be revoked as a route to investigate if there is anything recording a revocation state. I did check the CRL cache and the serial numbers of the 2 certs concerned (shown below) were not present but cleared it anyway knowing that clearing it would download a new copy. I didn’t suspect there to be a window where loads of malware lurking on my machine would suddenly start running.

I even checked for patches downloaded post install by sniffing the interface of a VM such as the content of http://updates-http.cdn-apple.com/2020/patches/001-55241/7C679F5A-4015-49F6-B3A2-44F59D200B8A/com_apple_MobileAsset_PKITrustSupplementals/6a5b8ee3299a8e4a956ef01d977791a393f68c45.zip to see if anything in there would give me a clue.

In the end, the fix for my upgrade of an existing volume to this now unsupported OS appears to have been:
1) Delete old certs from the key-chain and even the AppleWWDRCA that I don’t need.
2) Force the caching of certs that would normally be cached by verifying them.

This would likely work:
/usr/bin/security verify-cert -r AppleWWDRCA.cer -c ‘Apple Mac OS Application Signing.cer’ -R ocsp
/usr/bin/security verify-cert -r AppleWWDRCA.cer -c ‘Mac App Store and iTunes Store Receipt Signing.cer’ -R ocsp

This does result in trustd making some OCSP calls for each cert just incase there is some incorrect revocation state that needs clearing, that look like this where the request is encoded into the URI given it only needs a serial plus signing cert hash in it.
Request Method: GET
Request URI: /ocsp03-wwdr08/ME4wTKADAgEAMEUwQzBBMAkGBSsOAwIaBQAEFADrDMz0cWy6RiOj1S%2BY1D32MKkdBBSIJxcJqbYYYIvs67r2R1nFUlSjtwIIBMqB931eM%2Fc%3D
Request Version: HTTP/1.1
Host: ocsp.apple.com\r\n
Connection: close\r\n
User-Agent: trustd (unknown version) CFNetwork/902.6 Darwin/17.7.0 (x86_64)\r\n

The tool RB App Check Lite was what I used initially to work out which certs the 2 certs were, and it does this verification too so it ends up being the fix tool.

However on my machine things would revert to not working after a while.
It took me a few days to figure out that it was the expired AppleWWDRCA certificate that was getting in the way and that was only about an hour ago.
There was some kind of race/fight clearly going on there between some other cache and what keychain was saying. The fix is that some stuff shouldn’t be in keychain I presume.
I’m going to leave the VM running and also my main machine running with all these certs deleted from all keychains as they wouldn’t normally be there.
In 3 hours or so I’ll know if it worked I’d think.
It it doesn’t I’ll just give up an setup a cronjob to run a script including the two calls to ‘security verify-cert’ above.

In any case, the machine was in a state where without manual intervention many of my apps would no longer run.
For a user to go on a quest to make their machine run (especially when the OS is unsupported and Apple wouldn’t help) doesn’t appear unreasonable to me.

I have a MacBook which I do always update and generally use that as a testbed to see which of my older apps will fail.
I have for example a VST plugin I bought in 2006 for which there was only ever one 32-bit intel release before the company making it was bought.
I have old music projects that I’d like to be able to open and only until recently have I purchased newer software than refused to run on 10.12 which is why I bit the bullet to go to 10.13 even if super late in the day.

People drive classic cars and doing that has it’s risks. Anybody running a classic OS should know what the risks are but I’d agree with you that many people don’t.

Just to re-iterate what I’ve stated earlier though, you’ve been 100% correct in what you’ve said.
So whilst my issue wasn’t to do with CRLs or OCSP really after all, I think that if it wasn’t for the Apple incident and your blog articles about it, I’d have literally found no starting point to investigate why machine wasn’t running apps from the App Store after upgrading from Sierra to High Sierra.

Thanks once again for your blog.

LikeLiked by 1 person
- 43
  
  hoakley on November 30, 2020 at 6:07 am
  
  Thank you.
  I’m glad you seem to have found a solution to your problems.
  Howard.
  
  LikeLike
  - 44
    
    cavenewt on December 8, 2020 at 6:46 pm
    
    johnno summarized things nicely at https://drive.google.com/file/d/1Ngyty19xiUYkqCXbrgvcMocnUqqm-UtS/view
    
    Yes, it’s a certificate problem. Hopefully you will see the evidence you desire in there, if you’re interested. We have some workarounds for now. But it does appear to be something that Apple needs to fix on their end.
    
    LikeLiked by 1 person
    - 45
      
      hoakley on December 8, 2020 at 9:37 pm
      
      Thank you – so not the revocation checks performed online with Apple’s servers, but one of the local databases accessed by trustd. That is going to take Apple to fix, I’m afraid.
      I hope that someone has submitted a comprehensive Feedback report, and that a great many have contained Apple Support.
      Howard.
      
      LikeLike
46

Myob on February 21, 2021 at 2:08 pm

I apologize for posting this here as at best it might be only tangential. It seemed to be the most appropriate location that came up in a site search.

Not being able to update a Mac App Store (MAS) app isn’t a show stopper for me, it might be for others. I did have a Safari extension that needed updating is why I came upon the workaround. Mojave appears to be the only (?) OS affected.

If one can’t update a MAS app try:

In Activity Monitor, Force Quit appstoreagent then retry the update. It worked for me––app successfully updated. This Force Quit action likely won’t survive a reboot or an extended period of OS use time without reboot. Just repeat the action as needed until the issue is fixed.

Source:
https://discussions.apple.com/thread/252471707

LikeLiked by 1 person
- 47
  
  hoakley on February 22, 2021 at 12:11 am
  
  Thank you.
  Unfortunately, it isn’t only Mojave which is being affected – even Big Sur is at present.
  Howard.
  
  LikeLike