Just when you thought the dust of notarization had settled, it all kicks up again. First there was another strange case of an old app which had mysteriously been notarized without its developer being aware.

However spooky that might be for someone who last built and shipped an app four years ago, it’s actually a feature of Apple’s security which is encouraged. To quote from the Apple Platform Security guide:
“On macOS, code signing and notarization work independently — and can be performed by different actors — for different goals. Code signing is performed by the developer using their Developer ID certificate (issued by Apple), and verification of this signature proves to the user that a developer’s software hasn’t been tampered with since the developer built and signed it. Notarization can be performed by anyone in the software distribution chain and proves that Apple has been provided a copy of the code to check for malware and no known malware was found. The output of Notarization is a ticket, which is stored on Apple servers and can be optionally stapled to the app (by anyone) without invalidating the signature of the developer.”

More interesting is who has performed these ghost notarizations on old third-party products, and why. Although there’s no shortage of organisations and people with appropriate Developer Certificates, several have suggested that most if not all of these ghost notarizations were performed by Apple. If that’s true, we should ask why Apple didn’t have the courtesy to inform developers that it was doing this to their products at the time, particularly as it was simultaneously putting the screws on developers to get their apps notarized. After all, these are products which are distributed outside the App Store, therefore Apple isn’t part of the “software distribution chain” at all.

I’m not suggesting for a moment that these ghost notarizations are misleading or compromise security, but they are thoroughly weird for the developers affected.

The bigger story, though, started with Allan Odgaard’s article titled macOS 10.15: Slow by Design, in which he claims that every time a user runs a new script in Catalina, it’s checked to see whether it’s notarized, imposing a significant pause while that’s checked with Apple’s servers. This was further explored by Jeff Johnson, who concludes that Catalina does indeed check whether unsigned executable code, including shell scripts, has been notarized.

So far, the evidence remains strong but circumstantial. The first time that Catalina runs an app, command tool, shell script, etc., there is a significant delay during launching which disappears when the same item is run subsequently. Using packet tracing, Jeff has demonstrated that this delay coincides with outgoing connections to the domain api.apple-cloudkit.com, which he identifies as being responsible for performing checks on notarization.

This is a very difficult event to investigate without understanding how macOS performs checks on notarization. Unfortunately the unified log is of no help at all. Launching an app with a quarantine flag set results in a long series of entries including trust evaluations, Gatekeeper assessment, and a scan by XProtect. The first time that you run a shell script, even with its quarantine flag set, all you see in the log is
0.723508 blowhole libsystem_info.dylib Retrieve User by ID
0.725138 co.eclecticlight.blowhole celerity test 3
the second entry being the script completing by writing a message to the log.

But, as Apple warned us at WWDC a year ago, every command tool gets similar treatment. It’s easy to verify this by taking an existing command tool and making a small change to it. When you try to run the modified tool, the kernel will quickly come back with an error such as
CODE SIGNING: process 43699[blowhole]: rejecting invalid page at address 0x10394b000 from offset 0x4000 in file "/Users/hoakley/Documents/blowhole" (cs_mtime:1590243390.615510768 == mtime:1590243390.615510768) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)
and the tool will be crashed.

This shouldn’t be surprising for apps and command tools: Apple did make it abundantly clear that Catalina was going to subject every run, not just first quarantined runs, to Gatekeeper’s checks. But this is bizarre for shell scripts, which are after all mere text files. Although Jeff claims that shell scripts can’t be code signed, they actually can; indeed, in principle at least, there’s no reason that you can’t ‘code’ sign any file on your Mac. This is explained, for instance, by Carl Ashley in the context of PPPCP profiles.

There are two options available for signing single files, as opposed to the bundles that form apps. For command tools, the best approach is to merge the signature and any Info.plist file into the single binary, a process I have detailed for those who need to get command tools notarized. You can’t do that with a shell script, of course, so the codesign tool will add four extended attributes (xattrs) named com.apple.cs.CodeDirectory, com.apple.cs.CodeRequirements, com.apple.cs.CodeRequirements-1, and com.apple.cs.CodeSignature. This is standard practice, for example when signing AppleScripts.

When a shell script has been signed, macOS has its identifier and certificate details which it can use to check whether it has been notarized. But supplied with just a plain text file, the only information which could be sent to Apple’s servers is its checksum or hash, which won’t be unique by any means. In those circumstances, it seems impossible for macOS to attempt to check notarization.

One other strange thing which happens to shell scripts the first time that they are run in Catalina is that a com.apple.macl xattr is added to them, containing a UUID which is common across several scripts, at least. That doesn’t appear to contribute to any delay in launching the script, but is further evidence that what is recorded in the unified log is no reflection on the processes which have taken place. It also raises further questions about the purpose of this new type of xattr, which had previously been associated with per-document privacy control by TCC.

As Jeff points out, none of this is even hinted at in Apple’s developer documentation on notarization, nor anywhere else. If we think we’ve got the hang of notarization and how Gatekeeper works, it’s based solely on the verbose log entries made for apps. What happens in other cases, with command tools and shell scripts, remains secret. For now.