Apple wants those using Catalina and Big Sur to use apps from itself, obtained from its App Store, or – if you must – those from third-party developers which are notarized. Apps which you download from those third-parties normally gain a quarantine flag, so that they undergo full first run checks by Gatekeeper. When you then try to open one, macOS displays a dialog intended to check that you really do want to open it.

Click on the Open button and the quarantine flag will be cleared, and the app will open normally thereafter. This also applies to apps which you transfer between Macs using AirDrop, which isn’t considered to be fully secure.

Note that both of these contain the key sentence “Apple checked it for malicious software and none was detected”, which confirms that its notarization has been checked successfully.

Apps moved using iCloud don’t gain a quarantine flag, though, so don’t undergo the same thorough first run checks.

If the app you’ve downloaded hasn’t been notarized, macOS gets a bit more awkward about opening it. Attempting to double-click the app to open it will be met with refusal. You can then either use the General tab in the Security & Privacy pane to instruct macOS to ignore its lack of notarization and open the app, or you can use the Finder’s Open command to do the same.

You’re then offered a final dialog in which you can proceed to open your app.

Internals

What happens to that app in macOS is also a two-step process, involving notarization and other checks which are performed before the first and refusing dialog, and a streamlined approval process for the second dialog.

On both occasions, LaunchServices and RunningBoard are quickly fired up and start to launch the app, which is then suspended for its security checks. Because the app has a quarantine flag set, it is then translocated to a hidden temporary folder to ensure that it can’t pull any tricks which rely on it knowing where it has been launched from.

Before the first dialog can be displayed, several checks have to be completed. These include checking its executable code using hashes, determining whether it has been notarized, and checking any signature to see whether its developer certificate has been revoked. It also undergoes a scan by XProtect, to determine if it is known malware. Several of these are common to all executable code when it’s loaded in these recent versions of macOS, and detailed here.

Hash and signature checks are performed remotely with Apple’s online services when they’re available. When those services can’t be reached, for example if your Mac isn’t connected to the internet, those checks are skipped. For notarized apps, this relies on their ‘ticket’ being ‘stapled’ to the app. That isn’t always possible with other notarized code, which may rely on macOS being able to look its ticket up with Apple. If that isn’t possible on its first run, the code may be treated as unnotarized.

You can examine whether ticket lookup was attempted, and is successful, by looking for log entries from com.apple.syspolicy such as
CKTicketStore network reachability: 1, Wed Nov 18 15:03:29 2020
which confirms Apple’s service could be reached. Failure to obtain a notarization ticket is then reported as
com.apple.syspolicy ticket not available: <private>
com.apple.syspolicy completing lookup: <private>, 3
com.apple.securityd MacOS error: 3
com.apple.securityd Error checking with notarization daemon: 3

Armed with the information that this app has its quarantine flag set but hasn’t been notarized, com.apple.syspolicy.exec announces its decision:
App gets first launch prompt because responsibility: <private>, <private>
GK eval - was allowed: 0, show prompt: 1

As there’s no option for the user to press ahead and open the app, it’s then terminated:
com.apple.syspolicy.exec Terminating process due to Gatekeeper rejection: 860, <private>
ASP: Security policy would not allow process: 860, /Users/hoakley/Downloads/LockRattler.app/Contents/MacOS/LockRattler
and LaunchServices and RunningBoard then remove the app.

The second time around, after LaunchServices and RunningBoard do their preparatory work, com.apple.syspolicy.exec knows from the security databases that this app has only recently undergone full checks:
Code already evaluated, using results.
scan returning quickly for code: PST: (vuid: A170B5D8-7895-423E-94C3-3D161CB4D4B6), (objid: 6136628), (team: QWY4LRW926), (id: (null)), (bundle_id: (null))
App gets first launch prompt because responsibility: <private>, <private>
GK eval - was allowed: 0, show prompt: 1

Once the user makes the final approval for the app to run, the quarantine flag is cleared and launch is allowed to proceed:
com.apple.launchservices updateQuarantineFlags: finished for identifier=0FAEF941-87E0-4F12-B071-7DB851F4E23E, path=<private>
com.apple.syspolicy.exec Marked application as user approved: 1, PST: (vuid: A170B5D8-7895-423E-94C3-3D161CB4D4B6), (objid: 6136628), (team: QWY4LRW926), (id: (null)), (bundle_id: (null))
com.apple.syspolicy.exec Clearing Gatekeeper denial breadcrumb: PST: (vuid: A170B5D8-7895-423E-94C3-3D161CB4D4B6), (objid: 6136628), (team: QWY4LRW926), (id: (null)), (bundle_id: (null))

This final log entry refers to the “denial breadcrumb” which was set during the first pass through the clearance process. This appears to be located in one of the security databases, and isn’t reflected in any change in the quarantine flag or other extended attributes of the app.

Log information is here taken from macOS 11.0.1 (release) running on an Intel Mac.