Open a document: how it works, and where it can go wrong, 2: Security check

We have double-clicked a document in the Finder, and macOS is going through the processes required to open it. Having ascertained the document’s UTI, macOS has worked out what it needs to do next. As this involves opening a file, the proposed action is passed to the Security Server, to check that it has no objections.

In this example, the URL at file:///Users/hoakley/Documents/0newDownloads/apfs/copyonwrite5.jpg resolves to a file of type public.jpeg. As this is a type which the Security Server watches for potential malware, it returns an error:
08:14:05.417444 CoreServicesUIAgent Error -60005 creating authorization
which indicates that authorisation to open that file has been denied pending checking, and triggers checks by Gatekeeper and XProtect.

Gatekeeper can have little work to do: this file is a document, not a bundle, is unsigned, and bears no quarantine xattr:
08:14:05.419048 XprotectService Gatekeeper check with file type public.jpeg

However, it could still be malicious, as some malformed JPEGs are, so it is up to XProtect to check:
08:14:05.419629 XprotectService Calling SecAssessmentCreate with URL file:///Users/hoakley/Documents/0newDownloads/apfs/copyonwrite5.jpg, context {
LSDownloadContentTypeKey = "public.jpeg";
LSDownloadCorrectedMIMETypeKey = "image/jpeg";
LSDownloadDepressRisk = 0;
LSDownloadDestinationURLKey = "file:///Users/hoakley/Documents/0newDownloads/apfs/copyonwrite5.jpg";
LSDownloadRiskCategoryKey = LSRiskCategorySafe;
LSDownloadRiskLevelKey = 0;
LSDownloadSuggestedFilenameKey = "copyonwrite5.jpg";
"context:feedback" = "<__NSStackBlock__: 0x700006510bc8>";
"context:uti" = "public.jpeg";
operation = "operation:lsopen";
08:14:05.448124 XprotectService SecAssessment results: {
"assessment:authority" = {
LSDownloadRiskCategoryKey = LSRiskCategorySafe;
"assessment:authority:flags" = 0;
"assessment:authority:source" = "_XProtect";
"assessment:remote" = 1;
"assessment:verdict" = 1;
} (null)
08:14:05.448133 XprotectService Gatekeeper enabled: 1
08:14:05.448158 XprotectService This is not malware. All checks passed.

In less than 0.03 second, XProtect has put the file through its checks, and concluded that it is safe to open.

When the document to be opened is a compound type, such as an RTFD package of type, macOS iterates through all the individual files within that package performing security checks on each.

Any problems detected by security checks are reported immediately in an alert, and no further attempt is made to open the document. In the case of, say, a JPEG embedded in an RTFD package, it may be worth discarding individual components which fail testing, but in general the best course is to trash the whole document and run additional checks for malware.

While this has been taking place, LaunchServices and other systems have been starting the app ready to open the document in it.