To understand why current versions of Safari appear to be having problems connecting to some sites, particularly those affected by the recent Let’s Encrypt certificate changes, I’ve been exploring what’s recorded in the Unified log. This article casts more light on the checks which Safari runs, and how they can fail. For this, I’m using Safari 15.0 (16612.1.29.41.4, 16612) running on macOS 11.6 (20G165).

All that I did was to press the Return key to connect Safari to a site known to be using Let’s Encrypt certificates which had previously been rejected. Log entries are given with the time in seconds at the start, followed by the relevant process or subsystem name, and the message. I have edited out thousands of other messages which aren’t directly relevant.

The first entry records the keypress being sent to Safari to handle, following which one of the responses is to create a network task through App Transport Security (ATS), which becomes connection 41, for which TLS is enabled.
0.631303 Safari AppKit sendAction:
0.636017 ATS com.apple.WebKit.Networking CFNetwork Task <6E5C8F63-3126-4223-8B4E-AF80B0F54927>.<1> {strength 0, tls 4, ct 0, sub 0, sig 1, ciphers 0, bundle 1, builtin 0}
0.636655 com.apple.WebKit.Networking libnetwork.dylib nw_connection_create_with_id [C41] create connection to Hostname#b1c3128f:443
0.636661 CFNetwork Connection 41: enabling TLS
0.636662 CFNetwork Connection 41: starting, TC(0x0)

About 0.03 seconds after the keypress, Safari Safe Browsing looks up the URL I had entered, to check whether it’s a blacklisted site.
0.666825 SafariSafeBrowsing Look up a url
0.666972 SafariSafeBrowsing Send GetSafeBrowsingEnabledState message to safe browsing service
0.668560 SafariSafeBrowsing Receive GetSafeBrowsingEnabledState response from safe browsing service
0.668582 SafariSafeBrowsing Send GetDatabases message to safe browsing service with protection type 1
0.668605 SafariSafeBrowsing Already waiting for GetDatabases response with protection type 1
0.668641 SafariSafeBrowsing Received GetDatabases message with protection type 1
0.668675 SafariSafeBrowsing Send GetDatabases reply with protection type 1
0.668700 SafariSafeBrowsing Receive GetDatabases response from safe browsing service with protection type: 1
0.668701 SafariSafeBrowsing Receive GetDatabases response from safe browsing service
0.668748 SafariSafeBrowsing Perform url lookup in the database
0.668881 SafariSafeBrowsing There are no matches in databases with given url
It’s interesting to note the time delays involved here: the first response took around 0.0015 seconds, and GetDatabases messages were responded to in around 0.0001 seconds, implying that these are local and not remote lookups.

Safari also uses machine learning to determine whether sites are likely to be part of a phishing attack, a result which is reported in the log.
0.697267 MLPhishing Safari SafariSharedUI Classified URL <private> as LikelyNotPhishing

macOS uses an open source derivative of OpenSSL named BoringSSL to handle its TLS connections. At this stage, the server certificate is obtained and verified before calling for its trust evaluation.
0.779674 libboringssl.dylib boringssl_context_info_handler(1873) [C41.1:2][0x7fcd7a23e730] Client handshake state: TLS client read_server_certificate
0.779695 libboringssl.dylib boringssl_context_info_handler(1873) [C41.1:2][0x7fcd7a23e730] Client handshake state: TLS client read_certificate_status
0.779696 libboringssl.dylib boringssl_context_info_handler(1873) [C41.1:2][0x7fcd7a23e730] Client handshake state: TLS client verify_server_certificate
0.779786 libboringssl.dylib boringssl_context_evaluate_trust_async(1547) [C41.1:2][0x7fcd7a23e730] Performing external trust evaluation
0.779813 libboringssl.dylib boringssl_context_evaluate_trust_async_external(1532) [C41.1:2][0x7fcd7a23e730] Asyncing for external verify block
0.779815 libboringssl.dylib boringssl_session_handshake_incomplete(96) [C41.1:2][0x7fcd7a23e730] Handshake incomplete: certificate evaluation result pending [16]

Strict trust evaluation, performed by trustd the macOS certificate trust evaluation service, finds a problem with the TemporalValidity not of the server (leaf) certificate, but in its chain of trust. Note that there’s no evidence that those certificates are downloaded.
0.779840 CFNetwork Connection 41: asked to evaluate TLS Trust
0.779990 Security SecTrustEvaluateIfNecessaryFastAsync
0.780573 trustd cert[1]: TemporalValidity =(leaf)[]> 0
0.780575 trustd Security SecItemCopyParentCertificates_ios
0.801844 trustd cert[1]: TemporalValidity =(leaf)[]> 0
0.802065 trustd cert[2]: TemporalValidity =(leaf)[]> 0
0.802136 trustd Security SecItemCopyParentCertificates_ios
0.804712 trustd cert[1]: TemporalValidity =(leaf)[]> 0
0.804718 trustd cert[2]: TemporalValidity =(leaf)[]> 0
0.804731 trustd cert[1]: TemporalValidity =(leaf)[]> 0
0.804784 trustd cert[1]: TemporalValidity =(path)[]> 0
0.804786 trustd cert[2]: TemporalValidity =(path)[]> 0
0.805392 Security Trust evaluate failure: [ca1 TemporalValidity] [root TemporalValidity]
0.805394 CFNetwork StrictTrustEvaluate failed
0.805490 Security Trust evaluate failure: [ca1 TemporalValidity] [root TemporalValidity]
0.806455 Security Trust evaluate failure: [ca1 TemporalValidity] [root TemporalValidity]

This is reported as an error with the code -9814, officially errSSLCertExpired, which is reported upward as code -1202, kCFURLErrorServerCertificateUntrusted.
0.806476 CFNetwork Connection 41: default TLS Trust evaluation failed(-9814)
0.806476 CFNetwork Connection 41: TLS Trust result -9814
0.806477 CFNetwork Connection 41: TLS Trust encountered error 3:-9814
0.806477 CFNetwork Connection 41: encountered error(3:-9814)
0.806478 CFNetwork Connection 41: cleaning up
0.806566 CFNetwork Connection 41: unable to determine interface type without an established connection
0.806610 CFNetwork Task <6E5C8F63-3126-4223-8B4E-AF80B0F54927>.<1> can retry(N) with reason(2) for error [3:-9814]
0.806614 CFNetwork Task <6E5C8F63-3126-4223-8B4E-AF80B0F54927>.<1> HTTP load failed, 0/0 bytes (error code: -1202 [3:-9814])
0.806637 CFNetwork Task <6E5C8F63-3126-4223-8B4E-AF80B0F54927>.<1> summary for task failure {transaction_duration_ms=170, response_status=0, connection=41, reused=1, request_start_ms=0, request_duration_ms=0, response_start_ms=0, response_duration_ms=0, request_bytes=0, response_bytes=0, cache_hit=0}
0.806638 libboringssl.dylib boringssl_context_evaluate_trust_async_external_block_invoke(1520) [C41.1:2][0x7fcd7a23e730] Cancelled during verify block
0.806653 libnetwork.dylib [C41 Hostname#b1c3128f:443 tcp, bundle id: com.apple.Safari, pid: 9933, account id: 408aa850, url hash: 58ae3951, tls] dealloc
0.806660 CFNetwork Connection 41: done
0.806790 SymptomEvaluator [0.000] 9936 com.apple.WebKi (null) SYMPTOM_CERT_ERROR cfnet 00000003 ffffffffffffd9aa 00000027 00000001
0.806824 SymptomEvaluator CERT_ERROR from com.apple.WebKi, domain 3 error -9814 num attempts 39 num symptoms 1
0.806825 SymptomEvaluator Remove cert error from history: too old, retain time 60
0.806828 SymptomEvaluator Remove cert error from history: too old, retain time 60
0.806830 SymptomEvaluator evaluate gives null, count 1 while threshold 3
That’s the first of three attempts to evaluate the server’s certificate.

Second and third attempts end in the same error.
0.923955 CFNetwork Connection 42: asked to evaluate TLS Trust
0.927388 CFNetwork Connection 42: default TLS Trust evaluation failed(-9814)
0.927756 SymptomEvaluator evaluate gives null, count 2 while threshold 3

Faced with that, Safari then displays the certificate error to the user.
0.984727 JavaScriptCore Displayed certificate error with error code: -1202

The certificate information provided by Safari on that Mac showed it was the intermediate certificate which had already expired, a day before the Root did. However, as I have already recorded, connecting to exactly same site using Safari 15 on Monterey beta resulted in success, with the certificate information reporting the updated certificates, neither of which had expired.

As far as I can see, the only explanation consistent with these logs and the observations on the two systems running side-by-side on the same network is that the Big Sur system obtained its intermediate and root certificate information locally, from a cache or database which hadn’t been updated for the new certificates, while the Monterey system obtained fresh certificate information which did reflect the changes.

I welcome alternative explanations, provided that those too fit these facts.