macOS security has been quietly changing in Ventura, and an account given at WWDC has provided revelations of more to come in macOS 14 Sonoma. This involves a sea change from largely static detection and protection to those based more on behaviours. Here I try to explain how some of these changes are likely to take effect.

Behavioural detection

In the past, macOS has largely relied on static methods for tasks like the detection of malware. The older XProtect system is a good example: it relies on detection signatures, updated periodically by Apple in Yara definition files. When new apps are checked by Gatekeeper, they’re scanned to see if they match any of the signatures of known malware; if they’re clear, XProtect and Gatekeeper consider the app is free from malware.

That can work for a lot of malicious software, but it can only ever detect what’s known, and can also be played or tricked into approving malicious code. One alternative is to monitor potentially malicious behaviour, as you do when you use a software firewall like Lulu or Little Snitch to monitor and block outgoing connections, a common behaviour of much malicious software. Behavioural detection can tackle and identify malware not previously known, but it’s also more difficult to ensure that it doesn’t generate many false positives.

Bastion and more XProtect

If you cruise Ventura’s logs, you may have come across two new names there: Bastion and XProtectBehaviorService. For example, here’s a pair of entries seen during a first-run Gatekeeper check:
1.037060 syspolicyd Got bastion violation BastionRule-3 path /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder responsible path: /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder
1.080139 XProtectBehaviorService Processed behavioral violation for process file:///System/Library/CoreServices/Finder.app/Contents/MacOS/Finder file:///System/Library/CoreServices/Finder.app/Contents/MacOS/Finder BastionRule-3

@tsunek0h has looked in some detail at the rules for what looks like behaviour-based malware detection. There are currently four Bastion rules, one of which monitors the behaviour of processes accessing folders in ~/Library/Application Support that contain sensitive data for third-party messaging apps like WhatsApp. At present, in Ventura, processes that access those aren’t blocked, but their behaviour is recorded in the XProtectBehaviorService database.

Environment constraints

In Robert Kendall-Kuppe’s presentation at WWDC last week, he spoke about what Apple refers to as environment constraints, rules to describe the expected behaviour of non-malicious code. These have already been implemented in Ventura for much of its system code, and in Sonoma extend to third-party software as well.

They can reduce the attack surface of third-party code, and initially are mainly aimed at helper apps that might be run as Launch Agents, Launch Daemons and Login Items, although they can extend to libraries and more. In Sonoma, apps are encouraged where appropriate to provide a dictionary containing a set of requirements, both in terms of facts and predicates, that determine what can use their helper apps, and how. Ventura version 13.3 introduced launch constraints, and the whole scheme of environment constraints is intended to be fully enforced in Sonoma.

Effects

Until last year, little could be done about malicious software until it had been detected and analysed by Apple’s security engineers. If any of it was found to be signed using an Apple Developer certificate, or notarized, its signature or ticket could be revoked immediately. Following that, a Yara rule to detect the malware would be developed and released in the next update to XProtect’s data files, by which time the malware developer may well have started abusing different signing certificates, and maybe even evaded the new Yara rule too.

The promise of behavioural detection is that malicious behaviour should be prevented before it can occur, without having to wait for detection and analysis by Apple’s engineers. These are early days still, and we can but wait and see how well Sonoma performs.