Apple’s products have become more popular with malware authors too. How does Yosemite improve defences against Trojans and other nasties?

Since OS X 10.5 Leopard, when you run an installer for an app that you have downloaded, and the first time that you run that installed app, Finder alerts you to the fact that it was downloaded from the Internet, asking you to confirm your actions.

This is because the installer and the app that it installs have a hidden flag set in their extended attributes indicating that they are quarantined. This flag is cleared once you have run the app the first time, so does not trouble you again. However items you have installed from the (Mac) App Store do not show this behaviour, being deemed safe to use without passing through quarantine.

Code signatures

Gatekeeper in Yosemite extends this process of quarantine. Almost all apps should now have code signatures embedded in them: to check this, select Apple’s Mail app in the Finder and Control-click for the Finder’s contextual menu. There select Show Package Contents, so that the app appears as a single folder named Contents: open that and you will see a folder named _CodeSignature, containing Mail’s code signature.

Code signatures are neither new to the Mac, nor a recent innovation. However left to their own devices developers have been slow to sign their products, and as a consequence OS X has not been able to use them to enhance security. This was a similar situation to that on Windows, where code signing has had limited impact on its malware jungle.

Signatures consist of two parts: a secure certificate that identifies the developer against an approved whitelist (maintained by Apple and kept locally on each Mac), and a hash that verifies the integrity of the contents of the installer or app; if anything tampers with those contents the hash is invalidated.

Gatekeeper’s settings

Leave Gatekeeper at its default setting, and you are allowed to install and run apps from the App Store and from elsewhere with valid code signatures; all this does is alters what happens to installers and apps in quarantine.

Try to run an installer and Gatekeeper checks whether it has a valid code signature. If it does not, you will not be able to run the installer normally; if it does, then it works as previously. Once the app is installed, but remains in quarantine, when you try to launch it Gatekeeper checks the app’s code signature. Again, if the signature is missing, you will not be allowed to run it in the normal way; if it is present and valid, the app will run as intended, and is released from quarantine.

A few remaining developers have not yet started signing their installers and applications. You can still install and run these, either by setting Gatekeeper to its off position (allowing all apps to run), or by opening them using the Finder’s contextual menu command Open. In the latter case you will be confronted with a further warning, through which you can click, run the app, and bring it out of quarantine.

You should not leave Gatekeeper turned off, allowing all apps to install and run, as that would leave your Mac vulnerable to malware. If using the contextual menu is temporarily onerous, remind yourself to turn Gatekeeper back on as soon as possible.

Behaviour

As of Mountain Lion and Yosemite, once an installer or app is out of quarantine, Gatekeeper will not trouble you over it again. Neither will Gatekeeper fret about installers and apps that you have not downloaded, but copied across, say, from an optical disk or USB memory stick, as they do not go into quarantine. Furthermore at present Gatekeeper does not keep a watch on plug-ins or other extensions: there is nothing to stop an app from running an old or dangerous plug-in.

Gatekeeper is capable of a great deal more than this fairly simple and unobtrusive task. When read from apps, signatures are added to a new SecurityPolicy database kept in /var/db, and Gatekeeper consults that to help validate code signatures. Individual apps can have different security policies, something that can currently only be set using the ‘spctl’ command in Terminal’s command shell.

For the moment Yosemite uses Gatekeeper to guard against the most obvious threat: that of installing and running Trojans.

As the threat landscape changes during the lifetime of OS X 10.10, we could see security updates to modify Gatekeeper’s rules and behaviour, to address any the changing threat. If malware authors start pushing out malicious Safari plug-ins, for instance, it would be relatively simple for Apple to put Gatekeeper in their way. However Apple has spared us such rigours for the moment, as few of our current plug-ins have code signatures, so the impact on users could outweigh the current benefits.

The Sandbox

Code signing is a requirement for the other main security protection offered by all App Store apps, the App Sandbox, and often confused with it.

Apps that run with the Sandbox are isolated from other apps, protecting them and OS X from interference by malware. They are limited in the sensitive features of OS X that they access: this does not mean that they cannot do potentially risky things like look up passwords and access the Internet directly, but that they are specifically authorised to do what they need to do.

The App Sandbox ensures that an app that is not licensed to open your keychain or access USB devices cannot do so. Sandboxed apps are also restricted as to which folders they can use for storing preferences and other files.

Protection

For the user, the combination of code signing, the App Sandbox, and Yosemite’s generally fastidious security features should bring high assurance that apps obtained from the App Store (and third party sources meeting the same standards) are not malicious, and cannot be subverted by malware.

So long as you only run the current release of OS X and such resilient apps, you should have no fear of Trojans or other attacks. However there remains sufficient flexibility to treat Mac users as grown-ups: we can still install and use software development environments, access Terminal’s shell commands and scripts, and do other powerful but potentially dangerous things.

Yosemite leaves you in charge; although in some other respects iOS and OS X may be converging, there are no plans to lock OS X down in the way that iOS is – nor is there any benefit in doing so.

Some software developers have expressed their fears that Apple is taking complete control of the Mac platform, as it will have the ability to repudiate the authority of code signatures, and lock down the functionality of products. This position of power could, they claim, be abused to damage those whose products compete against Apple’s, and could disadvantage independent developers who do not wish to sell through the App Store.

Any allegations of such abuse would of course be open to legal action, and in the US at least could put Apple in breach of anti-trust legislation, something that would threaten its continuing corporate integrity. However Apple does need to come up with a robust and fair process for dealing with any disputes with developers.

Achilles heel

Although no malware has yet been found which penetrates Gatekeeper, particularly in combination with a sandbox, this year security experts, notably Patrick Wardle of Synack, have discovered weaknesses in its current implementation.

These are mainly in the executable code that is not properly protected by Gatekeeper, in the form of dynamic libraries or dylibs. Apps and other processes can load dylibs which are kept elsewhere and not under their immediate control. There is a mechanism whereby a malware author could provide a malicious dylib, most probably in a Disk Image or compressed archive (e.g. Zip) file, which you could inadvertently install on your Mac.

Because these dylibs do not have to be signed or subjected to Gatekeeper’s scrutiny, Patrick has shown how a user could inadvertently make a malware dylib available to an existing app, and thus could subvert that app. You can read more about this here, which also links to Patrick’s security research if you want full details from the expert himself.

It is also important to appreciate that this mechanism is ideal for an attacker to get you to install persistent malware, which sits there quietly sending back details of financial transactions or other very sensitive information. It opens up OS X to the sort of malevolent hacks that Windows has suffered in DLL hijacking: not an entertaining prospect at all.

We all hope that Apple will come up with a fix for the Gatekeeper system which will address this issue. In the interim the best advice is not to put yourself at risk by downloading any code-containing files unless you can be confident that they are genuine and free from malware. For updates and apps, and potentially enticing plug-ins and enhancements, that means downloading them over a secure network connection (HTTPS for instance) with a trusted website, or the secure connections provided by the App Store, Adobe, and similar systems.

Hopefully Gatekeeper and the sandbox will resume their intended high level of protection very soon.

Technique: Signed Apps

Installing and using properly signed apps has barely changed in Yosemite, if you leave Gatekeeper’s security setting at the default, to trust apps from the App Store and others that are correctly signed. Those that you obtain from the App Store install without any further ado, and run straight off.

Installers and packages downloaded from other sites do undergo proper checking by Gatekeeper. To get past those checks, the developer needs to have signed both the download and the app within it. When you try to install the app, Gatekeeper checks the installation signature, and if that is acceptable, lets you install it as normal.

The first time that you run the app, when it is still in quarantine, Gatekeeper checks its code signature when you have confirmed in the usual alert that you do want to run the app. If everything is in order, the app is brought out of quarantine so that it will not undergo such checks again, and it is released from Gatekeeper’s watchful eye.

These checks only apply to installation packages and the apps within them which have been downloaded from the Internet, as that origin marks them out and puts them in quarantine. Although almost every app and tool that can download from the Internet works with Gatekeeper, a few less widely used tools such as curl may bypass it.

Installers, packages, and apps that arrive on your Mac from other sources such as optical media, USB memory sticks, and copied across your network, do not pass into quarantine either. However if the files they contain were marked as being in quarantine and the transfer preserves that information, they will still be subject to Gatekeeper’s checks when you first open them. Some media such as memory sticks may not normally preserve quarantine flags.

Technique: Unsigned is Untrusted

The most common problem that you may encounter with Gatekeeper is that of the unsigned download, and unsigned app within. If you have changed your Gatekeeper security setting in the Security pane to allow all apps to be run, then you should still see the normal warnings that have resulted from opening quarantined items since OS X 10.5. However you can click straight through those, the app will install, and run with no more than a further warning.

With your Gatekeeper settings left at their default, missing code signatures from an item in quarantine will be intercepted by Gatekeeper. Double-click the app, for instance, and you will be politely informed that it cannot be opened.

You should now think carefully whether you do really want to trust this app: if you are confident that its missing code signature is acceptable, and does not indicate that it could be malware, you can select the app icon in the Finder, open the contextual menu by right- or Control-click and hold, and choose the Open command in that.

Gatekeeper will offer you one last warning alert, but within that you will now be offered the option to open the app. Once you have done so, it will come out of quarantine and can then be started normally as if it were signed.

If there is a problem with a code signature – perhaps that developer is no longer trusted by Apple, the signature has been stolen from another app, or the integrity of the contents of the app are in doubt – Gatekeeper offers no option to open the app regardless. You should remove the app immediately, and treat it as malware. It is possible that your download became corrupted, but that is an issue that you should report to the developer as quickly as possible.

Tools: DHS and Knock Knock

Gatekeeper’s current imperfection should make you concerned enough to keep a careful watch on your Mac for the time being, at least.

Patrick Wardle has provided two important new tools to help this, dylib Hijack Scanner (DHS) and Knock Knock. Both are free from his Objective-See site.

DHS checks all the applications and other executable code on your Mac which could be vulnerable to dylib hijacking, bypassing Gatekeeper. It reports back those applications which may be already hijacked, and those which appear vulnerable to attack. You can perform just a quick scan of those apps which are running, or an in-depth check of every app that it can find.

Knock Knock tackles the other side of the vulnerability, by checking executable code which could have been altered following an attack, in Authorization Plugins, Browser Extensions, Kernel Extensions (KEXTs), Launch Items, Library Inserts, Login Items, and Spotlight Importers. For each it checks whether it is correctly signed, and reports its rating in the Virus-Total database.

I have also provided a couple of simple, free AppleScript droplet apps, SigCheck and DylibScan; you can download them (via HTTPS!) here. DylibScan lists apps, frameworks and dylibs, and is useful for screening suspect Disk Images, folders, etc. SigCheck asks Gatekeeper to check the signatures of apps, frameworks, and dylibs in apps, mounted Disk Images, folders, etc., and is the next line in checking suspect items.

Rewritten from the original, which was first published in MacUser volume 28 issue 21, 2012.