LockRattler’s Help Book outlines how to address some of the problems which it can reveal, but here is some more detailed advice, which also applies to anyone using other tools, including those in Terminal.

SIP

If System Integrity Protection (SIP) is disabled, you will want to enable it as quickly as possible, so that all your system files are properly protected. The standard way to do that is to boot into Recovery mode, open Terminal, and there use the
csrutil enable
command. Once that is done, restart normally and check again using LockRattler.

In Sierra 10.12.2 and later, you may also be able to do this more conveniently, by opening Terminal and typing in
sudo csrutil clear
then entering your admin password. Restart, check with LockRattler, and if necessary resort to the standard method in Recovery mode if that didn’t work.

XProtect blacklists, security subsystem policy

If XProtect blacklists are shown as being disabled, or this returns an error, try opening Terminal and typing the following command
spctl --status
You can also get slightly fuller information with
spctl --status -v

If that also returns an error, or a report that the service is disabled, you can try enabling the whole system using
sudo spctl --master-enable
which then requires you to authenticate with your admin password.

These commands should return the standard success code of 0, if they deliver anything more than the text result. A code of 1 indicates that the operation failed, 2 that the arguments in the command were incorrect, and 3 that the operation was denied but no other error has occurred.

It is worth checking the General tab in your Security & Privacy pane: that should allow one of two options in Sierra, either App Store, or App Store and identified developers. If it reads Anywhere (or anything else) that will disable XProtect blacklists, something which Sierra is supposed not to permit. Set that control back to one of the two standard options and protection should be restored.

If you still cannot turn security assessment on, make a careful note of the above results and report them to Apple support. You may find yourself applying the latest Combo updater or re-installing macOS to try to restore their function. They should still work normally when FileVault is turned on, and FileVault does not make this part of your Mac’s security unnecessary.

FileVault disk encryption

If FileVault is shown as being off when you think it should be on, open the Security & Privacy pane, and select the FileVault tab. You will then need to click on the padlock icon to authenticate, following which you should be able to turn FileVault on. If this says that FileVault is on but LockRattler reports that it is not, try restarting and repeating. If there remains a conflict, try the Terminal command
fdesetup status

If you still cannot resolve the issue, make a note of the results of these, and report them to Apple support, or make an appointment at your nearest Genius Bar.

Out of date security data

If any of the version numbers are lower than those shown here for Sierra, follow the routines for automatic software updates below, to force an update and correct that. Updates don’t arrive instantly, but should be delivered within minutes or a couple of hours at most.

If any version numbers are missing or corrupt, then this suggests that you have disk corruption or file loss. Restart in Recovery mode, run Disk Utility, and there apply First Aid to your startup drive. You may end up having to reinstall macOS to correct this, or the latest Combo updater might do the trick.

Automatic software updates

If Software update shows that automatic check is not on, open the App Store pane. Ensure the Automatically check for updates and Install system data files and security updates items are ticked.

You can force an update by clicking on the Check Now button, and/or by typing in Terminal
sudo softwareupdate --background-critical
then entering your admin password at the prompt. In my experience, the command line tends to result in a more rapid response.

After action

With any security anomaly, you should ask carefully what might have resulted in that anomaly occurring. Have you recently installed any software from an untrusted source? Have you run anything which might not be completely legitimate? Could you have malware installed which has tampered with your Mac’s security systems, perhaps?

A high index of suspicion could save your data.