Service with a Smile

The basic version of Lingon available from the App Store gives you limited control over services.

Macs are increasingly reliant on background services, and increasingly vulnerable to their problems. Mastering the services on your Mac will help make it faster and more stable.

Quit all applications and start up Activity Monitor, the utility that details processes running on your Mac. You will now see a list of over 50 such processes.

Order the list by process identity (PID), and you will see, starting from the lowest PIDs, kernel_task (Mac OS X kernel, the heart of the operating system), launchd, and plenty of other processes with names ending in ‘d’ indicating that they are daemons or background services.

Even when largely inactive, these processes take up memory and some processor time, as shown in Activity Monitor’s window. Those that are the leftovers from old and otherwise uninstalled products can crash, conflict with other software, and at worst make mayhem. Open Console a minute or two after your Mac has restarted and you will see their footprints over the logs after that restart, and sometimes recurrent errors or warnings when they have gone awry.

launchd

Most of these services and daemons are loaded automatically when your Mac starts up, these days courtesy of the master launcher of them all, launchd. The first software to load during startup is the kernel, which runs up whilst the screen is grey. Low-level drivers and kernel extensions are loaded into the kernel, and the startup volume is made fully accessible.

The launchd service kicks in when the screen changes to show a progress bar, then loads up everything that is needed before user login. These services are normally run as root, or sometimes using special-purpose user identities. They used to include items listed in /Library/Preferences/com.apple.systemloginitems.plist in older versions of OS X but no longer do so, and the last to be launched in this phase is loginwindow, which displays the user login dialog (when the Mac is not configured to log into a user account automatically).

Once you have logged in, your working environment is loaded up and set according to your preferences, that you have configured using System Preference items. Your desktop picture is loaded, as are the Finder and Dock, after which loginwindow launches those applications specified in the Login Items list in the Accounts pane (listed in ~/Library/Preferences/loginwindow.plist), and launchd works through the list of services that run under that user’s account.

Diagnosing startup problems

Knowing what loads when is a great help in solving startup problems. Freezing before the login dialog appears is rarer now than in the past, and often the result of a fatal error when launchd is trying to load services and other items prior to user login, those most commonly run as the root user, and located in the /System/Library or /Library folders.

Crashes and other problems that strike after logging in often result from services, login items, and similar software that are loaded from the Library folder in your Home folder. You can circumvent the latter by logging in as a different user, whose account does not contain the troublesome components. Starting up ‘clean’ (Safe Mode), with the Shift key held down, should block the loading of all software other than Apple products, and can enable startup to progress beyond an early ‘screen of death’, for instance.

Controlling services

The launchd service relies on XML property list (.plist) files that define which daemons, applications, scripts, and other items are run, and when. Mac OS X system daemons, loaded first, are held in /System/Library/LaunchDaemons, and system agents in /System/Library/LaunchAgents.

Additionally, Apple and third-party daemons and agents held in /Library/LaunchDaemons and /Library/LaunchAgents are loaded early, as are traditional items placed in /System/Library/StartupItems (normally empty now) and /Library/StartupItems (also normally empty). Those third-party products defined or held in folders within /Library are most likely to be responsible for clashes and problems, although some Apple updates in the past have caused spectacular issues within /System/Library.

Log entries will not only list the services concerned, but often link them with launchctl rather than launchd. launchctl is the active member of the partnership, submitting loading and unloading tasks to the launchd service, and generally controlling it. Although they normally work fine in their default configurations, servers in particular that run many services under heavy loads can run out of available processes. When they do, the System log contains errors such as
(com.apple.launchctl.Background): fork() failed, will try again in one second: Resource temporarily unavailable.

Provided that Mac has adequate hardware resources, in terms of processor power and memory, you may be able to raise the limit on the maximum number of processes allowed, and work around such errors by creating a configuration file for launchd at /etc/launchd.conf containing a line like
limit maxproc 1000 2000
This can be tuned further by progressively reducing the values of 1000 and 2000. This might require kernel tuning to adjust the overall limits on processes, as detailed in man sysctl and sysctl.conf: that starts to get into tiger country, and there does not seem to be much reliable documentation to help!

Property lists

Services – divided by Apple into daemons (system-wide) and agents (per-user) – have job labels that look like reversed Internet domain names, such as com.apple.sshd, and are associated with launchd property list files bearing the same name plus .plist, thus com.apple.sshd.plist in this example.

Keys that are required in the property list include the service’s label, and a true or false value for whether the service is disabled. Disabled services are not submitted by launchctl to launchd, providing a simple mechanism for turning them on and off.

Normally launchd only starts most service jobs when there is demand for them, which ensures that your Mac is not cluttered all the time with every service that might ever be needed. The KeepAlive key is used by services that, should they terminate or crash, need to be restarted. When set to true, or a dictionary of conditions is provided for the key, launchd will try to keep the service alive if the conditions are met.

This is most commonly used for essential services being provided in OS X Server, but can equally be used in client installations when needed.

cron replaced

An elaborate system of keys with times and dates ensure that launchd can completely replace cron and traditional Unix devices for running tasks periodically.

Thus you can configure a service’s property list so that it is run at fixed time intervals, such as the tasks that run automatically every day, week, and month to perform system housekeeping, including log rotation. Indeed, cron is now run by launchd and has its own property list, to ensure backward compatibility. Unfortunately some older releases of Mac OS X have suffered issues in the reliability of running periodic tasks using launchd, and you may have to resort to cron and its crontab system on those.

If you have altered service configurations, for instance in their property list files, then at the very least you are likely to need to restart those services for changes to take effect. However in most cases it is wisest to restart your Mac, so that launchctl and launchd can start afresh with their new settings.

Managing services and solving problems in them has appeared to be a Black Art, largely because launchd seems complex and dangerous. Given reasonable care, and avoidance of tinkering with services from /System/Library, it is actually far more friendly and accessible.

Technique: Tracing Services

Understand how launchd works and it is not hard to work out which services are which, and to diagnose their ailments. Key tools are Console, for browsing logs, and Activity Monitor, for listing all processes that are running.

Restart, open Console, and set it to show All Messages. Scroll back through the log to the last entry of systemShutdown true followed shortly afterwards by SHUTDOWN_TIME: these are among the last entries before shutdown.

You will then see a time gap of 10 or 20 seconds before the first entries are made in the startup sequence, including one giving the Darwin Kernel Version number. You can then trace various entries relating to the kernel and its extensions (KEXTs) before launchd proudly announces that it has started up.

Another series of entries follows, many of which are services and other items being started by the combination of launchctl and launchd, first from /System/Library, then from /Library. Eventually you will see entries concerning the Login Window, culminating in the time at which you successfully logged in as a user, recorded by the line “Login Window done”.

After that, entries come from processes being launched as Login Items (specific to that user), and launchd services controlled by entries in ~/Library.

Once you have traced these through your logs, open Activity Monitor, set its top popup menu to list all processes, and order them by PID, with 0 for the kernel_task at the top. As PIDs are issued in numeric order, you should now be able to work up the numbers, matching some processes listed in Activity Monitor with their corresponding log entries in Console.

Activity Monitor listing of the first processes started, beginning with kernel_task and launchd.
Activity Monitor listing of the first processes started, beginning with kernel_task and launchd.

PID 1 is launchd itself, following which numbers skip to 50 for the main system services, up to WindowServer, then through another series of services up to UserEventAgent. Higher PID numbers are then generally owned by the user rather than root.

Activity Monitor listing of user processes, starting with UserEventAgent, which are mainly run as the user rather than root.
Activity Monitor listing of user processes, starting with UserEventAgent, which are mainly run as the user rather than root.

Tool: Lingon

Knowing now how you control services using their property list files, it is puzzling that Apple has not provided a convenient way to edit those plists. OS X Wizards can of course use their favourite text editor, such as BBEdit or even Terminal-based editors.

Long a freeware tool developed by Peter Borg, Lingon is the only tool that helps you understand the maze of property lists that your Mac relies on. It is available in two different versions: that from the App Store is cheaper and should suffice for most users, but the raw power comes with Lingon X 2.0, which must run outside the Sandbox, and thus is purchased from Peter Borg direct.

The basic version of Lingon available from the App Store gives you limited control over services.
The basic version of Lingon available from the App Store gives you limited control over services.

Once the X 2.0 version is installed and opened, you will see daemons and agents classified by their property lists into three categories: those running as root, those for all users, and those for ‘me’, the current user.

Lingon X 2.0 gives you complete control over services, in this much longer listing.
Lingon X 2.0 gives you complete control over services, in this much longer listing.

You can enable or disable each service, change the command to be run, and set its major launchd controls. The latter include whether it is run on demand, offering options such as run when a volume is mounted or on file/folder modification.

Lingon coupled with launchd can be used as a cron replacement, as Basic Mode also allows you to set periodic running by time or date intervals.

Updated from the original, which was first published in MacUser volume 27 issue 10, 2011.