Controlling processes and environments

When you run a process, such as a command tool either from a shell such as zsh or bash, or from within an app, there are two ways in which you control what that process does: the arguments passed to it when it’s launched, and the environment in which it’s run. Recent discussions here about a language problem I encountered when running system_profiler within an app revealed that environments aren’t well understood. This article attempts to clarify, hopefully without making any glaring errors in the process (or environment!).

Launching a Process within an app

Compiled apps which need the services of a command tool or other Mach-O binary may do so using a Process (or NSProcess). Typical code for setting up and launching might run:
// create a new Process
let task = Process()
// get the default environment
var theEnvDict = ProcessInfo.processInfo.environment
// you can add or change values in that environment as you wish,
// then set the Process’s environment
task.environment = theEnvDict
// specify the full path of the command tool to be launched
task.launchPath = "/usr/sbin/system_profiler"
// set its argument as an array of strings
task.arguments = ["SPiBridgeDataType"]
// set up a Pipe for stdout
let outPipe = Pipe() task.standardOutput = outPipe
// launch the task
task.launch()
// read stdout as data
let fileHandle = outPipe.fileHandleForReading let data = fileHandle.readDataToEndOfFile()
// wait until the Process completes
task.waitUntilExit()
// check its returned status, handling errors, etc.
let status = task.terminationStatus
which runs what in Terminal would have been entered as
system_profiler SPiBridgeDataType

We have to specify the arguments in accordance with that tool’s man page, but how does the environment work?

As elsewhere, the environment is a collection of settings, most easily viewed as a dictionary of key-value pairs. This is determined by the context in which that app is being run. For example, when you run an app in Xcode’s debug environment, that includes more than 30 environmental variables, which might fall to a dozen when that app is run outside Xcode. In the latter case, these might be
"__CF_USER_TEXT_ENCODING": "0x1F5:0x0:0x2" "SSH_AUTH_SOCK": "/private/tmp/com.apple.launchd.vCfYRNPYdD/Listeners" "SHELL": "/bin/zsh" "XPC_SERVICE_NAME": "application.co.eclecticlight.Mints.139115359.146228395" "TMPDIR": "/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/" "XPC_FLAGS": "0x0" "HOME": "/Users/hoakley" "USER": "hoakley" "PATH": "/usr/bin:/bin:/usr/sbin:/sbin" "LOGNAME": "hoakley" "COMMAND_MODE": "unix2003" "__CFBundleIdentifier": "co.eclecticlight.Mints"

In each case, the pair is given with the string key first, followed by a colon, and its value is another string. For example,
"HOME": "/Users/hoakley"
would be the same as setting the following in a normal shell environment:
HOME=/Users/hoakley

The command tool can access the variables set in its own environment at launch. For example, a command which wants to know the name of the current user can retrieve that from the USER setting in its environment, and the Mac SDK provides convenience methods for obtaining such information. However, Mach-O binaries are usually quite simple beasts, and if they don’t make use of a variable there’s nothing that macOS will do about it.

A good example here is setting languages or locales. You can set the well-known environmental variable LANG to en_GB.UTF-8, but there’s nothing that automatically happens as a result. A command tool can ignore that, or it can respect it by converting any strings which it returns into those selected from collections of localised strings.

One variable which is normally set is the User Text Encoding, in __CF_USER_TEXT_ENCODING, which I examine next.

CFUserTextEncoding

At the top level of every user’s Home folder is a hidden file .CFUserTextEncoding containing a format string whose purpose is explained in this Tech Note. This contains a string of the form M:0:N, where each of the three numbers is usually given in hexadecimal, for example as 0x1F5:0x0:0x2, which specifies that user number 501 (hex 1F5) uses encoding number 2. Normally, in the .CFUserTextEncoding file the user number is left as 0x0 for the system to convert into that user’s UID.

This doesn’t set the language or locale for that user, but the encoding used by CFString, and the full list of supported encodings is provided in Apple’s reference documentation for CFString. However, this value now appears to be largely ignored, and you’ll encounter users whose assigned encodings might be 0x2, which is supposed to be macChineseTrad, or 0x5 for macHebrew, which bear no relation to their language setting. As UTF-8 is the string encoding used throughout macOS, it’s hard to see what purpose this setting now serves, but you’ll still see it in environments as __CF_USER_TEXT_ENCODING.

Running a command tool in Terminal

When you run a command tool in a shell, such as zsh, the default in Terminal from macOS 10.15 onwards, the same principles apply with the addition of the environment settings for the shell. You can inspect those using the command env, which should return a list similar to my current environment:
TERM_PROGRAM=Apple_Terminal SHELL=/bin/zsh TERM=xterm-256color TMPDIR=/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/ TERM_PROGRAM_VERSION=440 TERM_SESSION_ID=E005CA40-5EEB-4EA3-99F0-C8A364F72E8E USER=hoakley SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.vCfYRNPYdD/Listeners PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/go/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS __CFBundleIdentifier=com.apple.Terminal PWD=/Users/hoakley/Documents XPC_FLAGS=0x0 XPC_SERVICE_NAME=0 SHLVL=1 HOME=/Users/hoakley LOGNAME=hoakley OLDPWD=/Users/hoakley/Documents LANG=en_GB.UTF-8 _=/usr/bin/env

An important addition here are the paths set in PATH, which save you from having to type the full path to well-known commands, and setting the PATH in your environment is one of the commonest reasons for changing it. You can also inspect language settings using the locale command, which lists each of the language-related variables, such as
LANG="en_GB.UTF-8" LC_COLLATE="en_GB.UTF-8" LC_CTYPE="en_GB.UTF-8" LC_MESSAGES="en_GB.UTF-8" LC_MONETARY="en_GB.UTF-8" LC_NUMERIC="en_GB.UTF-8" LC_TIME="en_GB.UTF-8" LC_ALL=

As with other environmental variables, these are accessible to command tools, but don’t in themselves change anything. It’s up to the command tool to look up its environment and make any changes it deems necessary in response to a setting such as the language.

Running an app

Running an app from the command line is quite different from a command tool. Apps don’t simply parse the arguments passed from the command line, as they invariably rely on frameworks like AppKit and other code to switch to using translated strings, for example. Accordingly, there are two relevant options which the user can invoke, -AppleLanguages and -AppleLocale. For example, to launch an app in German regardless of the system language setting, use a command such as
/Applications/MyApp.app/Contents/MacOS/MyApp -AppleLanguages "(de)"

Of course, this only works if that specific app has the appropriate localisation requested, and is either handled by its frameworks or by custom code in the app itself.

When you launch an app from the Finder, the only way you have of setting it to use different behaviours and variables is through its preferences, otherwise it simply uses those set by the system. Some apps allow you to set their language and other features using a command such as
defaults write com.apple.TextEdit AppleLanguages '("en-US")'
Unfortunately, as with most preference settings, this may well not be documented, so you’ll need to experiment to discover what works. However, for users this provides a convenient way of running an app using a language other than that set by the system, when it’s supported.

Call chains

When you run a simple command tool directly, there’s little to go wrong: you pass it an environment, and that determines how it behaves. This becomes more complex when the tool calls on other code to handle your command. I’ve been looking at how this works with system_profiler in particular.

system_profiler is surprisingly complex. The command tool in /usr/sbin/ turns out to be a small stub which relies on calling helper tools stored in the /System/Library/SystemProfiler folder as .spreporter bundles. Each of those contains another Mach-O executable complete with its own localised strings, and in some cases such as SPiBridgeReporter.spreporter there are also XPC services, which in turn have their localised strings.

You can normally trace a call chain by browsing the log from the moment that the command is invoked. Here is an excerpt which traces the call chain for the system_profiler command tool, either launched from within an app using Process() or run from Terminal’s command line, on an M1 Mac.

The system_profiler command tool next dispatches a request for the helper tool in /System/Library/SystemProfiler:
0.544816 com.apple.SPSupport Reporting system_profiler SPSupport -[SPDocument reportForDataType:] -- Dispatching helperTool request for dataType SPiBridgeDataType. 0.545010 com.apple.SPSupport Reporting system_profiler SPSupport -[SPDocument _reportFromHelperToolForDataType:completionHandler:]_block_invoke -- Launching task to collect SPiBridgeDataType 0.551195 com.apple.SPSupport Reporting system_profiler SPSupport -[SPDocument _reportFromBundlesForDataType:] -- Starting task to collect SPiBridgeDataType

That helper tool in turn calls the XPC service iBridgeDiscovery, which is handled first by opendirectoryd:
0.552426 com.apple.opendirectoryd session opendirectoryd opendirectoryd UID: 501, EUID: 501, GID: 20, EGID: 20, PID: 1103, PROC: system_profiler RPC: getpwuid, Module: SystemCache, rpc_version: 2, uid: 501

The XPC service goes to work:
0.623138 com.apple.CoreAnalytics client iBridgeDiscovery CoreAnalytics Daemon status changed from 0 to 1 0.623262 com.apple.analyticsd xpc analyticsd analyticsd Setting new client connection handler. 169 active connections 0.623390 com.apple.CoreAnalytics client iBridgeDiscovery CoreAnalytics Received configuration update from daemon and there was no CoreAnalyticsFramework external config. 0.623390 com.apple.CoreAnalytics client iBridgeDiscovery CoreAnalytics Received configuration update from daemon (initial) 0.626248 com.apple.xpc.remote RemoteServiceDiscovery iBridgeDiscovery RemoteServiceDiscovery Starting browsing: <private> 0.626371 com.apple.xpc.remote RemoteServiceDiscovery iBridgeDiscovery RemoteServiceDiscovery Started browsing. 0.626380 com.apple.xpc.remote RemoteServiceDiscovery iBridgeDiscovery RemoteServiceDiscovery Connection invalidated 0.626391 com.apple.xpc.remote RemoteServiceDiscovery iBridgeDiscovery RemoteServiceDiscovery Connection invalidated

With the result back from the XPC service, that service is exited:
0.626472 com.apple.SPSupport Reporting system_profiler SPSupport -[SPDocument _reportFromBundlesForDataType:] -- Completed task to collect SPiBridgeDataType. 0.633413 com.apple.opendirectoryd session opendirectoryd opendirectoryd PID: 1103, Client: 'system_profiler', exited with 0 session(s), 0 node(s) and 0 active request(s) 0.633719 com.apple.opendirectoryd session opendirectoryd opendirectoryd PID: 1104, Client: 'AppleSiliconDiscovery', exited with 0 session(s), 0 node(s) and 0 active request(s)

Finally, the command tool has got the output data, returns it, and exits:
0.697375 com.apple.SPSupport Reporting system_profiler SPSupport -[SPDocument _reportFromHelperToolForDataType:completionHandler:]_block_invoke_2 -- Completed task to collect SPiBridgeDataType. terminationStatus is 0 0.733831 com.apple.opendirectoryd session opendirectoryd opendirectoryd PID: 1102, Client: 'system_profiler', exited with 0 session(s), 0 node(s) and 0 active request(s)

The calling app, in this case a special test version of Mints, has its own environment. When it calls system_profiler, it passes an environment which the app controls. system_profiler runs in that environment, but then invokes its helper tool, which has its own environment, and that in turn uses an XPC service executing in its own environment, which isn’t controlled by system_profiler, to gather and return the information. As that service has its own localised strings, it can perform conversion of the text in its response without any knowledge of the environment in which system_profiler is running.

Complex call chains like this make control of the environment far more difficult.

4Comments

Add yours

1

nudge on December 10, 2020 at 2:01 pm

Other than the first process, all processes are children who inherit certain traits (environment) from their parents, as in real life. This inheritance aspect of processes was explained in Apple’s old Tech Note about launch agents and daemons, iirc. How does that fit in with what you are saying here ?

LikeLiked by 2 people
- 2
  
  hoakley on December 10, 2020 at 2:20 pm
  
  That’s exactly what I’m spelling out in detail here.
  When you launch a command tool from within an app, for instance using the code that I give, that tool inherits the environment passed on by the app which is launching it. You can of course modify that, as I show. Similarly, when you run a command tool from Terminal, it inherits the environment of the shell, now by default zsh, and again you can modify that by changing environmental variables too. An app is a bit more complex, as it’s launched through LaunchServices, but it too gets its own environment. An XPC service is different again.
  But the environment doesn’t require the tool to do anything, like respect the environmental settings for language. If the environment sets the LANG variable to any specific language, it doesn’t mean that all tools will be launched in that language, even if they could support it.
  Howard.
  
  LikeLiked by 1 person
  - 3
    
    nudge on December 10, 2020 at 4:36 pm
    
    I see no mention of inheritance, you use the word ‘passing’ which sounds optional. Anyway, what originally caught my interest was the first article where you explained that system_profiler calls upon other ‘helper’ services that don’t appear to respect locale / language settings. If these helper services are running under root then I would expect their locale to be read from /Library/Preferences/.GlobalPreferences.plist (or perhaps /var/root/Library/Preferences/.GlobalPreferences.plist or even nvram) during system initialisation and then inherited from there on down the chain. It’s not clear to me whether or not you’ve explored changing locale for the root processes during your testing. You seem quite sure but please forgive me for wanting more details about this before fully accepting your findings.
    
    LikeLiked by 2 people
    - 4
      
      hoakley on December 10, 2020 at 5:17 pm
      
      Thank you.
      Yes, inheritance is a fine term which is often used for this. However, many of us associate inheritance with its OOP connotations, which can be confusing. It also doesn’t fit easily with what happens with (NS)Process, where you do actually set a variable Process.environment which determines the environment of the child process. The default is simply to pass the inherited environment, but you can set any environment you like.
      Regarding the XPC service, you’re now not referring to an environment, but system preferences, which don’t appear in the environment. Instead, a binary would use (NS)Locale to check those, in order to localise according to the system setting. Changing the system Locale in the way that you suggest would change it for all processes which checked the locale, an unforgivable sin for any developer to perpetrate in an app. And I can’t see any way of creating a ‘local’ locale in which the system would launch a single XPC service. Maybe I’m just missing something obvious?
      This isn’t a question of accepting anyone’s “findings”. What I have described above is my understanding of how this works in macOS. I’m very happy to be corrected where I’m wrong.
      Howard.
      
      LikeLiked by 1 person

Share this:

Related