The first phase of many a doctoral research project is spent discovering the limitations of the techniques to be used. It’s an exciting time: I recall using Lego Technic constructions to move transducers through out-of-date human blood, and long days tuning the settings on equipment. At the end of that phase, you must have the confidence that what you’re about to do in your main study can be defended in front of your examiners.

Although many like to refer to the investigations they undertake on computers as research, and we’re falling over those describing themselves as security researchers, I see remarkably little attention being paid to techniques. Except as a generic excuse for choosing to ignore the evidence of others.

Phoning home

The first of my examples is over-reliance on evidence from software firewalls, as if they can be used to draw conclusions about the purpose of outgoing connections. As a tool, a software firewall can be invaluable for demonstrating connections made by malicious software, and for protecting the user from attempts at such connections. But some are too ready to jump to conclusions when processes that should be running anyway appear to make outgoing connections.

Like it or not, macOS is complex, and many of its features have come to rely on connections to services provided by Apple. From checking signing certificates on apps to recognising the faces of our friends in Photos, throughout the time our Macs are awake, they connect to remote servers. Add iCloud and you’ve guaranteed a wealth of reports from a firewall. Even running a VM without an Apple ID, and a bare minimum of apps, you’ll see macOS checking whether ancillary data for camera RAW images is up to date.

The only way to make any sense of these connections from macOS is to demonstrate their initiation on that Mac, and establish the purpose of that connection. Without that you will never know why.

Reversing

Perhaps the most common way to dismiss almost anything is to require reading of the source code, or disassembling/reversing of binaries. There’s an important assumption that, just because code can do something, that it not only will, but does so when you expect it to. I learned my lesson many years ago, when I wanted to use what was widely considered to be the state of art model of human thermoregulation.

This had started as a hobby project by a distinguished professor of chemical engineering, and just grew, in Fortran too. When I did manage to get a copy of the source code, it was the most mangled that I have ever read. When I started asking questions about some of the many suspect subroutines I discovered, I learned that its author had little insight into what it did, and which code had been superseded and was now redundant.

We’ve all encountered bugs where we expected particular code to be run, but it never seems to get near the processor. Even stepping through that code with a debugger can sometimes leave us baffled. The moral is simple: source and disassembled code can only reveal a potential, not define a reality. To discover what actually happens when that code runs, you have to observe it running. Otherwise most bugs couldn’t exist in the first place.

Unknowns

My last example is looking at core allocation in Apple silicon chips without measuring their frequency. This was one of the first lessons taught by M1 Pro and Max chips when they shipped in October 2021. Given that an M1 Max has only half the number of E cores as the basic M1, does that mean that background tasks, like Time Machine backups, normally run almost exclusively on E cores, will take twice the time on an M1 Mac as on a basic M1?

It was only by running synthetic benchmarks that it became clear that macOS set the frequency of E cores to ensure that didn’t happen. Background threads run on the four E cores in a basic M1 normally do so with those cores at half their maximum frequency. The same two or more threads running on the two E cores of a Pro or Max chip are run at maximum frequency, ensuring that they complete in the same time or less.

Sophisticated instrumentation to obtain fine detail on core allocation doesn’t provide measurements of core frequency, which is only available from the powermetrics command tool. It’s all very well knowing that a few threads might be run on P cores as well, but missing the bigger picture of frequency and in-core performance makes that largely irrelevant. If you don’t measure something that could be influential if not decisive, then you can’t interpret what you find.

Observation and experiment

Investigating what happens on a computer is a balance between observation and experiment. While careful measurements of what happens in the real world are essential, modern operating systems are so complex that it’s often necessary to repeat those observations in a simplified experiment.

With the ready availability of lightweight virtual machines running macOS on Apple silicon, this has become much easier, and they’re usually my first choice now. They run without the complications brought by an Apple ID, and lack Wi-Fi too, keeping them free from distractions and their logs much cleaner as a result. When you can’t see the wood from the trees, an experiment can be decisive.

Knowledge

Three maxims were inscribed on the Temple of Delphi for the eternal guidance of those who study macOS: the most important read γνῶθι σεαυτόν (gnōthi seautón), know yourself, including your methods and their limitations; nothing to excess, and certainty brings insanity. At least we can rest assured that we can never be certain of anything any more.