Last Week on My Mac: The problems of structured data

Many years ago I had a problem with medical reports we had written for printing on paper. As we were transitioning from physical records to an electronic database, I needed to transfer several thousand laid-out reports from Adobe InDesign to the database I was building in Claris FileMaker. I discovered I could export the original reports in IDML format, extract individual fields such as patient names and dates of birth, and import those to populate fresh records in FileMaker. Although this went remarkably smoothly, because the main body of the report was unstructured, I was unable to separate the test results each contained. It was another reminder of the limitations of unstructured data.

Perhaps the greatest strength of Apple’s Unified log is that its contents are structured, although most seem to access it through the log command, which discards all structure. To see the difference, compare what you might read in a single entry using log show:
2026-07-06 22:44:00.689232+0100 69455 General error SpotlightServices Spotlight 726 com.apple.SpotlightServices 11610 [SSQueryParser][POMMES][qid=3] Got an empty SPQUParse
with the same entry expressed in JSON:
"date":"2026-07-06 22:44:00.689232+0100", "activityIdentifier":"69455", "category":"General", "level":"error", "sender":"SpotlightServices", "process":"Spotlight", "processIdentifier":"726", "subsystem":"com.apple.SpotlightServices", "threadIdentifier":"11610", "composedMessage":"[SSQueryParser][POMMES][qid=3] Got an empty SPQUParse", "type":1
where each component is identified separately. This allows you to filter entries according to subsystem, to search for those recorded as errors, and to count how many messages refer to SPQUParse over a given period – none of which can be done with unstructured plain text entries.

If JSON is a little heavyweight and you want to analyse entries using a spreadsheet or database, you could export them in CSV format instead, as that too retains their structure by listing the value of each field in fixed order, separated by commas.

When data is inherently structured, as Unified log entries are, it will normally retain that when being manipulated in an app, using data structures in memory. For LogUI, each of the thousands of log entries it reads and displays consists of a LogEntry structure with fields for the keys listed in the JSON files it writes, every entry being given a unique identity with a UUID. Writing and reading those requires just a few lines of Swift to encode or decode them using built-in functions.

It’s when they’re saved to disk storage that the problems begin for structured data, though. As documents, the app that creates and accesses them can only do so when they’re open in the app. As it has no control over where those files are, it can’t replace Spotlight’s indexing and search without a great deal of additional code, and imposing constraints on where they can be stored. What works well for apps managing their own database (in the singular) becomes impractical for those working with multiple documents containing similar structures.

At the same time, Spotlight has no insight into their data fields, and isn’t intended to store their contents in any structured way. If it were to try to index their contents, it would do so by discarding their structure and storing them using the same unstructured plain text as is generated by log show, much as it already does for HTML and PDF contents.

The same problem has been playing out with standard types of documents containing text, such as Rich Text, HTML and PDF. Early attempts to structure text semantically using the grandfather of markup languages, SGML, were generally supplanted by markup based on appearance and layout rather than semantics, and that’s now the basis for much of the structure in text files. There are some surviving exceptions in Donald Knuth’s TeX and its relatives, the Text Encoding Initiative, and in XML, but even those seem to be falling by the wayside.

Trying to build such marked-up text into structures with meaning is thus futile, so for search purposes all markup is removed to leave plain text content that fails to distinguish chapters, sections, footnotes, bibliographies, or any of the rich structural aids that we see on our displays and in print. Spotlight is unable to look for references containing specific author names, for example, except for the limited metadata that can be saved for each file, but seldom is. Even basic structural features such as headings can’t be distinguished, so there’s no way to look for a book chapter title containing a specified term, for example.

This isn’t confined to Spotlight either. Although a great deal of effort has been put into the indexing and search of plain unstructured text, few systems can make good use of structure where it’s available, and they tend to be narrow in their scope. For everything else we’re stuck with the lowest common denominator, and discarding the rest.

2Comments

Add yours

1

jzonedotcom on July 19, 2026 at 4:47 pm

Although this comment is might be slightly off topic in that it focuses on structured -vs- unstructured data more than it focus on Spotlight, metadata, and Search, I just couldn’t stop thinking about my great love for the Unix / CLI “1-Liner” as I read this post.

Puzzling through a Unix pipeline, battling with it at times to emerge victorious with a successful & reusable 1-Liner demands love for Unix, familiarity with a plethora of tools, commands & their flags (or options), plus persistence as well as a pleasing amount of creativity & mental flexibility. Perhaps the most critical of these skills is a level of comfort with regular expressions. Proficiency at quickly scanning & searching Man Pages helps a lot too.

Many of the curveballs that one must negotiate are a consequence of the unstructured text output flowing from one command to the next. In this area tools like “grep”, “cut” (often using multiple iterations with a different delimiter each time), “tr” (simple but extremely useful), in addition to “sed” & “awk” are all indispensable.

There is something about this process that I find super enjoyable, rewarding and fun.

Now for the “structured part”. While reading this post and pondering my affinity for the CLI in general and 1-Liners specifically, I couldn’t help thinking about the Microsoft elephant in the room with my typical begrudging yet undeniable respect. Namely “Project Monad” or the codename for what became the impressive & venerable PowerShell – which was a profound rethink of structured -vs- unstructured data as it exists in the command line environment and led to an extremely effective & impressive new type of shell.

Specifically, PowerShell’s designers set out to leverage structured data from the ground up (in the form of .NET objects) which naturally replaces your familiar unstructured text-based pipeline with an object-oriented “hot-rod”. So structured data is baked in from the start and this decision leads first to a paradigm shift and then demands a careful, studied rethink of how to operate most effectively in this new world – moving away from manipulating unstructured text strings & ultimately to learning how to leverage the extremely powerful new capabilities of a pipeline based on structured data / .NET objects.

I will always maintain my love of the Unix CLI & Unix 1-Liners and I also am enamored with the ambitious design decisions that brought something really new to the CLI in the form of PowerShell. A tip of the hat to Microsoft engineers for this one, it’s evolutionary and it’s so much more than just changing the Unix forward-slash to a “Microsoft” backslash! ;-}

I do sincerely hope that I succeeded in demonstrating the relevance of this comment as it pertains to the structured -vs- unstructured data topic that permeated this latest edition of “Last Week on My Mac” 🤞

P.S. If I happened to pique your interest in PowerShell this Reddit link from the PowerShell Subreddit is a great place to start.

LikeLiked by 1 person
- 2
  
  hoakley on July 19, 2026 at 8:48 pm
  
  Thank you. Although that’s an interesting way to approach a command shell, I’m not sure how much it helps with the tasks you can’t simply script there. Among the most difficult are those of Spotlight, in indexing contents and searching them. That’s not something you can readily implement in command tools or even compiled languages like Swift – it requires extensive support in the OS.
  While it’s supremely simple, for example, to convert structured data in memory to JSON or CSV, there’s no standard way for a user who doesn’t write code to perform search operations on, say, one field within that structured data across multiple files. Something like find me all the log files that have the term SPQUParse in their composedMessage field.
  Given that I have many log extracts with over 100,000 entries in a single file, that’s not something you can do without indexing, and not something that Spotlight indexes do.
  Howard.
  
  LikeLiked by 1 person

Share this:

Related