So far in this series, I have only considered Spotlight search accessed at a system level, typically using the Finder and third-party apps such as HoudahSpot. Recent releases of macOS, Catalina and Big Sur in particular, have changed search behaviour in certain apps such as Mail and Notes. This is because of the introduction in macOS of what Apple calls Core Spotlight, which is the basis for search in iOS.

However, Apple doesn’t appear to have explained to users the differences between existing macOS Spotlight and Core Spotlight, and its developer documentation is common to iOS, which doesn’t have anything comparable to regular Spotlight in macOS. We’re therefore left to guess the limitations of Core Spotlight and in-app search where it uses Core Spotlight.

As far as I can tell, in-app search using Core Spotlight uses the same indexing system as regular Spotlight, but access to content indexed for Core Spotlight is strictly limited to the app which controls it and rare exceptions including Spotlight’s menu bar search. This means that Apple’s Mail app, and its plug-ins, is the only app which can search Mail’s mailboxes now. Similarly, a general Spotlight search in the Finder can’t see content indexed for the Notes app.

Although this might appear good for privacy, it’s exceedingly bad for search, particularly as there’s no way that a third-party app can be given user consent to search indexed content from apps like Mail and Notes. Even adding an app to the Full Disk Access list can’t enable its access to these protected parts of Spotlight’s indexes.

If you use HoudahSpot, you’ll be aware that it’s currently the only third-party app which can search Mail’s mailboxes using Spotlight’s indexes. FoxTrot Search can search them, but only using its own indexes, not those of Spotlight. HoudahSpot achieves this feat using an ingenious workaround: it installs a Mail plug-in, which gains access to Mail’s protected Spotlight indexes. So long as you’re running Mail with that plug-in, HoudahSpot can search your mailboxes.

For apps like Notes, the situation is even worse. Without a plug-in, third party apps can’t gain access to its indexed contents, and there’s not even a permission available in the Privacy pane which could grant access to its data. Notes doesn’t store your notes in documents, but in a database, which isn’t normally accessible through the Finder’s Spotlight search either. Under Core Spotlight behaviours, indexing of notes isn’t a task performed routinely by Spotlight’s mdworker processes either, but it’s up to Notes to call for indexing to be performed when it deems it necessary.

Apple warns developers that Core Spotlight “works best when you have no more than a few thousand items”. Although I can’t imagine anyone wanting more than a hundred or so notes, a great many users have more than a few thousand messages in their Mail mailboxes, which they might now expect to have sub-optimal search.

Further details of Core Spotlight’s indexing are here, and its developer interface is documented here.

In practice, Core Spotlight search in apps like Mail and Notes is poor in comparison with mainstream macOS Spotlight.

The Mailbox Search menu command in Apple’s Mail app claims to use ‘natural language search’, which is one of the most infuriating interfaces that I have come across. I suspect this is the non-oral version of Siri search, which is fine for speech but not in the least suited to trackpad and keyboard. At times, it decides that parts of the search text function as predicates, at other times it behaves quite differently. For anyone used to Finder search, it is simply too frustrating to use. For repeated searches, Apple recommends setting up Smart Mailboxes, which are different, and have their own problems.

For comparison, here’s the search feature in my chosen mail client, Postbox, which is what you might have expected in Mail, given its widespread adoption.

Notes apparently has the same ‘natural language search’ as that in Mail, and appears equally idiosyncratic in its behaviour. However, the biggest problem that I encountered searching in Notes is that it proved unable to find anything in locally-stored notes, and could only find text in notes stored in iCloud. I must presume that this is a bug.

The behaviour of Core Spotlight outside those apps is even more bizarre. Search using the Spotlight magnifying-glass tool in the menu bar, and its results include both messages in Mail’s mailboxes and items in Notes which are stored in iCloud, but not on local storage. Perform exactly the same search in a Finder window, though, and notes are excluded completely. Mail messages are there shown in the Finder as raw message files, rather than being previewed as messages in QuickLook. Third-party apps such as Find Any File and HoudahSpot – even when given Full Disk Access – aren’t given any search results from Core Spotlight, and can’t see Mail messages or entries in Notes at all.

The current state of Spotlight search in Big Sur is lamentable. Depending on what you try to search for, the Spotlight menu bar tool, Finder search, search in third-party apps, and in-app search each deliver different results using different interfaces, some of which don’t work at all well. None of the current macOS human interfaces to Spotlight’s rich indexes is well-designed or suitable for serious use. In short, Spotlight in macOS 11 needs redesign from scratch.