Problems with Spotlight are common, and those attributed to corespotlightd in concert with Pages have reached a peak. Accounts by those who have wrestled with them, such as Mitch Stone’s, demonstrate how conventional wisdom and long experience with Spotlight have proved unsuccessful, and Michael Tsai has a compilation of horror stories on his blog. This article argues that, because Spotlight and Core Spotlight are different, you can’t tackle problems in one with tools and techniques for the other.

Spotlight

Spotlight is the successor to a series of search tools for Mac OS, and was introduced in Mac OS X 10.4 Tiger in 2005. It has been described extensively over the years, and consists of per-volume indexes stored in the hidden .Spotlight-V100 folder at the root of each volume that’s indexed.

When new files are created, or existing ones changed, Spotlight indexes them very quickly. The first mdworker process is spawned within a second, and others are added to it. They’re active for about 0.2 seconds before the new posting lists they create are ready to be added to that volume’s indexes. They may later be followed by CGPDFService and mediaanalysisd running similar image analysis to that performed in Live Text. Text extracted from the files is then compressed by mds_stores before adding it to that volume’s Spotlight indexes, within seven seconds or so of file creation.

These steps are summarised in the diagram above, where those in blue update metadata indexes, and those in yellow and green update content indexes. It’s most likely that each file to be added to the indexes has its own individual mdworker process that works with a separate mdimporter. Although third-parties can supplement them, the great majority of metadata is extracted by bundled mdimporters that cover broad classes of file types, such as images.

The Spotlight API provides apps with access to search functions whose scope includes all indexed metadata on all indexed volumes.

Tools used to work with Spotlight, its metadata and volume indexes include mdls and mdutil.

Core Spotlight

Core Spotlight is a relative newcomer, and was first released in iOS 9 in 2015. Its original description is still accessible here, and bears no resemblance to Spotlight on macOS. The Core Spotlight API reached macOS High Sierra two years later, and Apple’s current documentation is detailed and informative. Five years ago I reported some of the different behaviours of Core Spotlight.

From the outset, Apple differentiates between the per-volume Spotlight index, and the multiple Core Spotlight indexes:
“Help people access activities and items within your app by adding details about those items to a Core Spotlight index. The framework provides APIs to add your content to an index, and search for items in that index.”

Core Spotlight indexes are located in ~/Library/Metadata/CoreSpotlight, as per-user and per-app indexes, and apps can have multiple indexes. They appear to have internal structural similarities with per-volume Spotlight indexes, for example that at NSFileProtectionComplete/index.spotlightV3.

In Core Spotlight, the client app decides what to index:
“You decide what content makes sense to index, but typically you index anything that someone might look for in your app. For example, you might index photos, contacts, the items someone purchased, or data they see in your interface. You can then use Core Spotlight to search for your indexed content and display those results in your app.”

The client app also indexes content and maintains its own Core Spotlight indexes:
“Your app is responsible for indexing your app’s content and maintaining those indexes. You can index content when your app runs, or provide an app extension to index content when the system requests it. You can index any content your app manages, including files and other content that your app isn’t currently displaying. The indexes you create using Core Spotlight remain on device, and are private to the owner of the device. Devices don’t share indexed data with Apple, or synchronize that data with the person’s other devices.”

Individual indexes can be assigned a protection class for additional security by encryption. Apps explicitly call for indexing to be performed using indexSearchableItems() for instance, which can be batched. Full index regeneration is supported by the CSSearchableIndexDelegate protocol, which can be called by a delegate object or app extension. It seems likely that corespotlightd is responsible for executing those.

Although Core Spotlight support by command tools appears almost non-existent, mdutil can be used to force an app’s indexing extension to be run.

Tackling problems

What we have learned over the years to cajole and bludgeon Spotlight to work for us is of little or no use when we’re dealing with Core Spotlight. Given the responsibilities of the client app in Core Spotlight, that needs to be the centre of attention. Deleting and regenerating Spotlight’s per-volume indexes is extremely unlikely to tackle problems in indexes buried in ~/Library/Metadata/CoreSpotlight. Although client apps are required to regenerate their own Core Spotlight indexes, those that aren’t supported by an app extension may not do so until we want to use them.

What we need most is better insight into Core Spotlight indexes, how to diagnose Core Spotlight problems, and above all to make a clear distinction between Spotlight and Core Spotlight.