One of the most common reasons for Macs becoming sluggish or grinding to a halt are the processes involved in Spotlight and its indexing. You open Activity Monitor and see that mds_stores or an mdworker process have taken over your Mac. This article looks at how they should work, and what you can do to overpower them when they get out of hand.

Much of what happens with Spotlight is invisible, and doesn’t even get recorded in the log. In the diagram, the only parts that are visible to, or under direct control of, the user are shown in green.

Indexing

The process of indexing a file and its contents is initiated when an app (or other software) writes a new or changed file, as recorded in the hidden FSEvents database. This triggers an XPC call to process that file if it’s in a location within Spotlight’s scope.

The first step in discovering why Spotlight can’t find something is to inspect its Privacy Settings in System Settings > Siri & Spotlight (the Privacy tab in the Spotlight pane in System Preferences), to ensure that Spotlight should be indexing that location. There are other situations in which items won’t get indexed, because that has been blocked by another mechanism, including:

• appending the extension .noindex to the folder name (this previously worked using .no_index instead);
• making the folder invisible to the Finder by prefixing a dot ‘.’ to its name;
• putting an empty file named .metadata_never_index inside the folder; this no longer works in recent macOS.

Provided the changed file isn’t in an excluded location, an mdworker process should then start adding its contents to the volume indexes. To do this, it first checks what type of file it is, in terms of its UTI. If that’s incorrect, then the remainder of the steps won’t work properly. In most cases these days, that means the file must have the correct extension for its type. If it doesn’t, then mdworker won’t be able to index it correctly.

Spotlight then looks up the correct mdimporter for that type. For many file types, those are provided as part of macOS and stored in the system, in /System/Library/Spotlight. Importers for third party apps may be in /Library/Spotlight or its equivalent in your Home folder Library, or in the /Library/Spotlight folder inside the app itself. Use the command
mdimport -L
to list all mdimporter plugins currently installed, with their paths.

Spotlight importers and mdworker itself can crash when there’s a bug, or the mdimporter encounters a malformed file. If that happens, the log normally records repeated crashes and restarts of that mdworker process. In the past, this has been a common cause of more general problems. If you can identify and remove the file that’s causing those crashes, that should allow indexing and CPU use to return to normal.

Once the mdworker has extracted the data from the file, those are added to the volume’s indexes in the hidden folder .Spotlight-V100. This is typically seen in a log entry from mds_stores containing the message
compressing 5686 bytes to <private>
or similar, for each file which has had text content extracted and added to the indexes. Those indexes may be missing or damaged, in which case you’ll need to force them to be rebuilt.

Search

At my last count, there were at least four different flavours of Spotlight, including internet search, local search with or without limits, and Core Spotlight’s in-app search. Although their scope differs, local searches perform a query using a search predicate on the Spotlight indexes. There’s plenty of opportunity for error here: the scope of the search may be incorrect, for instance excluding specific folders, or Mail’s messages. The search predicate can be incorrect, and the search may fail partially or completely. As there’s no other way to make a direct query of the Spotlight database, these are almost impossible to check.

Searching may start with a log entry from the Spotlight server reading something like
KEEP!!!!! Client <private>[2892 2892] xpc checking in
following which there’s a directQueryOpenReply and QueryOpen is successful. Spotlight index then reports that it has sorted a number of flat pages, giving the time taken to do so. Spotlight’s response is then distinguished by the server responding with a directQueryFetchResultsReply, which gives a Quality of Service (qos) value for that search response.

Spotlight searches continue until they’re complete, and it’s possible for a calling app not to wait sufficiently long for all results to be gathered. To diagnose your Spotlight problems, try my free utility Mints, which gives you detailed information about each process, and will examine custom file types and their Importers too.

Rich Text

For almost a year before the release of macOS 11.3, there was a bug in the standard Rich Text mdimporter that failed to index the contents of all Rich Text files. Updating to 11.3 or later fixes this bug, but that will only index newly created or changed Rich Text files. Those Rich Text files created or changed between 15 July 2020 (release of the 10.15.6 update) and 26 April 2021 (release of the 11.3 update) still won’t be added to a volume’s Spotlight indexes unless you force those indexes to be rebuilt for that whole volume.

Forcing rebuild of a volume’s Spotlight indexes

One simple way to improve the results from Spotlight searches is to force a rebuild of the indexes on each volume whose contents you want included in searches. That normally includes at least your Data volume.

To force a volume to be re-indexed, open System Settings > Siri & Spotlight and click the Spotlight Privacy… button at the bottom (in System Preferences, the Spotlight pane and select its Privacy view). Click the + button at the foot, select the volume and add it to the list, then click Done. Pause thirty seconds or so, click the Spotlight Privacy… button again, select that volume in the list, and click the – button to remove it from the list. You don’t normally need to close System Settings or restart between adding and removing the volume.

If you prefer, you can instead use the mdutil command in Terminal. The command
mdutil -E /
erases the indexes on your Data volume and forces them to be rebuilt, and you can use the same option on other volumes.

As Spotlight indexes are maintained and stored on each volume, for the contents of that volume, you’ll need to repeat this for each volume on which you want to be able to search files by their contents.

Is Spotlight re-indexing?

There was a day when it was simple to tell whether you had been successful in forcing these indexes to be rebuilt: within a minute or so, the hard disk in question could be heard in frantic activity. I’ve never heard an SSD do anything yet.

There are two reliable ways to check that re-indexing is taking place. Open Activity Monitor, and in its CPU view check that processes with names starting with md are taking plenty of CPU. These should include mds_stores, mdworker (often multiple copies) and mds itself.

Alternatively, check the Spotlight entries in the log using my free Mints. Time that log extract to cover the start of re-indexing and you should find entries stating something like:
21-05-12 13:13:00.673 Server IndexCreate diskStore:0x1608be4d3 device:16777235 rc:-1
21-05-12 13:13:00.674 Index Creating New Index
which are a giveaway.

If Spotlight doesn’t start re-indexing after a few minutes, add the volume back to the Spotlight Privacy list, restart, then remove it again, and see if that kickstarts it.

Tackling excessive mds_stores

In the more distant past, it was mdworker’s crashes that were often related to sluggish performance. More recently, particularly since the introduction of Time Machine backups to APFS, it has become mds_stores that is more likely to be responsible for such problems. They are easily told apart using Activity Monitor’s CPU section, listing All Processes by CPU %.

One quick and simple panacea to try is starting your Mac up in Safe mode, leaving it a couple of minutes, then restarting back into regular mode. If that doesn’t make mds_stores behave, then you’ll need to start rebuilding indexes.

Shut your Mac down and disconnect all external disks, including most importantly all Time Machine backup stores. Start it up, leave it five minutes to settle, and check mds_stores again in Activity Monitor. If it’s still hyperactive, rebuild the index on your Data volume, as explained above. Once that process is complete, which could be in a few hours, reconnect your external disks, one by one, and rebuild the index on each before adding the next.

An even better solution, if you can, is to upgrade to Ventura, where the problem with indexing Time Machine backup storage seems to have largely resolved.