hoakley Macs, Technology

What to do when Spotlight can’t find it

When Spotlight can’t find a local item that you think it should, there several quick checks to do immediately:

  1. If you’re using its Spotlight Search menu bar interface, switch to the Finder’s Find window instead, where you have better control over the search.
  2. Check meticulously the search text and settings. Typos are easy, and one incorrect letter can turn success into failure.
  3. Check that it’s searching “This Mac” rather than a single folder.
  4. Open System Settings > Siri & Spotlight, and check that the relevant search results are ticked.
  5. Click on the Spotlight Privacy… button below those, and check that the folder or volume likely to contain the item isn’t in the Privacy list.

The great majority of Spotlight failures will be fixed using one of those. If they’re all looking good, and you still can’t find what you want, working systematically through advanced checks and tests is more laborious, and relies on your understanding of how Spotlight works.

spotlightsteps1

Much of what happens with Spotlight is invisible, and doesn’t even get recorded in the log. In the diagram, the only parts which are visible to, or under direct control of, the user are shown in green.

Indexing

The process of indexing a file and its contents is initiated when an app (or other software) writes a new or changed file, as recorded in the hidden FSEvents database. This triggers an XPC call to process that file if it’s in a location within Spotlight’s scope.

The first cause of problems is that the file you’re interested in may be in a location which is excluded from indexing, hence the importance of inspecting the Spotlight Privacy settings, to ensure that Spotlight should be indexing that location. There are other situations in which items won’t get indexed, because that has been blocked by another mechanism, including:

  • appending the extension .noindex to the folder name (this previously worked using .no_index instead);
  • making the folder invisible to the Finder by prefixing a dot ‘.’ to its name;
  • putting an empty file named .metadata_never_index inside the folder; this no longer works in recent macOS.

Provided a changed file isn’t in an excluded location, an mdworker process should then start to add its contents to the volume indexes. To do this, it first checks what type of file it is, in terms of its UTI. If that’s incorrect, then the remainder of the steps won’t work properly. In most cases these days, that means the file must have the correct extension for its type. If it doesn’t, then mdworker won’t be able to index it correctly.

Spotlight then looks up the correct mdimporter for that type. For many file types, those are provided as part of macOS and stored in the system, in /System/Library/Spotlight. Importers for third party apps may be in /Library/Spotlight or its equivalent in your Home folder Library, or in a /Library/Spotlight folder inside the app itself. Use the command
mdimport -L
to list all mdimporter plugins currently installed, with their paths.

Spotlight importers and mdworker itself can crash when there’s a bug, or the mdimporter encounters a malformed file. If that happens, the log normally records repeated crashes and restarts of that mdworker process. In the past, it has been a common cause of problems more generally. If you can identify and remove the file that’s causing the problems, that can allow indexing to return to normal.

Once the mdworker has extracted the data from the file, those are added to the volume’s indexes in the hidden folder .Spotlight-V100. This is typically seen in a log entry from mds_stores containing the message
compressing 5686 bytes to <private>
or similar, for each file that has had text content extracted and added to the indexes. Those indexes may be missing or damaged, in which case you’ll need to force them to be rebuilt.

Search

There are at least four different flavours of Spotlight, including internet search, local search with or without limits, and Core Spotlight’s in-app search. Although their scope differs, local searches perform a query using a search predicate on the Spotlight indexes. There’s plenty of opportunity for error here: the scope of the search may be incorrect, for instance excluding specific folders, or Mail’s messages. The search predicate can be incorrect, and the search may fail partially or completely. As there’s no other way to make a direct query of the Spotlight database, these are almost impossible to check.

Searching is normally started with a log entry from the Spotlight server reading something like
KEEP!!!!! Client <private>[2892 2892] xpc checking in
following which there’s a directQueryOpenReply and QueryOpen is successful. Spotlight index then reports that it has sorted a number of flat pages, giving the time taken to do so. Spotlight’s response is then distinguished by the server responding with a directQueryFetchResultsReply, which gives a Quality of Service (qos) value for that search response.

Spotlight searches continue until they’re complete, and it’s possible for a calling app not to wait sufficiently long for all results to be gathered.

Mints

Mints has a whole section devoted to testing and observing Spotlight. This includes creation of a folder of nine test files in the user’s Home folder, carrying out a test search that should find those test files, custom log analysis, and cleaning up test files afterwards. This is all fully documented in the Spotlight check and test section of its Help book.

To start the test, click on the Create Test button at a known clock time, such as 12:30:00. Wait at least ten seconds, say until 12:30:10, then click the Check Search button. Give your Mac a good 20 seconds to complete that, then open the Log Window and view entries from 12:30:00 for a period of 30 seconds, to cover both the indexing of the test files and the search. Once you’re happy that all is well, you can delete the test files.

You can also customise the test files with other formats, provided that they contain the search text syzygy999 that Mints will look for in them.

In the current version of Mints, this test is fully compatible with all recent versions of macOS, including Sonoma.

Rebuilding the index

To force a volume to be re-indexed, open System Settings > Siri & Spotlight and click the Spotlight Privacy… button at the bottom (in System Preferences, the Spotlight pane and select its Privacy view). Click the + button at the foot, select the volume and add it to the list, then click Done. Pause thirty seconds or so, click the Spotlight Privacy… button again, select that volume in the list, and click the – button to remove it from the list. You don’t normally need to close System Settings or restart between adding and removing the volume.

If you prefer, you can instead use the mdutil command in Terminal. The command
mdutil -E /
erases the indexes on your Data volume and forces them to be rebuilt, and you can use the same option on other volumes.

As Spotlight indexes are maintained and stored on each volume, for the contents of that volume, you’ll need to repeat this for each volume on which you want to be able to search files by their contents.

The simplest way to check that re-indexing is taking place is to open Activity Monitor, and in its CPU view check that processes with names starting with md are taking plenty of CPU. These should include mds_stores, mdworker (often multiple copies) and mds itself. On Apple silicon, those processes run largely on the E cores, and are usually obvious in Activity Monitor’s CPU History window.

Mints is available free from here.