Spotlight on search: How Spotlight works

Spotlight – the macOS subsystem which supports search of both metadata and content – goes right back to Classic Mac OS days, in a feature which started off finding files in the Finder. It then became Sherlock, which controversially added web search features that had first appeared in Karelia’s app Watson. Spotlight arrived in 2005, almost 16 years ago, in Mac OS X 10.4 Tiger.

At that time, having 500 GB of hard disk space was considered impressive, and we were all more concerned with finding our documents than preventing others from reading their contents. Spotlight hasn’t stood still in the face of the many changes since its introduction: it now has to cope with remote storage in the form of iCloud, a whole slew of privacy controls, and Time Machine backups in APFS snapshots. In High Sierra, and increasingly since, macOS has been hiving off within-app search in what’s known as Core Spotlight.

For many users, Spotlight has steadily declined in value. A few years ago, it seemed to search everything (apart from system files), but now it can’t even find your Mail messages, and the whole data for apps like Notes are invisible to it. In these articles I’m going to explain how Spotlight works, and how to get the best out of what remains an amazingly powerful feature which is largely stunted by its tools.

Maintaining the indexes

At its heart is a hidden database in each volume’s .Spotlight-V100 folder, which is served by the mds daemon and helpers, running as root.

When any file in a watched folder is created or saved, Spotlight re-indexes that file for the volume’s Spotlight database. The process runs like this:

  1. The file change is recorded in that volume’s FSEvents database, in the volume’s .fseventsd folder.
  2. FSEvents notifies Spotlight that a file has been created or changed, prompting Spotlight to re-index the content and metadata of the file(s) concerned.
  3. One of the multiple copies of the mdworker daemon checks the type (UTI) of the changed file, and locates the appropriate mdimporter plugin bundle for that type. In the case of Rich Text files, this is /System/Library/Spotlight/RichText.mdimporter, for example. Additional plugins can be found in /Library/Spotlight/ and sometimes ~/Library/Spotlight/, but most app-specific plugins are now stored in the Library/Spotlight folder inside that app.
  4. The mdworker uses the mdimporter code to generate indexed content for the changed file.
  5. That indexed content is then added to the Spotlight database in the volume’s .Spotlight-V100 folder, for use in future searches.

That series of steps is usually completed within a second or two of the file being created or edited.

The shell command
mdimport -L
lists all mdimporter plugins currently installed, with their paths.

Limiting indexing

Spotlight doesn’t index every file on a volume. The simplest way to exclude volumes and folders from Spotlight’s database is to add them to the Privacy list in the Spotlight pane, in System Preferences. The Search Results list there determines which types of file will be included in the lists of hits from Spotlight. However, selecting an item such as Mail & Messages there doesn’t override system controls for privacy: in Catalina and Big Sur, Mail messages still won’t be included in general Spotlight search results. File types listed in Search Results are still indexed, but not included in search results.

You can also disable Spotlight indexing on a folder by traditional methods:

  • appending the extension .noindex to the folder name (this previously worked using .no_index instead);
  • making the folder invisible to the Finder by prefixing a dot ‘.’ to its name; as this isn’t allowed by the Finder, you’ll have to create that as a directory in Terminal;
  • putting an empty file named .metadata_never_index inside the folder; this no longer works in recent macOS.

Re-indexing

You can check the indexing status of mounted volumes using mdutil, which manages the metadata stores for Spotlight. The command
mdutil -s /
tells you whether indexing is enabled on your startup volume, for example, and
mdutil -E /
erases the indexes on your startup volume, forcing them to be rebuilt, a common way to try fixing Spotlight problems. You can achieve the same in the Spotlight pane in System Preferences: select the Privacy tab, then click the + button to add the volume to the exclusion list. Close the pane, and re-open it to remove the volume from that list using the – button. Spotlight will then start re-indexing that volume, which may take some time.

Sometimes forcing the rebuilding of Spotlight’s index files doesn’t fix a problem with Spotlight indexing, and log extracts contain repeated messages about mdworker or mdimporter failing to index a specific file. This occurs most commonly with components tucked away inside Library folders, for instance. You can fix this by replacing the file which is causing this grief; if you cannot find an undamaged copy, you may be able to trash the file, or hide it from being indexed by placing it inside a folder with the extension .noindex.

In the next article I’ll look at what Spotlight does index, and how its indexes can be searched.