Spotlight search has been unreliable for several years because of a bug that misidentifies the type of some plain text files, so failing to index their contents. This has now been fixed in macOS Golden Gate 27.0.

Bug

This class of bugs came to light here almost a year ago, when plain text files starting with certain sequences of characters were found not to have their contents indexed by Spotlight. The first example, reported by Jürgen, started with the two capital letters LG, and Drew then reported an identical problem in those beginning with HPA or Draw. It was Drew who suggested the underlying cause, misidentification of those files using the ‘magic’ database used by the file(1) command.

Although all the files tested had a correct filename extension and UTI type of public.plain-text, so were indexed using RichText.mdimporter, they appeared to be reclassified using a method similar to that of file(1), apparently resulting in them being assigned to other types, such as a binary vector graphics format used by the !Draw app for RISC OS 2 in 1989. That in turn led to a decoding error in mdwrite, and their contents weren’t added to the Spotlight index. All the user then saw was a file that should have been found in local Spotlight search, but remained invisible as far as its contents were concerned.

Because there’s no way for a third-party importer to override this behaviour, there was no workaround other than to change the opening characters of affected files, but as they couldn’t be located using Spotlight search, there was no practical solution in macOS prior to Golden Gate. This bug seems to have been introduced in Mojave eight years ago, and has affected every release of macOS since.

Tests

Tests were performed on a Mac mini M4 Pro running the release version of Golden Gate 27.0, and in virtual machines hosted on that, running Tahoe 26.6.2 and Golden Gate 27.0. SpotTest was used throughout, together with the Finder’s Find window, to search for test files containing the word cattle in a range of different forms. In particular, one plain text test file started with the characters HPA and two with Draw.

SpotTest was also used to examine output from
/usr/bin/file -bk filepath
/usr/bin/mdimport -t -d2 filepath
/usr/bin/mdls filepath
for test files.

The Spotlight importer RichText.mdimporter selected for indexing by macOS in all tests, regardless of macOS used, was version 6.9 (350).

Results

Golden Gate 27.0 consistently found all test files, including those starting with HPA or Draw. Tahoe 26.6.2 found all other test files, but never those starting with HPA or Draw.

Examination of log entries during initial indexing of a file starting with Draw didn’t reveal any errors reported by mdwrite or any other Spotlight component.

file(1) still reported the previous misidentification of the Draw test as “RISC OS Draw file data”. However, mdimport reported finding
kMDItemTextContent = “<<< Text content of 51 characters >>>”;
and mdls reported success in obtaining a _kMDItemPrimaryTextEmbedding.

Conclusions

• Failure to index the contents of plain text files starting with characters that could cause misidentification of their type, present in macOS Mojave to Tahoe, no longer occurs in Golden Gate.
• In this respect at least, Spotlight indexing of plain text files should now be reliable.
• If the version numbers of RichText.mdimporter are correct, this bug occurred because of handling of type or contents elsewhere, and not in the mdimporter.

But

For some years now, the command tool mdimport has crashed consistently when used with its -d2 option on image files, as in
/usr/bin/mdimport -t -d2 imagefilepath
That works fine for the -d1 option, but using –d2 returns
Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[NSNull length]: unrecognized selector sent to instance 0x1f53c32f0'

Although not part of the user’s Spotlight, it would be really nice if that bug were to be fixed.

I’m very grateful to Apple’s engineers for fixing one of the most arcane and enduring bugs I have come across.